gEDA-dev: SPICE rawwaveform file

[John Griessen]

Anthony J Bybell wrote:
> On Mon, 3 Dec 2007, Dan McMahill wrote:
>>I was envisioning ending up with this amount by
>> virtue of having lots and lots (10s of thousands) of waveforms in the
>> data file.  So the ability to pick out one of them efficiently and
>> without parsing all 10 Gb is important.
> 
> Right, that's the "fun" tricky part but it's not insurmountable.  There
> are plenty of ways to do this and a lot depends on the access patterns
> desired.  (e.g., read one/many waveforms across the whole dump, determine
> the values for everything at a given time value t, etc.)

I imagine a combination of both arrangements of data would be good for retrieving from disk drives.
First, record waveforms so they have values in a string for each wave, up to some time interval that is manageable, with time
segment ID at top of string, then switch to the next waveform until all done, then start the first wave for the next time 
interval, repeat.  That would let you find the chunks of data for wave 989 out of 3400 waves much quicker than
data arranged as all values per tiniest time step, repeat.

Perhaps the only non-database method to have speed for both kinds of searches is to record it both ways in
the same data file?

There is an assumption here that you can't have access to disk drive details like sector size, but maybe disk tools like LVM could 
help.  When you create a merge of two disks, can a tool like LVM create a new "virtual block size" that is smaller than the 
physical ones?  If so, you could create a storage method that uses the small block size as in "start all data chunks at a new 
block, with the fist line being a time segment and wave number tag.  Then some of the work done by a database would be done in 
mapping actual disk block zones of data to the small virtual blocks that let you _randomly_ access data better.

And the data could be just easy access text and still get the benefit I am thinking of above.

John G

-- 
Ecosensory   Austin TX
tinyOS devel on:  ubuntu Linux;   tinyOS v2.0.2;   telosb ecosens1