[webservices] continuous data with dataselect service

Mon Jul 14 11:18:48 PDT 2014

Hi

Just one thought that might help clarify things. There is no notion in
miniseed of a "trace" or even a "large contiguous set of data". It is
just a whole bunch of 512 or 4096 byte data records whose begin/end
times might or might not line up to form contiguous data. Any "trace"
that you are seeing is an artifact of whatever you are reading the
miniseed with. The segmentation you are seeing is probably caused by
gaps, and the trace is the longest contiguous section of data.

If what you want is miniseed files that correspond 1:1 with the lines
of the request, then it would most likely have to be some other
service that returned maybe a tar flle containing miniseed files for
example.

$0.02
Philip

On Mon, Jul 14, 2014 at 1:40 PM, John D. West <john.d.west at asu.edu> wrote:
> Hi, Chad.
>
> Maybe "block" is the wrong term, sorry if it was confusing.
>
> I assemble a post request containing 100 lines. In general, each line
> represents one channel of broadband data for one station for one day, from
> 00:00:00 to 23:59:59.9999.
>
> My expectation was to get back a miniseed file containing 100 one-day
> traces. Instead, contiguous traces are combined so I get some lesser number
> of traces, many of which are longer than one day. I am not in principle
> opposed to this behavior, but I can envision times when it may cause
> problems, e.g., if I have some processing that has limits on the length of
> the trace to which it can be applied. It seems to me that optimum behavior
> would be a flag set by the user controlling whether you combine traces or
> not.
>
> Example: A request containing the following POST data is asking for 100 days
> of data. It returns 27 traces of varying length.
>
> POST text:
> quality=B
> TA G06A -- BHE 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> TA G06A -- BHE 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> TA G06A -- BHE 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> TA G06A -- BHE 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> TA G06A -- BHE 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> TA G06A -- BHE 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> TA G06A -- BHE 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> TA G06A -- BHE 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> TA F10A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> TA F10A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> TA F10A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> TA F10A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> TA F10A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> TA F10A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> TA F10A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> TA F10A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> TA F10A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> TA F10A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> TA F10A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> TA F10A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> TA F10A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> TA G06A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> TA G06A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> TA G06A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> TA G06A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> TA G06A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> TA G06A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> TA G06A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> TA G06A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> TA G06A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> TA G06A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> TA G06A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> TA G06A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> TA G06A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> TA G06A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> TA G06A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> TA G06A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> TA G06A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> TA G06A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> TA G06A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> TA G06A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> TA F18A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> TA F18A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> TA F18A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> TA F18A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> TA F18A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> TA F18A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> TA F18A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> TA F18A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> TA F18A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> TA F18A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> TA F18A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> TA F18A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> TA F18A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> TA F18A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> TA F18A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> TA F18A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> TA F18A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> TA F18A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> TA F18A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> TA F18A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> TA G06A -- BHE 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> TA G06A -- BHE 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> TA G06A -- BHE 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> TA G06A -- BHE 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> TA G06A -- BHE 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> TA G06A -- BHE 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> TA G06A -- BHE 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> TA G06A -- BHE 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> TA G06A -- BHE 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> TA G06A -- BHE 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> TA G06A -- BHE 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> TA G06A -- BHE 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> TA G16A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> TA G16A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> TA G16A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> TA G16A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> TA G16A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> TA G16A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> TA G16A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> TA G16A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> TA G16A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> TA G16A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> TA G16A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> TA G16A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> TA G16A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> TA G16A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> TA G16A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> TA G16A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> TA G16A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> TA G16A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> TA G16A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> TA G16A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> TA F10A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> TA F10A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> TA F10A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> TA F10A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> TA F10A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> TA F10A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> TA F10A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
>
> Thanks!
>
>      -- John
>
>
> On Thu, Jul 10, 2014 at 7:08 PM, Chad Trabant <chad at iris.washington.edu>
> wrote:
>>
>>
>> Hi John,
>>
>> Can you provide a bit more detail about what you mean by getting the data
>> back in blocks?  In a nutshell, if the data is continuous in the archive,
>> what you get should be continuous as long as you have requested continuous
>> segments.
>>
>> In the archive, the internal blocking of the miniSEED data (referred to as
>> records) are sometimes truncated at UTC day boundaries and other times there
>> are records that cross the UTC day boundaries.  For the TA data in
>> particular, the R quality data that arrived in real time generally does not
>> include records that cross the day boundaries (it has been cut to perfectly
>> fit within days).  But the Q quality data (a preferred copy) that arrives
>> many months after real time does have records that cross day boundaries.  On
>> top of this, by default the dataselect service will trim the data requested
>> to the time window specified by the user.
>>
>> I think if you can clarify what you are mean by blocking and perhaps
>> provide some example requests we might be able to give you a better answer.
>>
>> Chad
>>
>> On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west at asu.edu> wrote:
>>
>> > Hi.
>> >
>> > I'm requesting continuous broadband data from (mostly) TA stations using
>> > the POST usage of the dataselect web service.
>> >
>> > When I request multiple successive one-day blocks in the same POST
>> > request, I sometimes get the data back in 1-day blocks and sometimes it is
>> > combined into longer multi-day blocks.
>> >
>> > What controls this behavior? Is it possible to force the data to be
>> > returned in single-day traces without resorting to sending multiple
>> > individual requests, one per day?
>> >
>> > Thanks!
>> >      -- John
>> > _______________________________________________
>> > webservices mailing list
>> > webservices at iris.washington.edu
>> > http://www.iris.washington.edu/mailman/listinfo/webservices
>>
>
>
> _______________________________________________
> webservices mailing list
> webservices at iris.washington.edu
> http://www.iris.washington.edu/mailman/listinfo/webservices
>