[webservices] continuous data with dataselect service

Mon Jul 14 12:21:27 PDT 2014

Thanks, Chad.

I was trying to avoid the individual queries to cut down on round-trip
overhead; if I do need one-day traces that is the way I will do it.

     -- John

On Mon, Jul 14, 2014 at 12:03 PM, Chad Trabant <chad at iris.washington.edu>
wrote:

>
> Hi John,
>
> Sounds like it’s mostly cleared up now (thanks Philip).
>
> If you really want the data base in 1-day traces for processing, storage,
> etc. you can always make individual queries.  Alternatively you could do
> the logic to split up multi-day segments on your end.
>
> Chad
>
> On Jul 14, 2014, at 11:35 AM, John D. West <john.d.west at asu.edu> wrote:
>
> Thanks, Philip! I didn't realize that. Since I am reading the returned
> file with ObsPy, presumably that it what is assembling the data into
> contiguous sections.
>
>      -- John
>
>
> On Mon, Jul 14, 2014 at 11:18 AM, Philip Crotwell <crotwell at seis.sc.edu>
> wrote:
>
>> Hi
>>
>> Just one thought that might help clarify things. There is no notion in
>> miniseed of a "trace" or even a "large contiguous set of data". It is
>> just a whole bunch of 512 or 4096 byte data records whose begin/end
>> times might or might not line up to form contiguous data. Any "trace"
>> that you are seeing is an artifact of whatever you are reading the
>> miniseed with. The segmentation you are seeing is probably caused by
>> gaps, and the trace is the longest contiguous section of data.
>>
>> If what you want is miniseed files that correspond 1:1 with the lines
>> of the request, then it would most likely have to be some other
>> service that returned maybe a tar flle containing miniseed files for
>> example.
>>
>> $0.02
>> Philip
>>
>>
>> On Mon, Jul 14, 2014 at 1:40 PM, John D. West <john.d.west at asu.edu>
>> wrote:
>> > Hi, Chad.
>> >
>> > Maybe "block" is the wrong term, sorry if it was confusing.
>> >
>> > I assemble a post request containing 100 lines. In general, each line
>> > represents one channel of broadband data for one station for one day,
>> from
>> > 00:00:00 to 23:59:59.9999.
>> >
>> > My expectation was to get back a miniseed file containing 100 one-day
>> > traces. Instead, contiguous traces are combined so I get some lesser
>> number
>> > of traces, many of which are longer than one day. I am not in principle
>> > opposed to this behavior, but I can envision times when it may cause
>> > problems, e.g., if I have some processing that has limits on the length
>> of
>> > the trace to which it can be applied. It seems to me that optimum
>> behavior
>> > would be a flag set by the user controlling whether you combine traces
>> or
>> > not.
>> >
>> > Example: A request containing the following POST data is asking for 100
>> days
>> > of data. It returns 27 traces of varying length.
>> >
>> > POST text:
>> > quality=B
>> > TA G06A -- BHE 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
>> > TA G06A -- BHE 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
>> > TA G06A -- BHE 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
>> > TA G06A -- BHE 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
>> > TA G06A -- BHE 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
>> > TA G06A -- BHE 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
>> > TA G06A -- BHE 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
>> > TA G06A -- BHE 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
>> > TA F10A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
>> > TA F10A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
>> > TA F10A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
>> > TA F10A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
>> > TA F10A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
>> > TA F10A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
>> > TA F10A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
>> > TA F10A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
>> > TA F10A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
>> > TA F10A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
>> > TA F10A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
>> > TA F10A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
>> > TA F10A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
>> > TA G06A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
>> > TA G06A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
>> > TA F18A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
>> > TA F18A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
>> > TA F18A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
>> > TA F18A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
>> > TA F18A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
>> > TA F18A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
>> > TA F18A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
>> > TA F18A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
>> > TA F18A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
>> > TA F18A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
>> > TA F18A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
>> > TA F18A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
>> > TA F18A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
>> > TA F18A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
>> > TA F18A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
>> > TA F18A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
>> > TA F18A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
>> > TA F18A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
>> > TA F18A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
>> > TA F18A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
>> > TA G06A -- BHE 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
>> > TA G06A -- BHE 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
>> > TA G06A -- BHE 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
>> > TA G06A -- BHE 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
>> > TA G06A -- BHE 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
>> > TA G06A -- BHE 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
>> > TA G06A -- BHE 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
>> > TA G06A -- BHE 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
>> > TA G06A -- BHE 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
>> > TA G06A -- BHE 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
>> > TA G06A -- BHE 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
>> > TA G06A -- BHE 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
>> > TA G16A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
>> > TA G16A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
>> > TA F10A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
>> > TA F10A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
>> > TA F10A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
>> > TA F10A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
>> > TA F10A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
>> > TA F10A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
>> > TA F10A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
>> >
>> > Thanks!
>> >
>> >      -- John
>> >
>> >
>> > On Thu, Jul 10, 2014 at 7:08 PM, Chad Trabant <chad at iris.washington.edu
>> >
>> > wrote:
>> >>
>> >>
>> >> Hi John,
>> >>
>> >> Can you provide a bit more detail about what you mean by getting the
>> data
>> >> back in blocks?  In a nutshell, if the data is continuous in the
>> archive,
>> >> what you get should be continuous as long as you have requested
>> continuous
>> >> segments.
>> >>
>> >> In the archive, the internal blocking of the miniSEED data (referred
>> to as
>> >> records) are sometimes truncated at UTC day boundaries and other times
>> there
>> >> are records that cross the UTC day boundaries.  For the TA data in
>> >> particular, the R quality data that arrived in real time generally
>> does not
>> >> include records that cross the day boundaries (it has been cut to
>> perfectly
>> >> fit within days).  But the Q quality data (a preferred copy) that
>> arrives
>> >> many months after real time does have records that cross day
>> boundaries.  On
>> >> top of this, by default the dataselect service will trim the data
>> requested
>> >> to the time window specified by the user.
>> >>
>> >> I think if you can clarify what you are mean by blocking and perhaps
>> >> provide some example requests we might be able to give you a better
>> answer.
>> >>
>> >> Chad
>> >>
>> >> On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west at asu.edu> wrote:
>> >>
>> >> > Hi.
>> >> >
>> >> > I'm requesting continuous broadband data from (mostly) TA stations
>> using
>> >> > the POST usage of the dataselect web service.
>> >> >
>> >> > When I request multiple successive one-day blocks in the same POST
>> >> > request, I sometimes get the data back in 1-day blocks and sometimes
>> it is
>> >> > combined into longer multi-day blocks.
>> >> >
>> >> > What controls this behavior? Is it possible to force the data to be
>> >> > returned in single-day traces without resorting to sending multiple
>> >> > individual requests, one per day?
>> >> >
>> >> > Thanks!
>> >> >      -- John
>> >> > _______________________________________________
>> >> > webservices mailing list
>> >> > webservices at iris.washington.edu
>> >> > http://www.iris.washington.edu/mailman/listinfo/webservices
>> >>
>> >
>> >
>> > _______________________________________________
>> > webservices mailing list
>> > webservices at iris.washington.edu
>> > http://www.iris.washington.edu/mailman/listinfo/webservices
>> >
>>
>
> _______________________________________________
> webservices mailing list
> webservices at iris.washington.edu
> http://www.iris.washington.edu/mailman/listinfo/webservices
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.iris.washington.edu/pipermail/webservices/attachments/20140714/7cd30731/attachment-0001.html>