[webservices] continuous data with dataselect service

Mon Jul 14 11:35:30 PDT 2014

Thanks, Philip! I didn't realize that. Since I am reading the returned file
with ObsPy, presumably that it what is assembling the data into contiguous
sections.

     -- John

On Mon, Jul 14, 2014 at 11:18 AM, Philip Crotwell <crotwell at seis.sc.edu>
wrote:

> Hi
>
> Just one thought that might help clarify things. There is no notion in
> miniseed of a "trace" or even a "large contiguous set of data". It is
> just a whole bunch of 512 or 4096 byte data records whose begin/end
> times might or might not line up to form contiguous data. Any "trace"
> that you are seeing is an artifact of whatever you are reading the
> miniseed with. The segmentation you are seeing is probably caused by
> gaps, and the trace is the longest contiguous section of data.
>
> If what you want is miniseed files that correspond 1:1 with the lines
> of the request, then it would most likely have to be some other
> service that returned maybe a tar flle containing miniseed files for
> example.
>
> $0.02
> Philip
>
>
> On Mon, Jul 14, 2014 at 1:40 PM, John D. West <john.d.west at asu.edu> wrote:
> > Hi, Chad.
> >
> > Maybe "block" is the wrong term, sorry if it was confusing.
> >
> > I assemble a post request containing 100 lines. In general, each line
> > represents one channel of broadband data for one station for one day,
> from
> > 00:00:00 to 23:59:59.9999.
> >
> > My expectation was to get back a miniseed file containing 100 one-day
> > traces. Instead, contiguous traces are combined so I get some lesser
> number
> > of traces, many of which are longer than one day. I am not in principle
> > opposed to this behavior, but I can envision times when it may cause
> > problems, e.g., if I have some processing that has limits on the length
> of
> > the trace to which it can be applied. It seems to me that optimum
> behavior
> > would be a flag set by the user controlling whether you combine traces or
> > not.
> >
> > Example: A request containing the following POST data is asking for 100
> days
> > of data. It returns 27 traces of varying length.
> >
> > POST text:
> > quality=B
> > TA G06A -- BHE 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> > TA G06A -- BHE 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> > TA G06A -- BHE 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> > TA G06A -- BHE 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> > TA G06A -- BHE 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> > TA G06A -- BHE 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> > TA G06A -- BHE 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> > TA G06A -- BHE 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> > TA F10A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> > TA F10A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> > TA F10A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> > TA F10A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> > TA F10A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> > TA F10A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> > TA F10A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> > TA F10A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> > TA F10A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> > TA F10A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> > TA F10A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> > TA F10A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> > TA F10A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> > TA G06A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> > TA G06A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> > TA G06A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> > TA G06A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> > TA G06A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> > TA G06A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> > TA G06A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> > TA G06A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> > TA G06A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> > TA G06A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> > TA G06A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> > TA G06A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> > TA G06A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> > TA G06A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> > TA G06A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> > TA G06A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> > TA G06A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> > TA G06A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> > TA G06A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> > TA G06A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> > TA F18A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> > TA F18A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> > TA F18A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> > TA F18A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> > TA F18A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> > TA F18A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> > TA F18A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> > TA F18A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> > TA F18A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> > TA F18A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> > TA F18A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> > TA F18A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> > TA F18A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> > TA F18A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> > TA F18A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> > TA F18A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> > TA F18A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> > TA F18A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> > TA F18A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> > TA F18A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> > TA G06A -- BHE 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> > TA G06A -- BHE 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> > TA G06A -- BHE 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> > TA G06A -- BHE 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> > TA G06A -- BHE 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> > TA G06A -- BHE 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> > TA G06A -- BHE 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> > TA G06A -- BHE 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> > TA G06A -- BHE 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> > TA G06A -- BHE 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> > TA G06A -- BHE 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> > TA G06A -- BHE 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> > TA G16A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> > TA G16A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> > TA G16A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> > TA G16A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> > TA G16A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> > TA G16A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> > TA G16A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> > TA G16A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> > TA G16A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> > TA G16A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> > TA G16A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> > TA G16A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> > TA G16A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> > TA G16A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> > TA G16A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> > TA G16A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> > TA G16A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> > TA G16A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> > TA G16A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> > TA G16A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> > TA F10A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> > TA F10A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> > TA F10A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> > TA F10A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> > TA F10A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> > TA F10A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> > TA F10A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> >
> > Thanks!
> >
> >      -- John
> >
> >
> > On Thu, Jul 10, 2014 at 7:08 PM, Chad Trabant <chad at iris.washington.edu>
> > wrote:
> >>
> >>
> >> Hi John,
> >>
> >> Can you provide a bit more detail about what you mean by getting the
> data
> >> back in blocks?  In a nutshell, if the data is continuous in the
> archive,
> >> what you get should be continuous as long as you have requested
> continuous
> >> segments.
> >>
> >> In the archive, the internal blocking of the miniSEED data (referred to
> as
> >> records) are sometimes truncated at UTC day boundaries and other times
> there
> >> are records that cross the UTC day boundaries.  For the TA data in
> >> particular, the R quality data that arrived in real time generally does
> not
> >> include records that cross the day boundaries (it has been cut to
> perfectly
> >> fit within days).  But the Q quality data (a preferred copy) that
> arrives
> >> many months after real time does have records that cross day
> boundaries.  On
> >> top of this, by default the dataselect service will trim the data
> requested
> >> to the time window specified by the user.
> >>
> >> I think if you can clarify what you are mean by blocking and perhaps
> >> provide some example requests we might be able to give you a better
> answer.
> >>
> >> Chad
> >>
> >> On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west at asu.edu> wrote:
> >>
> >> > Hi.
> >> >
> >> > I'm requesting continuous broadband data from (mostly) TA stations
> using
> >> > the POST usage of the dataselect web service.
> >> >
> >> > When I request multiple successive one-day blocks in the same POST
> >> > request, I sometimes get the data back in 1-day blocks and sometimes
> it is
> >> > combined into longer multi-day blocks.
> >> >
> >> > What controls this behavior? Is it possible to force the data to be
> >> > returned in single-day traces without resorting to sending multiple
> >> > individual requests, one per day?
> >> >
> >> > Thanks!
> >> >      -- John
> >> > _______________________________________________
> >> > webservices mailing list
> >> > webservices at iris.washington.edu
> >> > http://www.iris.washington.edu/mailman/listinfo/webservices
> >>
> >
> >
> > _______________________________________________
> > webservices mailing list
> > webservices at iris.washington.edu
> > http://www.iris.washington.edu/mailman/listinfo/webservices
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.iris.washington.edu/pipermail/webservices/attachments/20140714/311a3175/attachment.html>