[webservices] continuous data with dataselect service

Mon Jul 14 12:03:16 PDT 2014

Hi John,

Sounds like it’s mostly cleared up now (thanks Philip).

If you really want the data base in 1-day traces for processing, storage, etc. you can always make individual queries.  Alternatively you could do the logic to split up multi-day segments on your end.

Chad

On Jul 14, 2014, at 11:35 AM, John D. West <john.d.west at asu.edu> wrote:

> Thanks, Philip! I didn't realize that. Since I am reading the returned file with ObsPy, presumably that it what is assembling the data into contiguous sections. 
> 
>      -- John
> 
> 
> On Mon, Jul 14, 2014 at 11:18 AM, Philip Crotwell <crotwell at seis.sc.edu> wrote:
> Hi
> 
> Just one thought that might help clarify things. There is no notion in
> miniseed of a "trace" or even a "large contiguous set of data". It is
> just a whole bunch of 512 or 4096 byte data records whose begin/end
> times might or might not line up to form contiguous data. Any "trace"
> that you are seeing is an artifact of whatever you are reading the
> miniseed with. The segmentation you are seeing is probably caused by
> gaps, and the trace is the longest contiguous section of data.
> 
> If what you want is miniseed files that correspond 1:1 with the lines
> of the request, then it would most likely have to be some other
> service that returned maybe a tar flle containing miniseed files for
> example.
> 
> $0.02
> Philip
> 
> 
> On Mon, Jul 14, 2014 at 1:40 PM, John D. West <john.d.west at asu.edu> wrote:
> > Hi, Chad.
> >
> > Maybe "block" is the wrong term, sorry if it was confusing.
> >
> > I assemble a post request containing 100 lines. In general, each line
> > represents one channel of broadband data for one station for one day, from
> > 00:00:00 to 23:59:59.9999.
> >
> > My expectation was to get back a miniseed file containing 100 one-day
> > traces. Instead, contiguous traces are combined so I get some lesser number
> > of traces, many of which are longer than one day. I am not in principle
> > opposed to this behavior, but I can envision times when it may cause
> > problems, e.g., if I have some processing that has limits on the length of
> > the trace to which it can be applied. It seems to me that optimum behavior
> > would be a flag set by the user controlling whether you combine traces or
> > not.
> >
> > Example: A request containing the following POST data is asking for 100 days
> > of data. It returns 27 traces of varying length.
> >
> > POST text:
> > quality=B
> > TA G06A -- BHE 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> > TA G06A -- BHE 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> > TA G06A -- BHE 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> > TA G06A -- BHE 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> > TA G06A -- BHE 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> > TA G06A -- BHE 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> > TA G06A -- BHE 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> > TA G06A -- BHE 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> > TA F10A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> > TA F10A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> > TA F10A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> > TA F10A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> > TA F10A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> > TA F10A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> > TA F10A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> > TA F10A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> > TA F10A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> > TA F10A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> > TA F10A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> > TA F10A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> > TA F10A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> > TA G06A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> > TA G06A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> > TA G06A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> > TA G06A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> > TA G06A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> > TA G06A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> > TA G06A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> > TA G06A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> > TA G06A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> > TA G06A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> > TA G06A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> > TA G06A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> > TA G06A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> > TA G06A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> > TA G06A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> > TA G06A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> > TA G06A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> > TA G06A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> > TA G06A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> > TA G06A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> > TA F18A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> > TA F18A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> > TA F18A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> > TA F18A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> > TA F18A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> > TA F18A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> > TA F18A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> > TA F18A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> > TA F18A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> > TA F18A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> > TA F18A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> > TA F18A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> > TA F18A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> > TA F18A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> > TA F18A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> > TA F18A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> > TA F18A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> > TA F18A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> > TA F18A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> > TA F18A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> > TA G06A -- BHE 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> > TA G06A -- BHE 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> > TA G06A -- BHE 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> > TA G06A -- BHE 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> > TA G06A -- BHE 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> > TA G06A -- BHE 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> > TA G06A -- BHE 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> > TA G06A -- BHE 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> > TA G06A -- BHE 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> > TA G06A -- BHE 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> > TA G06A -- BHE 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> > TA G06A -- BHE 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> > TA G16A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> > TA G16A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> > TA G16A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> > TA G16A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> > TA G16A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> > TA G16A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> > TA G16A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> > TA G16A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
> > TA G16A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
> > TA G16A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
> > TA G16A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
> > TA G16A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
> > TA G16A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
> > TA G16A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
> > TA G16A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
> > TA G16A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
> > TA G16A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
> > TA G16A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
> > TA G16A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
> > TA G16A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
> > TA F10A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
> > TA F10A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
> > TA F10A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
> > TA F10A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
> > TA F10A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
> > TA F10A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
> > TA F10A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
> >
> > Thanks!
> >
> >      -- John
> >
> >
> > On Thu, Jul 10, 2014 at 7:08 PM, Chad Trabant <chad at iris.washington.edu>
> > wrote:
> >>
> >>
> >> Hi John,
> >>
> >> Can you provide a bit more detail about what you mean by getting the data
> >> back in blocks?  In a nutshell, if the data is continuous in the archive,
> >> what you get should be continuous as long as you have requested continuous
> >> segments.
> >>
> >> In the archive, the internal blocking of the miniSEED data (referred to as
> >> records) are sometimes truncated at UTC day boundaries and other times there
> >> are records that cross the UTC day boundaries.  For the TA data in
> >> particular, the R quality data that arrived in real time generally does not
> >> include records that cross the day boundaries (it has been cut to perfectly
> >> fit within days).  But the Q quality data (a preferred copy) that arrives
> >> many months after real time does have records that cross day boundaries.  On
> >> top of this, by default the dataselect service will trim the data requested
> >> to the time window specified by the user.
> >>
> >> I think if you can clarify what you are mean by blocking and perhaps
> >> provide some example requests we might be able to give you a better answer.
> >>
> >> Chad
> >>
> >> On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west at asu.edu> wrote:
> >>
> >> > Hi.
> >> >
> >> > I'm requesting continuous broadband data from (mostly) TA stations using
> >> > the POST usage of the dataselect web service.
> >> >
> >> > When I request multiple successive one-day blocks in the same POST
> >> > request, I sometimes get the data back in 1-day blocks and sometimes it is
> >> > combined into longer multi-day blocks.
> >> >
> >> > What controls this behavior? Is it possible to force the data to be
> >> > returned in single-day traces without resorting to sending multiple
> >> > individual requests, one per day?
> >> >
> >> > Thanks!
> >> >      -- John
> >> > _______________________________________________
> >> > webservices mailing list
> >> > webservices at iris.washington.edu
> >> > http://www.iris.washington.edu/mailman/listinfo/webservices
> >>
> >
> >
> > _______________________________________________
> > webservices mailing list
> > webservices at iris.washington.edu
> > http://www.iris.washington.edu/mailman/listinfo/webservices
> >
> 
> _______________________________________________
> webservices mailing list
> webservices at iris.washington.edu
> http://www.iris.washington.edu/mailman/listinfo/webservices

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.iris.washington.edu/pipermail/webservices/attachments/20140714/71a2411c/attachment.html>