Thread: continuous data with dataselect service

Hi John,

Sounds like it’s mostly cleared up now (thanks Philip).

If you really want the data base in 1-day traces for processing, storage,
etc. you can always make individual queries. Alternatively you could do
the logic to split up multi-day segments on your end.

Chad

On Jul 14, 2014, at 11:35 AM, John D. West <john.d.west<at>asu.edu> wrote:

Thanks, Philip! I didn't realize that. Since I am reading the returned
file with ObsPy, presumably that it what is assembling the data into
contiguous sections.

-- John


On Mon, Jul 14, 2014 at 11:18 AM, Philip Crotwell <crotwell<at>seis.sc.edu>
wrote:

Hi

Just one thought that might help clarify things. There is no notion in
miniseed of a "trace" or even a "large contiguous set of data". It is
just a whole bunch of 512 or 4096 byte data records whose begin/end
times might or might not line up to form contiguous data. Any "trace"
that you are seeing is an artifact of whatever you are reading the
miniseed with. The segmentation you are seeing is probably caused by
gaps, and the trace is the longest contiguous section of data.

If what you want is miniseed files that correspond 1:1 with the lines
of the request, then it would most likely have to be some other
service that returned maybe a tar flle containing miniseed files for
example.

$0.02
Philip


On Mon, Jul 14, 2014 at 1:40 PM, John D. West <john.d.west<at>asu.edu>
wrote:

Hi, Chad.

Maybe "block" is the wrong term, sorry if it was confusing.

I assemble a post request containing 100 lines. In general, each line
represents one channel of broadband data for one station for one day,

from

00:00:00 to 23:59:59.9999.

My expectation was to get back a miniseed file containing 100 one-day
traces. Instead, contiguous traces are combined so I get some lesser

number

of traces, many of which are longer than one day. I am not in principle
opposed to this behavior, but I can envision times when it may cause
problems, e.g., if I have some processing that has limits on the length

of

the trace to which it can be applied. It seems to me that optimum

behavior

would be a flag set by the user controlling whether you combine traces

or

not.

Example: A request containing the following POST data is asking for 100

days

of data. It returns 27 traces of varying length.

POST text:
quality=B
TA G06A -- BHE 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
TA G06A -- BHE 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
TA G06A -- BHE 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
TA G06A -- BHE 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
TA G06A -- BHE 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
TA G06A -- BHE 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
TA G06A -- BHE 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
TA G06A -- BHE 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
TA F10A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
TA F10A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
TA F10A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
TA F10A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
TA F10A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
TA F10A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
TA F10A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
TA F10A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
TA F10A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
TA F10A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
TA F10A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
TA F10A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
TA F10A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
TA G06A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
TA G06A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
TA G06A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
TA G06A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
TA G06A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
TA G06A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
TA G06A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
TA G06A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
TA G06A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
TA G06A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
TA G06A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
TA G06A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
TA G06A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
TA G06A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
TA G06A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
TA G06A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
TA G06A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
TA G06A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
TA G06A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
TA G06A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
TA F18A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
TA F18A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
TA F18A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
TA F18A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
TA F18A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
TA F18A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
TA F18A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
TA F18A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
TA F18A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
TA F18A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
TA F18A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
TA F18A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
TA F18A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
TA F18A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
TA F18A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
TA F18A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
TA F18A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
TA F18A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
TA F18A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
TA F18A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
TA G06A -- BHE 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
TA G06A -- BHE 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
TA G06A -- BHE 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
TA G06A -- BHE 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
TA G06A -- BHE 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
TA G06A -- BHE 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
TA G06A -- BHE 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
TA G06A -- BHE 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
TA G06A -- BHE 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
TA G06A -- BHE 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
TA G06A -- BHE 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
TA G06A -- BHE 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
TA G16A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
TA G16A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
TA G16A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
TA G16A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
TA G16A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
TA G16A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
TA G16A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
TA G16A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
TA G16A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
TA G16A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
TA G16A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
TA G16A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
TA G16A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
TA G16A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
TA G16A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
TA G16A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
TA G16A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
TA G16A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
TA G16A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
TA G16A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
TA F10A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
TA F10A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
TA F10A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
TA F10A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
TA F10A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
TA F10A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
TA F10A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999

Thanks!

-- John


On Thu, Jul 10, 2014 at 7:08 PM, Chad Trabant <chad<at>iris.washington.edu

wrote:

Hi John,

Can you provide a bit more detail about what you mean by getting the

data

back in blocks? In a nutshell, if the data is continuous in the

archive,

what you get should be continuous as long as you have requested

continuous

segments.

In the archive, the internal blocking of the miniSEED data (referred

to as

records) are sometimes truncated at UTC day boundaries and other times

there

are records that cross the UTC day boundaries. For the TA data in
particular, the R quality data that arrived in real time generally

does not

include records that cross the day boundaries (it has been cut to

perfectly

fit within days). But the Q quality data (a preferred copy) that

arrives

many months after real time does have records that cross day

boundaries. On

top of this, by default the dataselect service will trim the data

requested

to the time window specified by the user.

I think if you can clarify what you are mean by blocking and perhaps
provide some example requests we might be able to give you a better

answer.

Chad

On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west<at>asu.edu> wrote:

Hi.

I'm requesting continuous broadband data from (mostly) TA stations

using

the POST usage of the dataselect web service.

When I request multiple successive one-day blocks in the same POST
request, I sometimes get the data back in 1-day blocks and sometimes

it is

combined into longer multi-day blocks.

What controls this behavior? Is it possible to force the data to be
returned in single-day traces without resorting to sending multiple
individual requests, one per day?

Thanks!
-- John
_______________________________________________
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices

_______________________________________________
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices

_______________________________________________
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices