Thread: continuous data with dataselect service

Started: 2014-06-30 23:34:05
Last activity: 2014-07-14 21:18:48
Topics: Web Services
John West
2014-06-30 23:34:05
Hi.

I'm requesting continuous broadband data from (mostly) TA stations using
the POST usage of the dataselect web service.

When I request multiple successive one-day blocks in the same POST request,
I sometimes get the data back in 1-day blocks and sometimes it is combined
into longer multi-day blocks.

What controls this behavior? Is it possible to force the data to be
returned in single-day traces without resorting to sending multiple
individual requests, one per day?

Thanks!
-- John

  • Alex Hutko
    2014-07-01 01:41:57
    Hi John,

    I’m not sure if these are the answers you’re looking for, but:

    If there exists a gap in the day boundary, then you’ll get back two traces instead of one. Gaps can be 1ms small.

    The data downloading that you describe can easily be addressed with our FetchData perl script, a user-client that POSTs.

    https://seiscode.iris.washington.edu/projects/ws-fetch-scripts/wiki/Running_the_scripts_and_examples

    set day2 = 1
    foreach day1 ( 1 2 3 4 )
    @ day2++
    FetchData -C "LH?" -N TA -S M52A -s 2014,01,$day1,00,00,00 -e 2014,01,$day2,00,00,00 -o jday.$day1.M53A.mseed
    mseed2sac jday.$day1.M53A.mseed

    #permutations to get more than one station, leave fields blank to wildcard:
    #FetchData -C "LH?" -N TA -radius 41.44:-80.67:5 -s 2014,01,$day1,00,00,00 -e 2014,01,$day2,00,00,00 -msl 86399 -o jday.$day1.allTA_5deg_from_M53A.mseed
    #FetchData -C "LH?" -N TA -s 2014,01,$day1,00,00,00 -e 2014,01,$day2,00,00,00 -msl 86399 -o jday.$day1.allTA.mseed
    end

    If you still are noticing problems, can you please forward us examples so someone can address this better.
    Thanks,
    Alex


    On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west<at>asu.edu> wrote:

    Hi.

    I'm requesting continuous broadband data from (mostly) TA stations using the POST usage of the dataselect web service.

    When I request multiple successive one-day blocks in the same POST request, I sometimes get the data back in 1-day blocks and sometimes it is combined into longer multi-day blocks.

    What controls this behavior? Is it possible to force the data to be returned in single-day traces without resorting to sending multiple individual requests, one per day?

    Thanks!
    -- John
    _______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices



  • Chad Trabant
    2014-07-11 02:08:18

    Hi John,

    Can you provide a bit more detail about what you mean by getting the data back in blocks? In a nutshell, if the data is continuous in the archive, what you get should be continuous as long as you have requested continuous segments.

    In the archive, the internal blocking of the miniSEED data (referred to as records) are sometimes truncated at UTC day boundaries and other times there are records that cross the UTC day boundaries. For the TA data in particular, the R quality data that arrived in real time generally does not include records that cross the day boundaries (it has been cut to perfectly fit within days). But the Q quality data (a preferred copy) that arrives many months after real time does have records that cross day boundaries. On top of this, by default the dataselect service will trim the data requested to the time window specified by the user.

    I think if you can clarify what you are mean by blocking and perhaps provide some example requests we might be able to give you a better answer.

    Chad

    On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west<at>asu.edu> wrote:

    Hi.

    I'm requesting continuous broadband data from (mostly) TA stations using the POST usage of the dataselect web service.

    When I request multiple successive one-day blocks in the same POST request, I sometimes get the data back in 1-day blocks and sometimes it is combined into longer multi-day blocks.

    What controls this behavior? Is it possible to force the data to be returned in single-day traces without resorting to sending multiple individual requests, one per day?

    Thanks!
    -- John
    _______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices



    • John West
      2014-07-14 17:40:58
      Hi, Chad.

      Maybe "block" is the wrong term, sorry if it was confusing.

      I assemble a post request containing 100 lines. In general, each line
      represents one channel of broadband data for one station for one day, from
      00:00:00 to 23:59:59.9999.

      My expectation was to get back a miniseed file containing 100 one-day
      traces. Instead, contiguous traces are combined so I get some lesser number
      of traces, many of which are longer than one day. I am not in principle
      opposed to this behavior, but I can envision times when it may cause
      problems, e.g., if I have some processing that has limits on the length of
      the trace to which it can be applied. It seems to me that optimum behavior
      would be a flag set by the user controlling whether you combine traces or
      not.

      Example: A request containing the following POST data is asking for 100
      days of data. It returns 27 traces of varying length.

      POST text:
      quality=B
      TA G06A -- BHE 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
      TA G06A -- BHE 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
      TA G06A -- BHE 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
      TA G06A -- BHE 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
      TA G06A -- BHE 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
      TA G06A -- BHE 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
      TA G06A -- BHE 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
      TA G06A -- BHE 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
      TA F10A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
      TA F10A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
      TA F10A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
      TA F10A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
      TA F10A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
      TA F10A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
      TA F10A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
      TA F10A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
      TA F10A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
      TA F10A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
      TA F10A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
      TA F10A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
      TA F10A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
      TA G06A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
      TA G06A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
      TA G06A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
      TA G06A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
      TA G06A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
      TA G06A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
      TA G06A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
      TA G06A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
      TA G06A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
      TA G06A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
      TA G06A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
      TA G06A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
      TA G06A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
      TA G06A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
      TA G06A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
      TA G06A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
      TA G06A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
      TA G06A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
      TA G06A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
      TA G06A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
      TA F18A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
      TA F18A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
      TA F18A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
      TA F18A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
      TA F18A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
      TA F18A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
      TA F18A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
      TA F18A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
      TA F18A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
      TA F18A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
      TA F18A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
      TA F18A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
      TA F18A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
      TA F18A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
      TA F18A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
      TA F18A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
      TA F18A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
      TA F18A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
      TA F18A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
      TA F18A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
      TA G06A -- BHE 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
      TA G06A -- BHE 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
      TA G06A -- BHE 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
      TA G06A -- BHE 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
      TA G06A -- BHE 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
      TA G06A -- BHE 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
      TA G06A -- BHE 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
      TA G06A -- BHE 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
      TA G06A -- BHE 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
      TA G06A -- BHE 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
      TA G06A -- BHE 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
      TA G06A -- BHE 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
      TA G16A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
      TA G16A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
      TA G16A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
      TA G16A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
      TA G16A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
      TA G16A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
      TA G16A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
      TA G16A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
      TA G16A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
      TA G16A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
      TA G16A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
      TA G16A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
      TA G16A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
      TA G16A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
      TA G16A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
      TA G16A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
      TA G16A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
      TA G16A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
      TA G16A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
      TA G16A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
      TA F10A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
      TA F10A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
      TA F10A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
      TA F10A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
      TA F10A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
      TA F10A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
      TA F10A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999

      Thanks!

      -- John


      On Thu, Jul 10, 2014 at 7:08 PM, Chad Trabant <chad<at>iris.washington.edu>
      wrote:


      Hi John,

      Can you provide a bit more detail about what you mean by getting the data
      back in blocks? In a nutshell, if the data is continuous in the archive,
      what you get should be continuous as long as you have requested continuous
      segments.

      In the archive, the internal blocking of the miniSEED data (referred to as
      records) are sometimes truncated at UTC day boundaries and other times
      there are records that cross the UTC day boundaries. For the TA data in
      particular, the R quality data that arrived in real time generally does not
      include records that cross the day boundaries (it has been cut to perfectly
      fit within days). But the Q quality data (a preferred copy) that arrives
      many months after real time does have records that cross day boundaries.
      On top of this, by default the dataselect service will trim the data
      requested to the time window specified by the user.

      I think if you can clarify what you are mean by blocking and perhaps
      provide some example requests we might be able to give you a better answer.

      Chad

      On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west<at>asu.edu> wrote:

      Hi.

      I'm requesting continuous broadband data from (mostly) TA stations using
      the POST usage of the dataselect web service.

      When I request multiple successive one-day blocks in the same POST
      request, I sometimes get the data back in 1-day blocks and sometimes it is
      combined into longer multi-day blocks.

      What controls this behavior? Is it possible to force the data to be
      returned in single-day traces without resorting to sending multiple
      individual requests, one per day?

      Thanks!
      -- John
      _______________________________________________
      webservices mailing list
      webservices<at>iris.washington.edu
      http://www.iris.washington.edu/mailman/listinfo/webservices



      • Philip Crotwell
        2014-07-14 21:18:48
        Hi

        Just one thought that might help clarify things. There is no notion in
        miniseed of a "trace" or even a "large contiguous set of data". It is
        just a whole bunch of 512 or 4096 byte data records whose begin/end
        times might or might not line up to form contiguous data. Any "trace"
        that you are seeing is an artifact of whatever you are reading the
        miniseed with. The segmentation you are seeing is probably caused by
        gaps, and the trace is the longest contiguous section of data.

        If what you want is miniseed files that correspond 1:1 with the lines
        of the request, then it would most likely have to be some other
        service that returned maybe a tar flle containing miniseed files for
        example.

        $0.02
        Philip


        On Mon, Jul 14, 2014 at 1:40 PM, John D. West <john.d.west<at>asu.edu> wrote:
        Hi, Chad.

        Maybe "block" is the wrong term, sorry if it was confusing.

        I assemble a post request containing 100 lines. In general, each line
        represents one channel of broadband data for one station for one day, from
        00:00:00 to 23:59:59.9999.

        My expectation was to get back a miniseed file containing 100 one-day
        traces. Instead, contiguous traces are combined so I get some lesser number
        of traces, many of which are longer than one day. I am not in principle
        opposed to this behavior, but I can envision times when it may cause
        problems, e.g., if I have some processing that has limits on the length of
        the trace to which it can be applied. It seems to me that optimum behavior
        would be a flag set by the user controlling whether you combine traces or
        not.

        Example: A request containing the following POST data is asking for 100 days
        of data. It returns 27 traces of varying length.

        POST text:
        quality=B
        TA G06A -- BHE 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
        TA G06A -- BHE 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
        TA G06A -- BHE 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
        TA G06A -- BHE 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
        TA G06A -- BHE 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
        TA G06A -- BHE 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
        TA G06A -- BHE 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
        TA G06A -- BHE 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
        TA F10A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
        TA F10A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
        TA F10A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
        TA F10A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
        TA F10A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
        TA F10A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
        TA F10A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
        TA F10A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
        TA F10A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
        TA F10A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
        TA F10A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
        TA F10A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
        TA F10A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
        TA G06A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
        TA G06A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
        TA G06A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
        TA G06A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
        TA G06A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
        TA G06A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
        TA G06A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
        TA G06A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
        TA G06A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
        TA G06A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
        TA G06A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
        TA G06A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
        TA G06A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
        TA G06A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
        TA G06A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
        TA G06A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
        TA G06A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
        TA G06A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
        TA G06A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
        TA G06A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
        TA F18A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
        TA F18A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
        TA F18A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
        TA F18A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
        TA F18A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
        TA F18A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
        TA F18A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
        TA F18A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
        TA F18A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
        TA F18A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
        TA F18A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
        TA F18A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
        TA F18A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
        TA F18A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
        TA F18A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
        TA F18A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
        TA F18A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
        TA F18A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
        TA F18A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
        TA F18A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
        TA G06A -- BHE 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
        TA G06A -- BHE 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
        TA G06A -- BHE 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
        TA G06A -- BHE 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
        TA G06A -- BHE 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
        TA G06A -- BHE 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
        TA G06A -- BHE 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
        TA G06A -- BHE 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
        TA G06A -- BHE 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
        TA G06A -- BHE 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
        TA G06A -- BHE 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
        TA G06A -- BHE 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
        TA G16A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
        TA G16A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
        TA G16A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
        TA G16A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
        TA G16A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
        TA G16A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
        TA G16A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
        TA G16A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
        TA G16A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
        TA G16A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
        TA G16A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
        TA G16A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
        TA G16A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
        TA G16A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
        TA G16A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
        TA G16A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
        TA G16A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
        TA G16A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
        TA G16A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
        TA G16A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
        TA F10A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
        TA F10A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
        TA F10A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
        TA F10A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
        TA F10A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
        TA F10A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
        TA F10A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999

        Thanks!

        -- John


        On Thu, Jul 10, 2014 at 7:08 PM, Chad Trabant <chad<at>iris.washington.edu>
        wrote:


        Hi John,

        Can you provide a bit more detail about what you mean by getting the data
        back in blocks? In a nutshell, if the data is continuous in the archive,
        what you get should be continuous as long as you have requested continuous
        segments.

        In the archive, the internal blocking of the miniSEED data (referred to as
        records) are sometimes truncated at UTC day boundaries and other times there
        are records that cross the UTC day boundaries. For the TA data in
        particular, the R quality data that arrived in real time generally does not
        include records that cross the day boundaries (it has been cut to perfectly
        fit within days). But the Q quality data (a preferred copy) that arrives
        many months after real time does have records that cross day boundaries. On
        top of this, by default the dataselect service will trim the data requested
        to the time window specified by the user.

        I think if you can clarify what you are mean by blocking and perhaps
        provide some example requests we might be able to give you a better answer.

        Chad

        On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west<at>asu.edu> wrote:

        Hi.

        I'm requesting continuous broadband data from (mostly) TA stations using
        the POST usage of the dataselect web service.

        When I request multiple successive one-day blocks in the same POST
        request, I sometimes get the data back in 1-day blocks and sometimes it is
        combined into longer multi-day blocks.

        What controls this behavior? Is it possible to force the data to be
        returned in single-day traces without resorting to sending multiple
        individual requests, one per day?

        Thanks!
        -- John
        _______________________________________________
        webservices mailing list
        webservices<at>iris.washington.edu
        http://www.iris.washington.edu/mailman/listinfo/webservices



        _______________________________________________
        webservices mailing list
        webservices<at>iris.washington.edu
        http://www.iris.washington.edu/mailman/listinfo/webservices


        • John West
          2014-07-14 18:35:30
          Thanks, Philip! I didn't realize that. Since I am reading the returned file
          with ObsPy, presumably that it what is assembling the data into contiguous
          sections.

          -- John


          On Mon, Jul 14, 2014 at 11:18 AM, Philip Crotwell <crotwell<at>seis.sc.edu>
          wrote:

          Hi

          Just one thought that might help clarify things. There is no notion in
          miniseed of a "trace" or even a "large contiguous set of data". It is
          just a whole bunch of 512 or 4096 byte data records whose begin/end
          times might or might not line up to form contiguous data. Any "trace"
          that you are seeing is an artifact of whatever you are reading the
          miniseed with. The segmentation you are seeing is probably caused by
          gaps, and the trace is the longest contiguous section of data.

          If what you want is miniseed files that correspond 1:1 with the lines
          of the request, then it would most likely have to be some other
          service that returned maybe a tar flle containing miniseed files for
          example.

          $0.02
          Philip


          On Mon, Jul 14, 2014 at 1:40 PM, John D. West <john.d.west<at>asu.edu> wrote:
          Hi, Chad.

          Maybe "block" is the wrong term, sorry if it was confusing.

          I assemble a post request containing 100 lines. In general, each line
          represents one channel of broadband data for one station for one day,
          from
          00:00:00 to 23:59:59.9999.

          My expectation was to get back a miniseed file containing 100 one-day
          traces. Instead, contiguous traces are combined so I get some lesser
          number
          of traces, many of which are longer than one day. I am not in principle
          opposed to this behavior, but I can envision times when it may cause
          problems, e.g., if I have some processing that has limits on the length
          of
          the trace to which it can be applied. It seems to me that optimum
          behavior
          would be a flag set by the user controlling whether you combine traces or
          not.

          Example: A request containing the following POST data is asking for 100
          days
          of data. It returns 27 traces of varying length.

          POST text:
          quality=B
          TA G06A -- BHE 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
          TA G06A -- BHE 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
          TA G06A -- BHE 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
          TA G06A -- BHE 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
          TA G06A -- BHE 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
          TA G06A -- BHE 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
          TA G06A -- BHE 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
          TA G06A -- BHE 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
          TA F10A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
          TA F10A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
          TA F10A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
          TA F10A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
          TA F10A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
          TA F10A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
          TA F10A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
          TA F10A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
          TA F10A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
          TA F10A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
          TA F10A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
          TA F10A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
          TA F10A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
          TA G06A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
          TA G06A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
          TA G06A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
          TA G06A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
          TA G06A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
          TA G06A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
          TA G06A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
          TA G06A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
          TA G06A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
          TA G06A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
          TA G06A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
          TA G06A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
          TA G06A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
          TA G06A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
          TA G06A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
          TA G06A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
          TA G06A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
          TA G06A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
          TA G06A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
          TA G06A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
          TA F18A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
          TA F18A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
          TA F18A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
          TA F18A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
          TA F18A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
          TA F18A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
          TA F18A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
          TA F18A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
          TA F18A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
          TA F18A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
          TA F18A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
          TA F18A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
          TA F18A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
          TA F18A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
          TA F18A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
          TA F18A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
          TA F18A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
          TA F18A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
          TA F18A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
          TA F18A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
          TA G06A -- BHE 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
          TA G06A -- BHE 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
          TA G06A -- BHE 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
          TA G06A -- BHE 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
          TA G06A -- BHE 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
          TA G06A -- BHE 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
          TA G06A -- BHE 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
          TA G06A -- BHE 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
          TA G06A -- BHE 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
          TA G06A -- BHE 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
          TA G06A -- BHE 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
          TA G06A -- BHE 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
          TA G16A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
          TA G16A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
          TA G16A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
          TA G16A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
          TA G16A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
          TA G16A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
          TA G16A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
          TA G16A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
          TA G16A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
          TA G16A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
          TA G16A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
          TA G16A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
          TA G16A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
          TA G16A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
          TA G16A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
          TA G16A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
          TA G16A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
          TA G16A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
          TA G16A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
          TA G16A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
          TA F10A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
          TA F10A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
          TA F10A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
          TA F10A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
          TA F10A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
          TA F10A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
          TA F10A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999

          Thanks!

          -- John


          On Thu, Jul 10, 2014 at 7:08 PM, Chad Trabant <chad<at>iris.washington.edu>
          wrote:


          Hi John,

          Can you provide a bit more detail about what you mean by getting the
          data
          back in blocks? In a nutshell, if the data is continuous in the
          archive,
          what you get should be continuous as long as you have requested
          continuous
          segments.

          In the archive, the internal blocking of the miniSEED data (referred to
          as
          records) are sometimes truncated at UTC day boundaries and other times
          there
          are records that cross the UTC day boundaries. For the TA data in
          particular, the R quality data that arrived in real time generally does
          not
          include records that cross the day boundaries (it has been cut to
          perfectly
          fit within days). But the Q quality data (a preferred copy) that
          arrives
          many months after real time does have records that cross day
          boundaries. On
          top of this, by default the dataselect service will trim the data
          requested
          to the time window specified by the user.

          I think if you can clarify what you are mean by blocking and perhaps
          provide some example requests we might be able to give you a better
          answer.

          Chad

          On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west<at>asu.edu> wrote:

          Hi.

          I'm requesting continuous broadband data from (mostly) TA stations
          using
          the POST usage of the dataselect web service.

          When I request multiple successive one-day blocks in the same POST
          request, I sometimes get the data back in 1-day blocks and sometimes
          it is
          combined into longer multi-day blocks.

          What controls this behavior? Is it possible to force the data to be
          returned in single-day traces without resorting to sending multiple
          individual requests, one per day?

          Thanks!
          -- John
          _______________________________________________
          webservices mailing list
          webservices<at>iris.washington.edu
          http://www.iris.washington.edu/mailman/listinfo/webservices



          _______________________________________________
          webservices mailing list
          webservices<at>iris.washington.edu
          http://www.iris.washington.edu/mailman/listinfo/webservices



          • Chad Trabant
            2014-07-14 19:03:16

            Hi John,

            Sounds like it’s mostly cleared up now (thanks Philip).

            If you really want the data base in 1-day traces for processing, storage, etc. you can always make individual queries. Alternatively you could do the logic to split up multi-day segments on your end.

            Chad

            On Jul 14, 2014, at 11:35 AM, John D. West <john.d.west<at>asu.edu> wrote:

            Thanks, Philip! I didn't realize that. Since I am reading the returned file with ObsPy, presumably that it what is assembling the data into contiguous sections.

            -- John


            On Mon, Jul 14, 2014 at 11:18 AM, Philip Crotwell <crotwell<at>seis.sc.edu> wrote:
            Hi

            Just one thought that might help clarify things. There is no notion in
            miniseed of a "trace" or even a "large contiguous set of data". It is
            just a whole bunch of 512 or 4096 byte data records whose begin/end
            times might or might not line up to form contiguous data. Any "trace"
            that you are seeing is an artifact of whatever you are reading the
            miniseed with. The segmentation you are seeing is probably caused by
            gaps, and the trace is the longest contiguous section of data.

            If what you want is miniseed files that correspond 1:1 with the lines
            of the request, then it would most likely have to be some other
            service that returned maybe a tar flle containing miniseed files for
            example.

            $0.02
            Philip


            On Mon, Jul 14, 2014 at 1:40 PM, John D. West <john.d.west<at>asu.edu> wrote:
            Hi, Chad.

            Maybe "block" is the wrong term, sorry if it was confusing.

            I assemble a post request containing 100 lines. In general, each line
            represents one channel of broadband data for one station for one day, from
            00:00:00 to 23:59:59.9999.

            My expectation was to get back a miniseed file containing 100 one-day
            traces. Instead, contiguous traces are combined so I get some lesser number
            of traces, many of which are longer than one day. I am not in principle
            opposed to this behavior, but I can envision times when it may cause
            problems, e.g., if I have some processing that has limits on the length of
            the trace to which it can be applied. It seems to me that optimum behavior
            would be a flag set by the user controlling whether you combine traces or
            not.

            Example: A request containing the following POST data is asking for 100 days
            of data. It returns 27 traces of varying length.

            POST text:
            quality=B
            TA G06A -- BHE 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
            TA G06A -- BHE 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
            TA G06A -- BHE 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
            TA G06A -- BHE 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
            TA G06A -- BHE 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
            TA G06A -- BHE 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
            TA G06A -- BHE 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
            TA G06A -- BHE 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
            TA F10A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
            TA F10A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
            TA F10A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
            TA F10A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
            TA F10A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
            TA F10A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
            TA F10A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
            TA F10A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
            TA F10A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
            TA F10A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
            TA F10A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
            TA F10A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
            TA F10A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
            TA G06A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
            TA G06A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
            TA G06A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
            TA G06A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
            TA G06A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
            TA G06A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
            TA G06A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
            TA G06A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
            TA G06A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
            TA G06A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
            TA G06A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
            TA G06A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
            TA G06A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
            TA G06A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
            TA G06A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
            TA G06A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
            TA G06A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
            TA G06A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
            TA G06A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
            TA G06A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
            TA F18A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
            TA F18A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
            TA F18A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
            TA F18A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
            TA F18A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
            TA F18A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
            TA F18A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
            TA F18A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
            TA F18A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
            TA F18A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
            TA F18A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
            TA F18A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
            TA F18A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
            TA F18A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
            TA F18A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
            TA F18A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
            TA F18A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
            TA F18A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
            TA F18A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
            TA F18A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
            TA G06A -- BHE 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
            TA G06A -- BHE 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
            TA G06A -- BHE 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
            TA G06A -- BHE 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
            TA G06A -- BHE 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
            TA G06A -- BHE 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
            TA G06A -- BHE 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
            TA G06A -- BHE 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
            TA G06A -- BHE 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
            TA G06A -- BHE 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
            TA G06A -- BHE 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
            TA G06A -- BHE 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
            TA G16A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
            TA G16A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
            TA G16A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
            TA G16A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
            TA G16A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
            TA G16A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
            TA G16A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
            TA G16A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
            TA G16A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
            TA G16A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
            TA G16A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
            TA G16A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
            TA G16A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
            TA G16A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
            TA G16A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
            TA G16A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
            TA G16A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
            TA G16A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
            TA G16A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
            TA G16A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
            TA F10A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
            TA F10A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
            TA F10A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
            TA F10A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
            TA F10A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
            TA F10A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
            TA F10A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999

            Thanks!

            -- John


            On Thu, Jul 10, 2014 at 7:08 PM, Chad Trabant <chad<at>iris.washington.edu>
            wrote:


            Hi John,

            Can you provide a bit more detail about what you mean by getting the data
            back in blocks? In a nutshell, if the data is continuous in the archive,
            what you get should be continuous as long as you have requested continuous
            segments.

            In the archive, the internal blocking of the miniSEED data (referred to as
            records) are sometimes truncated at UTC day boundaries and other times there
            are records that cross the UTC day boundaries. For the TA data in
            particular, the R quality data that arrived in real time generally does not
            include records that cross the day boundaries (it has been cut to perfectly
            fit within days). But the Q quality data (a preferred copy) that arrives
            many months after real time does have records that cross day boundaries. On
            top of this, by default the dataselect service will trim the data requested
            to the time window specified by the user.

            I think if you can clarify what you are mean by blocking and perhaps
            provide some example requests we might be able to give you a better answer.

            Chad

            On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west<at>asu.edu> wrote:

            Hi.

            I'm requesting continuous broadband data from (mostly) TA stations using
            the POST usage of the dataselect web service.

            When I request multiple successive one-day blocks in the same POST
            request, I sometimes get the data back in 1-day blocks and sometimes it is
            combined into longer multi-day blocks.

            What controls this behavior? Is it possible to force the data to be
            returned in single-day traces without resorting to sending multiple
            individual requests, one per day?

            Thanks!
            -- John
            _______________________________________________
            webservices mailing list
            webservices<at>iris.washington.edu
            http://www.iris.washington.edu/mailman/listinfo/webservices



            _______________________________________________
            webservices mailing list
            webservices<at>iris.washington.edu
            http://www.iris.washington.edu/mailman/listinfo/webservices


            _______________________________________________
            webservices mailing list
            webservices<at>iris.washington.edu
            http://www.iris.washington.edu/mailman/listinfo/webservices


            • John West
              2014-07-14 19:21:27
              Thanks, Chad.

              I was trying to avoid the individual queries to cut down on round-trip
              overhead; if I do need one-day traces that is the way I will do it.

              -- John


              On Mon, Jul 14, 2014 at 12:03 PM, Chad Trabant <chad<at>iris.washington.edu>
              wrote:


              Hi John,

              Sounds like it’s mostly cleared up now (thanks Philip).

              If you really want the data base in 1-day traces for processing, storage,
              etc. you can always make individual queries. Alternatively you could do
              the logic to split up multi-day segments on your end.

              Chad

              On Jul 14, 2014, at 11:35 AM, John D. West <john.d.west<at>asu.edu> wrote:

              Thanks, Philip! I didn't realize that. Since I am reading the returned
              file with ObsPy, presumably that it what is assembling the data into
              contiguous sections.

              -- John


              On Mon, Jul 14, 2014 at 11:18 AM, Philip Crotwell <crotwell<at>seis.sc.edu>
              wrote:

              Hi

              Just one thought that might help clarify things. There is no notion in
              miniseed of a "trace" or even a "large contiguous set of data". It is
              just a whole bunch of 512 or 4096 byte data records whose begin/end
              times might or might not line up to form contiguous data. Any "trace"
              that you are seeing is an artifact of whatever you are reading the
              miniseed with. The segmentation you are seeing is probably caused by
              gaps, and the trace is the longest contiguous section of data.

              If what you want is miniseed files that correspond 1:1 with the lines
              of the request, then it would most likely have to be some other
              service that returned maybe a tar flle containing miniseed files for
              example.

              $0.02
              Philip


              On Mon, Jul 14, 2014 at 1:40 PM, John D. West <john.d.west<at>asu.edu>
              wrote:
              Hi, Chad.

              Maybe "block" is the wrong term, sorry if it was confusing.

              I assemble a post request containing 100 lines. In general, each line
              represents one channel of broadband data for one station for one day,
              from
              00:00:00 to 23:59:59.9999.

              My expectation was to get back a miniseed file containing 100 one-day
              traces. Instead, contiguous traces are combined so I get some lesser
              number
              of traces, many of which are longer than one day. I am not in principle
              opposed to this behavior, but I can envision times when it may cause
              problems, e.g., if I have some processing that has limits on the length
              of
              the trace to which it can be applied. It seems to me that optimum
              behavior
              would be a flag set by the user controlling whether you combine traces
              or
              not.

              Example: A request containing the following POST data is asking for 100
              days
              of data. It returns 27 traces of varying length.

              POST text:
              quality=B
              TA G06A -- BHE 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
              TA G06A -- BHE 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
              TA G06A -- BHE 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
              TA G06A -- BHE 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
              TA G06A -- BHE 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
              TA G06A -- BHE 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
              TA G06A -- BHE 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
              TA G06A -- BHE 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
              TA F10A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
              TA F10A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
              TA F10A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
              TA F10A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
              TA F10A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
              TA F10A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
              TA F10A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
              TA F10A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
              TA F10A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
              TA F10A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
              TA F10A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
              TA F10A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
              TA F10A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
              TA G06A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
              TA G06A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
              TA G06A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
              TA G06A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
              TA G06A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
              TA G06A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
              TA G06A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
              TA G06A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
              TA G06A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
              TA G06A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
              TA G06A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
              TA G06A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
              TA G06A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
              TA G06A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
              TA G06A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
              TA G06A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
              TA G06A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
              TA G06A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
              TA G06A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
              TA G06A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
              TA F18A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
              TA F18A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
              TA F18A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
              TA F18A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
              TA F18A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
              TA F18A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
              TA F18A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
              TA F18A -- BHN 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
              TA F18A -- BHN 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
              TA F18A -- BHN 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
              TA F18A -- BHN 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
              TA F18A -- BHN 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
              TA F18A -- BHN 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
              TA F18A -- BHN 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
              TA F18A -- BHN 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
              TA F18A -- BHN 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
              TA F18A -- BHN 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
              TA F18A -- BHN 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
              TA F18A -- BHN 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
              TA F18A -- BHN 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
              TA G06A -- BHE 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
              TA G06A -- BHE 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
              TA G06A -- BHE 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
              TA G06A -- BHE 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
              TA G06A -- BHE 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
              TA G06A -- BHE 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
              TA G06A -- BHE 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
              TA G06A -- BHE 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
              TA G06A -- BHE 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
              TA G06A -- BHE 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
              TA G06A -- BHE 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
              TA G06A -- BHE 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
              TA G16A -- BHZ 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
              TA G16A -- BHZ 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
              TA G16A -- BHZ 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
              TA G16A -- BHZ 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
              TA G16A -- BHZ 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
              TA G16A -- BHZ 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
              TA G16A -- BHZ 2008-03-14T00:00:00 2008-03-14T23:59:59.9999
              TA G16A -- BHZ 2008-03-13T00:00:00 2008-03-13T23:59:59.9999
              TA G16A -- BHZ 2008-03-12T00:00:00 2008-03-12T23:59:59.9999
              TA G16A -- BHZ 2008-03-11T00:00:00 2008-03-11T23:59:59.9999
              TA G16A -- BHZ 2008-03-10T00:00:00 2008-03-10T23:59:59.9999
              TA G16A -- BHZ 2008-03-09T00:00:00 2008-03-09T23:59:59.9999
              TA G16A -- BHZ 2008-03-08T00:00:00 2008-03-08T23:59:59.9999
              TA G16A -- BHZ 2008-03-07T00:00:00 2008-03-07T23:59:59.9999
              TA G16A -- BHZ 2008-03-06T00:00:00 2008-03-06T23:59:59.9999
              TA G16A -- BHZ 2008-03-05T00:00:00 2008-03-05T23:59:59.9999
              TA G16A -- BHZ 2008-03-04T00:00:00 2008-03-04T23:59:59.9999
              TA G16A -- BHZ 2008-03-03T00:00:00 2008-03-03T23:59:59.9999
              TA G16A -- BHZ 2008-03-02T00:00:00 2008-03-02T23:59:59.9999
              TA G16A -- BHZ 2008-03-01T10:23:45 2008-03-01T23:59:59.9999
              TA F10A -- BHN 2008-03-20T00:00:00 2008-03-20T22:22:22.9999
              TA F10A -- BHN 2008-03-19T00:00:00 2008-03-19T23:59:59.9999
              TA F10A -- BHN 2008-03-18T00:00:00 2008-03-18T23:59:59.9999
              TA F10A -- BHN 2008-03-17T00:00:00 2008-03-17T23:59:59.9999
              TA F10A -- BHN 2008-03-16T00:00:00 2008-03-16T23:59:59.9999
              TA F10A -- BHN 2008-03-15T00:00:00 2008-03-15T23:59:59.9999
              TA F10A -- BHN 2008-03-14T00:00:00 2008-03-14T23:59:59.9999

              Thanks!

              -- John


              On Thu, Jul 10, 2014 at 7:08 PM, Chad Trabant <chad<at>iris.washington.edu

              wrote:


              Hi John,

              Can you provide a bit more detail about what you mean by getting the
              data
              back in blocks? In a nutshell, if the data is continuous in the
              archive,
              what you get should be continuous as long as you have requested
              continuous
              segments.

              In the archive, the internal blocking of the miniSEED data (referred
              to as
              records) are sometimes truncated at UTC day boundaries and other times
              there
              are records that cross the UTC day boundaries. For the TA data in
              particular, the R quality data that arrived in real time generally
              does not
              include records that cross the day boundaries (it has been cut to
              perfectly
              fit within days). But the Q quality data (a preferred copy) that
              arrives
              many months after real time does have records that cross day
              boundaries. On
              top of this, by default the dataselect service will trim the data
              requested
              to the time window specified by the user.

              I think if you can clarify what you are mean by blocking and perhaps
              provide some example requests we might be able to give you a better
              answer.

              Chad

              On Jun 30, 2014, at 4:34 PM, John D. West <john.d.west<at>asu.edu> wrote:

              Hi.

              I'm requesting continuous broadband data from (mostly) TA stations
              using
              the POST usage of the dataselect web service.

              When I request multiple successive one-day blocks in the same POST
              request, I sometimes get the data back in 1-day blocks and sometimes
              it is
              combined into longer multi-day blocks.

              What controls this behavior? Is it possible to force the data to be
              returned in single-day traces without resorting to sending multiple
              individual requests, one per day?

              Thanks!
              -- John
              _______________________________________________
              webservices mailing list
              webservices<at>iris.washington.edu
              http://www.iris.washington.edu/mailman/listinfo/webservices



              _______________________________________________
              webservices mailing list
              webservices<at>iris.washington.edu
              http://www.iris.washington.edu/mailman/listinfo/webservices



              _______________________________________________
              webservices mailing list
              webservices<at>iris.washington.edu
              http://www.iris.washington.edu/mailman/listinfo/webservices




Page built 16:48:57 | v.995c67d7