Thread: Unreadable file format for miniseed data from web services

Started: 2011-02-16 01:38:06
Last activity: 2011-02-16 17:45:39
Topics: Web Services
Alessia Maggi
Unreadable file format for miniseed data from web services
2011-02-16 01:38:06
Dear Webservices people,

I am having difficulty reading the miniseed files obtained through webservices. I have tried several methods of obtaining the data, from straight http requests, to the Perl client scripts, to wget requests on the bulkdataselect service...

Every time, the miniseed files are incorrectly formatted, so qmerge and sdrsplit refuse to work:

e.g.

alessia@makalu:~$ qmerge -nv IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
Error: unrecognizable input file IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed

alessia@makalu:~$ sdrsplit -C IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
unknown header type: -1

The files have an integer number of 4096 byte blocs, and seem to start with somewhat correct blocs (as far as I can tell using od -c)...

Do you have any ideas? Any help would be gratefully received!

Alessia

PS: I attach an example of a miniseed file obtained by clicking on the following url from the http://www.iris.edu/ws/dataselect/ webpage

http://www.iris.edu/ws/dataselect/query?net=IU&sta=ANMO&loc=00&cha=BHZ&start=2010-02-27T06:30:00.000&end=2010-02-27T10:30:00.000





Attachments
  • Chad Trabant
    Re: Unreadable file format for miniseed data from web services
    2011-02-15 22:57:03

    Dear Alessia,

    I collected the data using the URL request you provided and found nothing wrong with the miniseed.

    My suspicion is that qmerge and sdrsplit do not support M quality SEED data records. This type of data record is a documented part of the SEED standard. By default the ws-bulkdataselect service will return the "best" data which is a potential merge of other data qualities, this data is always marked with an M quality. It is also possible to request a specific quality (non-merged) from ws-bulkdataselect, but that is only recommended if you really know which copy of the data you want. Merging and returning M quality data is also the default for BREQ_FAST and other request mechanisms at the DMC.

    My suggestion is to contact the author of those programs (Doug Neuhauser) if you really need them for your processing. Alternatively you can describe your processing steps and we can suggest alternatives.

    regards,
    Chad

    On Feb 15, 2011, at 8:38 AM, Alessia Maggi wrote:

    Dear Webservices people,

    I am having difficulty reading the miniseed files obtained through webservices. I have tried several methods of obtaining the data, from straight http requests, to the Perl client scripts, to wget requests on the bulkdataselect service...

    Every time, the miniseed files are incorrectly formatted, so qmerge and sdrsplit refuse to work:

    e.g.

    alessia@makalu:~$ qmerge -nv IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
    Error: unrecognizable input file IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed

    alessia@makalu:~$ sdrsplit -C IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
    unknown header type: -1

    The files have an integer number of 4096 byte blocs, and seem to start with somewhat correct blocs (as far as I can tell using od -c)...

    Do you have any ideas? Any help would be gratefully received!

    Alessia

    PS: I attach an example of a miniseed file obtained by clicking on the following url from the http://www.iris.edu/ws/dataselect/ webpage

    http://www.iris.edu/ws/dataselect/query?net=IU&sta=ANMO&loc=00&cha=BHZ&start=2010-02-27T06:30:00.000&end=2010-02-27T10:30:00.000




    <IU-1.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed>_______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices



  • Philip Crotwell
    Re: Unreadable file format for miniseed data from web services
    2011-02-16 01:36:20
    Not sure this will help you, but I was able to read this miniseed file
    with the miniseed library in the seisFile package. That doesn't mean
    there isn't a problem, but might mean it is more like in qmerge or
    sdrsplit than in the miniseed itself.

    If you want to look, seisFile is downloadable from
    http://www.seis.sc.edu/seisFile.html

    Philip

    On Tue, Feb 15, 2011 at 11:38 AM, Alessia Maggi
    <alessia.maggi<at>unistra.fr> wrote:
    Dear Webservices people,

    I am having difficulty reading the miniseed files obtained through webservices.  I have tried several methods of obtaining the data, from straight http requests, to the Perl client scripts, to wget requests on the bulkdataselect service...

    Every time, the miniseed files are incorrectly formatted, so qmerge and sdrsplit refuse to work:

    e.g.

    alessia@makalu:~$ qmerge -nv IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
    Error: unrecognizable input file IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed

    alessia@makalu:~$ sdrsplit -C IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
    unknown header type: -1

    The files have an integer number of 4096 byte blocs, and seem to start with somewhat correct blocs (as far as I can tell using od -c)...

    Do you have any ideas?  Any help would be gratefully received!

    Alessia

    PS: I attach an example of a miniseed file obtained by clicking on the following url from the http://www.iris.edu/ws/dataselect/ webpage

    http://www.iris.edu/ws/dataselect/query?net=IU&sta=ANMO&loc=00&cha=BHZ&start=2010-02-27T06:30:00.000&end=2010-02-27T10:30:00.000





    _______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices




  • Bruce Weertman
    Re: Unreadable file format for miniseed data from web services
    2011-02-16 04:10:41
    Alessia:

    Could this be a problem with the merged data quality indicator in the miniseed. It defaults to "M" in ws-dataselect and ws-bulkdataselect.

    Have you tried using "Q" or "R" or "D" which have been around for longer?

    For example:
    Try data from this query:
    http://www.iris.edu/ws/dataselect/query?net=IU&sta=ANMO&loc=00&cha=BHZ&start=2010-02-27T06.30.00&end=2010-02-27T10.30.00&quality=Q&ref=direct

    If that works, then the problem is mostly likely with incompatibility in qmerge - it can't handle the newer values.

    BTW:
    Here's an undocumented trick that you can do with the dataselect webservice
    set ref=token instead of direct, get the token back and then pass it to the undocumented ws-msi webservice
    (msi = miniseed inspector)

    For example here's a sequence:

    http://www.iris.edu/ws/dataselect/query?net=IU&sta=ANMO&loc=00&cha=BHZ&start=2010-02-27T06.30.00&end=2010-02-27T10.30.00&quality=Q&ref=token
    this returns:
    dataselect.1032b7cfd25a64e4d8d15cf019d1d056

    Then pass that to msi like this:

    http://www.iris.edu/ws/msi/query?token=dataselect.1032b7cfd25a64e4d8d15cf019d1d056

    you can set the information level to 0, 1 or 2 like this:
    http://www.iris.edu/ws/msi/query?token=dataselect.1032b7cfd25a64e4d8d15cf019d1d056&level=2

    Cheers,
    -Bruce


    On Feb 15, 2011, at 8:38 AM, Alessia Maggi wrote:

    Dear Webservices people,

    I am having difficulty reading the miniseed files obtained through webservices. I have tried several methods of obtaining the data, from straight http requests, to the Perl client scripts, to wget requests on the bulkdataselect service...

    Every time, the miniseed files are incorrectly formatted, so qmerge and sdrsplit refuse to work:

    e.g.

    alessia@makalu:~$ qmerge -nv IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
    Error: unrecognizable input file IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed

    alessia@makalu:~$ sdrsplit -C IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
    unknown header type: -1

    The files have an integer number of 4096 byte blocs, and seem to start with somewhat correct blocs (as far as I can tell using od -c)...

    Do you have any ideas? Any help would be gratefully received!

    Alessia

    PS: I attach an example of a miniseed file obtained by clicking on the following url from the http://www.iris.edu/ws/dataselect/ webpage

    http://www.iris.edu/ws/dataselect/query?net=IU&sta=ANMO&loc=00&cha=BHZ&start=2010-02-27T06:30:00.000&end=2010-02-27T10:30:00.000




    <IU-1.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed>_______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices



  • Doug Neuhauser
    Re: Unreadable file format for miniseed data from web services
    2011-02-16 05:59:26
    I believe the problem is a non-current version of qmerge
    and sdrsplit. I'll update the ftp site tomorrow and send out
    the URL of the current version.

    Thanks for the gentle reminder!

    - Doug N

    On 2/15/11 8:38 AM, Alessia Maggi wrote:
    Dear Webservices people,

    I am having difficulty reading the miniseed files obtained through webservices. I have tried several methods of obtaining the data, from straight http requests, to the Perl client scripts, to wget requests on the bulkdataselect service...

    Every time, the miniseed files are incorrectly formatted, so qmerge and sdrsplit refuse to work:

    e.g.

    alessia@makalu:~$ qmerge -nv IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
    Error: unrecognizable input file IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed

    alessia@makalu:~$ sdrsplit -C IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
    unknown header type: -1

    The files have an integer number of 4096 byte blocs, and seem to start with somewhat correct blocs (as far as I can tell using od -c)...

    Do you have any ideas? Any help would be gratefully received!

    Alessia

    PS: I attach an example of a miniseed file obtained by clicking on the following url from the http://www.iris.edu/ws/dataselect/ webpage

    http://www.iris.edu/ws/dataselect/query?net=IU&sta=ANMO&loc=00&cha=BHZ&start=2010-02-27T06:30:00.000&end=2010-02-27T10:30:00.000






    _______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices


    --
    Doug Neuhauser University of California, Berkeley
    doug<at>seismo.berkeley.edu Berkeley Seismological Laboratory
    Office: 510-642-0931 215 McCone Hall # 4760
    Fax: 510-643-5811 Berkeley, CA 94720-4760
    Remote: 530-752-5615 (Wed,Fri)

  • Doug Neuhauser
    Re: Unreadable file format for miniseed data from web services
    2011-02-16 06:35:58
    I've posted current version of
    qmerge
    sdrsplit
    qlib2
    to:
    ftp://www.ncedc.org/pub/quanterra/ucb/qmerge.2011.046.tar.gz
    ftp://www.ncedc.org/pub/quanterra/ucb/sdrsplit.2011.046.tar.gz
    ftp://www.ncedc.org/pub/quanterra/ucb/qlib2.2011.046.tar.gz

    Let me know if this does not address your problems.

    - Doug N

    On 2/15/11 8:38 AM, Alessia Maggi wrote:
    Dear Webservices people,

    I am having difficulty reading the miniseed files obtained through webservices. I have tried several methods of obtaining the data, from straight http requests, to the Perl client scripts, to wget requests on the bulkdataselect service...

    Every time, the miniseed files are incorrectly formatted, so qmerge and sdrsplit refuse to work:

    e.g.

    alessia@makalu:~$ qmerge -nv IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
    Error: unrecognizable input file IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed

    alessia@makalu:~$ sdrsplit -C IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
    unknown header type: -1

    The files have an integer number of 4096 byte blocs, and seem to start with somewhat correct blocs (as far as I can tell using od -c)...

    Do you have any ideas? Any help would be gratefully received!

    Alessia

    PS: I attach an example of a miniseed file obtained by clicking on the following url from the http://www.iris.edu/ws/dataselect/ webpage

    http://www.iris.edu/ws/dataselect/query?net=IU&sta=ANMO&loc=00&cha=BHZ&start=2010-02-27T06:30:00.000&end=2010-02-27T10:30:00.000






    _______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices


    --
    Doug Neuhauser University of California, Berkeley
    doug<at>seismo.berkeley.edu Berkeley Seismological Laboratory
    Office: 510-642-0931 215 McCone Hall # 4760
    Fax: 510-643-5811 Berkeley, CA 94720-4760
    Remote: 530-752-5615 (Wed,Fri)

    • Alessia Maggi
      Re: Unreadable file format for miniseed data from web services
      2011-02-16 17:45:39
      Dear all,

      Thank you very much for your rapid replies!

      Doug : I have downloaded and tested your updated versions of qmerge / sdrsplit and qlib2, and they work just fine now. Getting ms2sac to work just required recompiling it with the new qlib2 library, and was done in a matter of seconds. Thank you very mmuch!

      All my current automated routines for data analysis are based on the qug tools, but thank you to those who suggested using the seisFile package, I shall look into it.

      Sincerely yours!
      Alessia


      Le 16 févr. 2011 à 07:35, Doug Neuhauser a écrit :

      I've posted current version of
      qmerge
      sdrsplit
      qlib2
      to:
      ftp://www.ncedc.org/pub/quanterra/ucb/qmerge.2011.046.tar.gz
      ftp://www.ncedc.org/pub/quanterra/ucb/sdrsplit.2011.046.tar.gz
      ftp://www.ncedc.org/pub/quanterra/ucb/qlib2.2011.046.tar.gz

      Let me know if this does not address your problems.

      - Doug N

      On 2/15/11 8:38 AM, Alessia Maggi wrote:
      Dear Webservices people,

      I am having difficulty reading the miniseed files obtained through webservices. I have tried several methods of obtaining the data, from straight http requests, to the Perl client scripts, to wget requests on the bulkdataselect service...

      Every time, the miniseed files are incorrectly formatted, so qmerge and sdrsplit refuse to work:

      e.g.

      alessia@makalu:~$ qmerge -nv IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
      Error: unrecognizable input file IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed

      alessia@makalu:~$ sdrsplit -C IU.ANMO.00.BHZ.2010.058.06.30.00.000-2010.058.10.30.00.000.miniseed
      unknown header type: -1

      The files have an integer number of 4096 byte blocs, and seem to start with somewhat correct blocs (as far as I can tell using od -c)...

      Do you have any ideas? Any help would be gratefully received!

      Alessia

      PS: I attach an example of a miniseed file obtained by clicking on the following url from the http://www.iris.edu/ws/dataselect/ webpage

      http://www.iris.edu/ws/dataselect/query?net=IU&sta=ANMO&loc=00&cha=BHZ&start=2010-02-27T06:30:00.000&end=2010-02-27T10:30:00.000






      _______________________________________________
      webservices mailing list
      webservices<at>iris.washington.edu
      http://www.iris.washington.edu/mailman/listinfo/webservices


      --
      Doug Neuhauser University of California, Berkeley
      doug<at>seismo.berkeley.edu Berkeley Seismological Laboratory
      Office: 510-642-0931 215 McCone Hall # 4760
      Fax: 510-643-5811 Berkeley, CA 94720-4760
      Remote: 530-752-5615 (Wed,Fri)



NSF SAGE Facility has been certified as a trustworthy data repository by the CoreTrustSeal Standards and Certification Board
19:56:25 v.3514fbed