Thread: Federator question

Started: 2016-09-16 17:13:45
Last activity: 2016-09-17 00:04:52
Topics: Web Services
Keith Koper
Federator question
2016-09-16 17:13:45
Hi,

I've never been able to get the Federator (-F) option to work using
FetchData. Here is a little script that works fine when I don't use the -F
but when I include the -F, i=I get the following:






*Received 910 Bytes from federator catalog in 1.3 seconds (705
Bytes/s)Fetching data from GEOFON (http://geofon.gfz-potsdam.de
http://geofon.gfz-potsdam.de)Received 1.8 KB of metadata in 1.0 seconds
(1.8 KB/s)Processed metadata for 195 channel epochs in 2.1 secondsCannot
open output file 'GEOFON-./foo.mseed': No such file or directory*
and I don't get any data. Here is the script:

#!/bin/bash

elat=-4.952
elon=94.330
dmin=0.0
dmax=10.0
min_dur=3550
dur=3600
stime=2016-03-02,12:49:48
etime=2016-03-02,13:49:48

ddir=.
ofile=foo.mseed

fetch=/Users/keithkoper/Research_03/Wharton_xcorr/util/FetchData-2015.246

${fetch} -radius ${elat}:${elon}:${dmax}:${dmin} -s $stime -e $etime -F\
-chan "BH?,HH?,EH?,SH?" -msl $min_dur -o $ddir/$ofile -sd $ddir -m
$ddir/meta.data

Thanks,
-Keith

--
Keith D. Koper, PhD
Professor in Dept. of Geology & Geophysics
Director of Seismograph Stations
University of Utah

Follow Univ. Utah Seismograph Stations on Twitter
https://twitter.com/UUSS_Quake_Info

  • Rob Porritt
    Re: Federator question
    2016-09-17 00:04:52
    Hi Keith,

    Have you tried changing the -o to -O and -m to -M? I think when it uses the
    Federator it needs the capital letters to merge from multiple data centers.

    On Fri, Sep 16, 2016 at 9:45 AM Keith Koper <kkoper<at>gmail.com> wrote:

    Hi,

    I've never been able to get the Federator (-F) option to work using
    FetchData. Here is a little script that works fine when I don't use the -F
    but when I include the -F, i=I get the following:






    *Received 910 Bytes from federator catalog in 1.3 seconds (705
    Bytes/s)Fetching data from GEOFON (http://geofon.gfz-potsdam.de
    http://geofon.gfz-potsdam.de)Received 1.8 KB of metadata in 1.0 seconds
    (1.8 KB/s)Processed metadata for 195 channel epochs in 2.1 secondsCannot
    open output file 'GEOFON-./foo.mseed': No such file or directory*
    and I don't get any data. Here is the script:

    #!/bin/bash

    elat=-4.952
    elon=94.330
    dmin=0.0
    dmax=10.0
    min_dur=3550
    dur=3600
    stime=2016-03-02,12:49:48
    etime=2016-03-02,13:49:48

    ddir=.
    ofile=foo.mseed

    fetch=/Users/keithkoper/Research_03/Wharton_xcorr/util/FetchData-2015.246

    ${fetch} -radius ${elat}:${elon}:${dmax}:${dmin} -s $stime -e $etime -F\
    -chan "BH?,HH?,EH?,SH?" -msl $min_dur -o $ddir/$ofile -sd $ddir -m
    $ddir/meta.data

    Thanks,
    -Keith

    --
    Keith D. Koper, PhD
    Professor in Dept. of Geology & Geophysics
    Director of Seismograph Stations
    University of Utah

    Follow Univ. Utah Seismograph Stations on Twitter
    https://twitter.com/UUSS_Quake_Info

    ----------------------
    Web Services (http://ds.iris.edu/message-center/topic/webservices/)

    Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
    Update subscription preferences at http://ds.iris.edu/account/profile/


    • Keith Koper
      Re: Federator question
      2016-09-16 18:17:41
      OK, I just tried that and I did get a different kind of error message:

      Received 1.6 KB from federator catalog in 1.3 seconds (1.2 KB/s)























      *Fetching data from GEOFON (http://geofon.gfz-potsdam.de
      http://geofon.gfz-potsdam.de)Received 3.2 KB of metadata in 2.1 seconds
      (1.5 KB/s)Processed metadata for 363 channel epochs in 5.4 secondsError
      fetching time series data: 503 :: Service Unavailable------Error 503:
      Service Unavailableenforcing of minimum record length not supportedUsage
      details are available from
      /fdsnws/dataselect/1/Request:/fdsnws/dataselect/1/queryRequest
      Submitted:2016-09-16T17:14:46.333708Service Version:1.1.0------Retrying
      request in 10 secondsError fetching time series data: 503 :: Service
      Unavailable*
      And then it gets stuck in a loop and keeps repeating the error message.

      Thanks,
      -Keith


      On Fri, Sep 16, 2016 at 11:04 AM, Rob Porritt <porritt<at>email.arizona.edu>
      wrote:

      Hi Keith,

      Have you tried changing the -o to -O and -m to -M? I think when it uses
      the Federator it needs the capital letters to merge from multiple data
      centers.

      On Fri, Sep 16, 2016 at 9:45 AM Keith Koper <kkoper<at>gmail.com> wrote:

      Hi,

      I've never been able to get the Federator (-F) option to work using
      FetchData. Here is a little script that works fine when I don't use the -F
      but when I include the -F, i=I get the following:






      *Received 910 Bytes from federator catalog in 1.3 seconds (705
      Bytes/s)Fetching data from GEOFON (http://geofon.gfz-potsdam.de
      http://geofon.gfz-potsdam.de)Received 1.8 KB of metadata in 1.0 seconds
      (1.8 KB/s)Processed metadata for 195 channel epochs in 2.1 secondsCannot
      open output file 'GEOFON-./foo.mseed': No such file or directory*
      and I don't get any data. Here is the script:

      #!/bin/bash

      elat=-4.952
      elon=94.330
      dmin=0.0
      dmax=10.0
      min_dur=3550
      dur=3600
      stime=2016-03-02,12:49:48
      etime=2016-03-02,13:49:48

      ddir=.
      ofile=foo.mseed

      fetch=/Users/keithkoper/Research_03/Wharton_xcorr/util/FetchData-2015.246

      ${fetch} -radius ${elat}:${elon}:${dmax}:${dmin} -s $stime -e $etime -F\
      -chan "BH?,HH?,EH?,SH?" -msl $min_dur -o $ddir/$ofile -sd $ddir
      -m $ddir/meta.data

      Thanks,
      -Keith

      --
      Keith D. Koper, PhD
      Professor in Dept. of Geology & Geophysics
      Director of Seismograph Stations
      University of Utah

      Follow Univ. Utah Seismograph Stations on Twitter
      https://twitter.com/UUSS_Quake_Info

      ----------------------
      Web Services (http://ds.iris.edu/message-center/topic/webservices/)

      Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
      Update subscription preferences at http://ds.iris.edu/account/profile/




      --
      Keith D. Koper, PhD
      Professor in Dept. of Geology & Geophysics
      Director of Seismograph Stations
      University of Utah

      Follow Univ. Utah Seismograph Stations on Twitter
      https://twitter.com/UUSS_Quake_Info

      • Chad Trabant
        Re: Federator question
        2016-09-16 18:59:36

        Hi Keith,

        There are a small handful of things going on. To address some of them there is a new version of FetchData (2016.260) available here:
        https://seiscode.iris.washington.edu/projects/ws-fetch-scripts/files

        Direct link:
        https://seiscode.iris.washington.edu/attachments/download/1226/FetchData-2016.260

        With the latest version if you take out the -msl (minimum segment length) and -sd (SAC PZ directory) options in your calls the requests should complete.

        More details below:

        Rob's suggestion to use the -O and -M options will avoid the first problem you were seeing, you will not be able to see which data came from which data center though.

        * When federating, FetchData adds a data center identifier (e.g. IRISDMC, GEOFON) to the output files when using -o and -m. That didn't work well with the path you gave. This is fixed in the latest version.

        * Most data centers other than the DMC do not support the -msl (minimum segment length) and -lso (longest segment only) options. A warning has been added to FetchData when you use these options while federating. You probably shouldn't use them when federating, I don't think any center except the DMC supports them.

        * When there are problems with a request FetchData will retry, when it makes sense, up to 10 times, pausing 10 seconds between. With your version of the script it retried 60 times, so it looks like an endless loop but would eventually give up. The latest version will only retry 2 times when federating to reduce the wasted time.

        There is also a -nr (no retry) option, but it will cause the script to completely give up on any sort of errors and it would not continue on in the list of data centers.

        * Requesting SAC Poles and Zeros is only supported by the IRISDMC and NCEDC centers, other data centers do not have a SACPZ service. FetchData will print a warning that it cannot get SACPZ, but it might leave you wondering where the others went.

        cheers,
        Chad

        On Sep 16, 2016, at 10:53 AM, Keith Koper <kkoper<at>gmail.com> wrote:

        OK, I just tried that and I did get a different kind of error message:

        Received 1.6 KB from federator catalog in 1.3 seconds (1.2 KB/s)
        Fetching data from GEOFON (http://geofon.gfz-potsdam.de http://geofon.gfz-potsdam.de/)
        Received 3.2 KB of metadata in 2.1 seconds (1.5 KB/s)
        Processed metadata for 363 channel epochs in 5.4 seconds
        Error fetching time series data: 503 :: Service Unavailable
        ------
        Error 503: Service Unavailable

        enforcing of minimum record length not supported

        Usage details are available from /fdsnws/dataselect/1/

        Request:
        /fdsnws/dataselect/1/query

        Request Submitted:
        2016-09-16T17:14:46.333708

        Service Version:
        1.1.0

        ------
        Retrying request in 10 seconds
        Error fetching time series data: 503 :: Service Unavailable

        And then it gets stuck in a loop and keeps repeating the error message.

        Thanks,
        -Keith


        On Fri, Sep 16, 2016 at 11:04 AM, Rob Porritt <porritt<at>email.arizona.edu <porritt<at>email.arizona.edu>>wrote:
        Hi Keith,

        Have you tried changing the -o to -O and -m to -M? I think when it uses the Federator it needs the capital letters to merge from multiple data centers.

        On Fri, Sep 16, 2016 at 9:45 AM Keith Koper <kkoper<at>gmail.com <kkoper<at>gmail.com>> wrote:
        Hi,

        I've never been able to get the Federator (-F) option to work using FetchData. Here is a little script that works fine when I don't use the -F but when I include the -F, i=I get the following:

        Received 910 Bytes from federator catalog in 1.3 seconds (705 Bytes/s)
        Fetching data from GEOFON (http://geofon.gfz-potsdam.de http://geofon.gfz-potsdam.de/)
        Received 1.8 KB of metadata in 1.0 seconds (1.8 KB/s)
        Processed metadata for 195 channel epochs in 2.1 seconds
        Cannot open output file 'GEOFON-./foo.mseed': No such file or directory

        and I don't get any data. Here is the script:

        #!/bin/bash

        elat=-4.952
        elon=94.330
        dmin=0.0
        dmax=10.0
        min_dur=3550
        dur=3600
        stime=2016-03-02,12:49:48
        etime=2016-03-02,13:49:48

        ddir=.
        ofile=foo.mseed

        fetch=/Users/keithkoper/Research_03/Wharton_xcorr/util/FetchData-2015.246

        ${fetch} -radius ${elat}:${elon}:${dmax}:${dmin} -s $stime -e $etime -F\
        -chan "BH?,HH?,EH?,SH?" -msl $min_dur -o $ddir/$ofile -sd $ddir -m $ddir/meta.data

        Thanks,
        -Keith

        --
        Keith D. Koper, PhD
        Professor in Dept. of Geology & Geophysics
        Director of Seismograph Stations
        University of Utah

        Follow Univ. Utah Seismograph Stations on Twitter https://twitter.com/UUSS_Quake_Info
        <>
        ----------------------
        Web Services (http://ds.iris.edu/message-center/topic/webservices/)

        Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
        Update subscription preferences at http://ds.iris.edu/account/profile/



        --
        Keith D. Koper, PhD
        Professor in Dept. of Geology & Geophysics
        Director of Seismograph Stations
        University of Utah

        Follow Univ. Utah Seismograph Stations on Twitter https://twitter.com/UUSS_Quake_Info
        <>
        ----------------------
        Web Services (http://ds.iris.edu/message-center/topic/webservices/)

        Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
        Update subscription preferences at http://ds.iris.edu/account/profile/

NSF SAGE Facility has been certified as a trustworthy data repository by the CoreTrustSeal Standards and Certification Board
21:07:51 v.b4412d20