Thread: redundant inventory requests

Started: 2016-10-27 18:22:23
Last activity: 2017-08-23 21:24:50
Topics: Web Services
lisa
2016-10-27 18:22:23
Hi.

I'm chasing down issues setting up to download and process a heap of data in python, using obspy. I've done the following test on various os, using python 2.7, with obspy 1.0.2 and 0.10.2 but all from the same ip route, which is behind what is probably a heinous campus firewall complex.

The simple test:

from obspy.core import UTCDateTime
from obspy.fdsn.client import Client #obspy 0.10
client = Client('IRIS')
t = UTCDateTime(2009,1,1,10)
inv = client.get_stations(network = 'TA',station = 'E24A',starttime = t,endtime =t+400)
print inv

I expect to see the inventory list, and sometimes I do:

Inventory created at 2016-10-27T17:42:15.000000Z
Created by: IRIS WEB SERVICE: fdsnws-station | version: 1.1.20
http://service.iris.edu/fdsnws/station/1/query?starttime=2009-01-01...
Sending institution: IRIS-DMC (IRIS-DMC)
Contains:
Networks (1):
TA
Stations (1):
TA.E24A (Baker, MT, USA)
Channels (0):

but sometimes I get this:

Traceback (most recent call last):
File "testping.py", line 6, in <module>
inv = client.get_stations(network = 'TA',station = 'E24A',starttime = t,endtime =t+400)
File "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/python2.7/site-packages/obspy-0.10.2-py2.7-linux-x86_64.egg/obspy/fdsn/client.py", line 471, in get_stations
raise ValueError(msg)
ValueError: The current client does not have a station service.


Sometimes I can request inv over and over and over. More often, I can request it once and then it does not work again for an inconsistent amount of time. If I alter the requests so that they are not identical and submit them back to back, it seems to make no difference, meaning if I'm being prevented from making redundant requests- the redundancy is determined by my incoming ip or identifier, not by the request parameters. It is also the case that the first request isn't necessarily the successful one. Can anyone shed light on this?

Cheers,

Lisa Linville
Ph.D. Student Geophysics
University of Utah
115 South 1460 East
Salt Lake City, Utah 84112
  • Lion Krischer
    2016-10-28 03:39:17
    Hi Lisa,

    this question might be better suited for the ObsPy mailing list but here
    you go.


    I'm chasing down issues setting up to download and process a heap of
    data in python, using obspy. I've done the following test on various
    os, using python 2.7, with obspy 1.0.2 and 0.10.2 but all from the
    same ip route, which is behind what is probably a heinous campus
    firewall complex.

    The FDSN webservices go over HTTP and usually port 80 so if you can
    browse the web you should be able to download data just fine. ObsPy
    should also pick up a potential proxy.

    Traceback (most recent call last): File "testping.py", line 6, in
    <module> inv = client.get_stations(network = 'TA',station =
    'E24A',starttime = t,endtime =t+400) File
    "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/python2.7/site-packages/obspy-0.10.2-py2.7-linux-x86_64.egg/obspy/fdsn/client.py",
    line 471, in get_stations raise ValueError(msg) ValueError: The
    current client does not have a station service.


    At Client initialization time ObsPy checks what services a particular
    fdsn-ws implementation offers by parsing their application.wadl files.
    What you describe can happen for example if either the service is down
    (but the IRIS implementation tends to be very stable) or if you have a
    flaky internet connection.


    Sometimes I can request inv over and over and over. More often, I can
    request it once and then it does not work again for an inconsistent
    amount of time. If I alter the requests so that they are not
    identical and submit them back to back, it seems to make no
    difference, meaning if I'm being prevented from making redundant
    requests- the redundancy is determined by my incoming ip or
    identifier, not by the request parameters. It is also the case that
    the first request isn't necessarily the successful one. Can anyone
    shed light on this?

    That does sound weird. Does your institution maybe have some kind of
    proxy that buffers requests and/or rate limits users? It might misbehave
    - but this is just a wild guess.

    In any case: can you please also update your ObsPy implementation to the
    latest version to make sure its not some bug that has been fixed in the
    meanwhile.


    Cheers!

    Lion

    • Lisa Linville
      2016-10-27 20:13:12
      Thank you for the ideas and info. I'll try the Obspy boards.
      To be more clear, I'm up to date on obspy but some of the servers I checked
      from are not. I did this test on both new and old with the same results.
      Both are reporting the same version of FDSN 'station' services (v1.1.22)

      Appreciate it!

      On Thu, Oct 27, 2016 at 12:40 PM, Lion Krischer <
      krischer<at>geophysik.uni-muenchen.de> wrote:

      Hi Lisa,

      this question might be better suited for the ObsPy mailing list but here
      you go.


      I'm chasing down issues setting up to download and process a heap of
      data in python, using obspy. I've done the following test on various
      os, using python 2.7, with obspy 1.0.2 and 0.10.2 but all from the
      same ip route, which is behind what is probably a heinous campus
      firewall complex.

      The FDSN webservices go over HTTP and usually port 80 so if you can
      browse the web you should be able to download data just fine. ObsPy
      should also pick up a potential proxy.

      Traceback (most recent call last): File "testping.py", line 6, in
      <module> inv = client.get_stations(network = 'TA',station =
      'E24A',starttime = t,endtime =t+400) File
      "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/
      python2.7/site-packages/obspy-0.10.2-py2.7-linux-x86_64.egg/
      obspy/fdsn/client.py",
      line 471, in get_stations raise ValueError(msg) ValueError: The
      current client does not have a station service.


      At Client initialization time ObsPy checks what services a particular
      fdsn-ws implementation offers by parsing their application.wadl files.
      What you describe can happen for example if either the service is down
      (but the IRIS implementation tends to be very stable) or if you have a
      flaky internet connection.


      Sometimes I can request inv over and over and over. More often, I can
      request it once and then it does not work again for an inconsistent
      amount of time. If I alter the requests so that they are not
      identical and submit them back to back, it seems to make no
      difference, meaning if I'm being prevented from making redundant
      requests- the redundancy is determined by my incoming ip or
      identifier, not by the request parameters. It is also the case that
      the first request isn't necessarily the successful one. Can anyone
      shed light on this?

      That does sound weird. Does your institution maybe have some kind of
      proxy that buffers requests and/or rate limits users? It might misbehave
      - but this is just a wild guess.

      In any case: can you please also update your ObsPy implementation to the
      latest version to make sure its not some bug that has been fixed in the
      meanwhile.


      Cheers!

      Lion

      ----------------------
      Web Services (http://ds.iris.edu/message-center/topic/webservices/)

      Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
      Update subscription preferences at http://ds.iris.edu/account/profile/


      • Mark Panning
        2017-08-22 17:31:58
        Hi Lisa,

        Sorry to hijack an old thread, but I'm getting similar errors right now for some scripts I've been running that worked until very recently, and was wondering if you had resolved this issue in some repeatable fashion.
  • Alex Hutko
    2017-08-23 21:24:50
    Mark,

    This may be related to limits with how fast you're connecting. If you have a fast connection, e.g. from across the street at UW like me, then running into limits happens often when you're making many requests either with your script or with wildcards. This happens with small items like inventories, a station lat, or SACPZ files rather than waveforms.

    If you exceed the max connections per second (of order 50-100?), then your IP gets put in the penalty box for 20? seconds. To address this, the latest IRIS Fetch scripts have 50ms delays between requests built in. I use 25ms delays in my python scripts which seems to be sufficient.

    If this isn't your problem, sorry for adding to the noise.

    Alex


    On Oct 27, 2016, at 11:22 AM, lisa <linville<at>seis.utah.edu> wrote:

    Hi.

    I'm chasing down issues setting up to download and process a heap of data in python, using obspy. I've done the following test on various os, using python 2.7, with obspy 1.0.2 and 0.10.2 but all from the same ip route, which is behind what is probably a heinous campus firewall complex.

    The simple test:

    from obspy.core import UTCDateTime
    from obspy.fdsn.client import Client #obspy 0.10
    client = Client('IRIS')
    t = UTCDateTime(2009,1,1,10)
    inv = client.get_stations(network = 'TA',station = 'E24A',starttime = t,endtime =t+400)
    print inv

    I expect to see the inventory list, and sometimes I do:

    Inventory created at 2016-10-27T17:42:15.000000Z
    Created by: IRIS WEB SERVICE: fdsnws-station | version: 1.1.20
    http://service.iris.edu/fdsnws/station/1/query?starttime=2009-01-01...
    Sending institution: IRIS-DMC (IRIS-DMC)
    Contains:
    Networks (1):
    TA
    Stations (1):
    TA.E24A (Baker, MT, USA)
    Channels (0):

    but sometimes I get this:

    Traceback (most recent call last):
    File "testping.py", line 6, in <module>
    inv = client.get_stations(network = 'TA',station = 'E24A',starttime = t,endtime =t+400)
    File "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/python2.7/site-packages/obspy-0.10.2-py2.7-linux-x86_64.egg/obspy/fdsn/client.py", line 471, in get_stations
    raise ValueError(msg)
    ValueError: The current client does not have a station service.


    Sometimes I can request inv over and over and over. More often, I can request it once and then it does not work again for an inconsistent amount of time. If I alter the requests so that they are not identical and submit them back to back, it seems to make no difference, meaning if I'm being prevented from making redundant requests- the redundancy is determined by my incoming ip or identifier, not by the request parameters. It is also the case that the first request isn't necessarily the successful one. Can anyone shed light on this?

    Cheers,

    Lisa Linville
    Ph.D. Student Geophysics
    University of Utah
    115 South 1460 East
    Salt Lake City, Utah 84112

    ----------------------
    Web Services (http://ds.iris.edu/message-center/topic/webservices/)

    Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
    Update subscription preferences at http://ds.iris.edu/account/profile/


Page built 08:11:31 | v.e7d5ec69