Thread: More Stations service problems

Started: 2023-01-28 18:46:36
Last activity: 2023-02-06 13:44:48
Topics: Web Services
John West
2023-01-28 18:46:36
Hi! I'm using Obspy to call the stations web service to get BH* and HH* channels for a given network and 2-hour time window. Code snippet:
Channels = DCclient.get_stations(startbefore=FileStart, endafter=FileEnd,
network=N.code, channel=CHANNELS, level='channel')

I get the following error:
could not retrieve channels for datacenter IRIS: KY Not a gzipped file (b'a\r') Traceback (most recent call last): File "/home/adept/code/Channels.py", line 213, in FDSNchannels Channels = DCclient.get_stations(startbefore=FileStart, endafter=FileEnd, File "/usr/local/lib/python3.8/dist-packages/obspy/clients/fdsn/client.py", line 725, in get_stations data_stream = self._download(url) File "/usr/local/lib/python3.8/dist-packages/obspy/clients/fdsn/client.py", line 1393, in _download code, data = download_url( File "/usr/local/lib/python3.8/dist-packages/obspy/clients/fdsn/client.py", line 1819, in download_url data = io.BytesIO(f.read()) File "/usr/lib/python3.8/gzip.py", line 292, in read return self._buffer.read(size) File "/usr/lib/python3.8/gzip.py", line 479, in read if not self._read_gzip_header(): File "/usr/lib/python3.8/gzip.py", line 427, in _read_gzip_header raise BadGzipFile('Not a gzipped file (%r)' % magic) gzip.BadGzipFile: Not a gzipped file (b'a\r')

I don't know if this is related to the issue reported by Philip Crotwell last week. The code works with other data centers, and used to work with the IRIS service.

Thanks!
-- John
  • Rob Casey
    2023-01-30 16:19:11

    Hi John-

    KY has little in the way of BH and HH channels, which you can see in this query:

    http://service.iris.edu/fdsnws/station/1/query?network=KY&format=text&level=channel

    In fact, if you apply a channel filter to look for just those types, you will only find HH channels for CUSSO, so that will be all you can hope to extract. We can look at the time extent for data availability here:

    http://service.iris.edu/fdsnws/availability/1/extent?network=KY&format=text&cha=HH* http://service.iris.edu/fdsnws/availability/1/extent?network=KY&format=text&cha=HH*>

    Aside from only seeing three channels, you should be able to get data from them. We have still not sorted out why we get this error via ObsPy, but we think that something built into the ObsPy client is expecting a Gzipped return and getting something else instead. This could be a mis-marked content type header from the sending service. It's possible that what is being sent back is some kind of error message, but I don't recall that we have that content figured out.

    We have seen this problem before with some of our internal routines, and it specifically points to ObsPy and its current interaction with our station service. It's still on our problem list.

    Have you tried different time lengths with this specific query, or narrowed it to individual channels? Sorry that I don't have a definitive answer for this one. We'll try to solve it when we can spare people cycles.

    -Rob


    On Jan 28, 2023, at 10:46 AM, John West (via IRIS) <webservices-bounce<at>lists.ds.iris.edu> wrote:

    Hi! I'm using Obspy to call the stations web service to get BH* and HH* channels for a given network and 2-hour time window. Code snippet:
    Channels = DCclient.get_stations(startbefore=FileStart, endafter=FileEnd,
    network=N.code, channel=CHANNELS, level='channel')

    I get the following error:
    could not retrieve channels for datacenter IRIS: KY Not a gzipped file (b'a\r') Traceback (most recent call last): File "/home/adept/code/Channels.py", line 213, in FDSNchannels Channels = DCclient.get_stations(startbefore=FileStart, endafter=FileEnd, File "/usr/local/lib/python3.8/dist-packages/obspy/clients/fdsn/client.py", line 725, in get_stations data_stream = self._download(url) File "/usr/local/lib/python3.8/dist-packages/obspy/clients/fdsn/client.py", line 1393, in _download code, data = download_url( File "/usr/local/lib/python3.8/dist-packages/obspy/clients/fdsn/client.py", line 1819, in download_url data = io.BytesIO(f.read()) File "/usr/lib/python3.8/gzip.py", line 292, in read return self._buffer.read(size) File "/usr/lib/python3.8/gzip.py", line 479, in read if not self._read_gzip_header(): File "/usr/lib/python3.8/gzip.py", line 427, in _read_gzip_header raise BadGzipFile('Not a gzipped file (%r)' % magic) gzip.BadGzipFile: Not a gzipped file (b'a\r')

    I don't know if this is related to the issue reported by Philip Crotwell last week. The code works with other data centers, and used to work with the IRIS service.

    Thanks!
    -- John

    ----------------------
    Web Services
    Topic home: http://ds.iris.edu/message-center/topic/webservices/ | Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu

    Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
    Update subscription preferences at http://ds.iris.edu/account/profile/


    • Philip Crotwell
      2023-02-03 16:49:26
      Hi

      I am still occasionally seeing what appears to be the same error John is
      seeing, on a fdsn station call on the CO network where I know it exists.
      Exact same query works many times then once in a while fails. From the
      error message and looking inside obspy, it appears that the http request is
      completing successfully, status=200 and the headers have Content-Encoding
      as "gzip", but then the payload has zero bytes. Since a gzip file always
      has a 12 byte header, zero bytes of payload causes the gzip decomp to fail.

      Very weird, and it appears to me that this can happen on random iris web
      services, so not a function of the actual request being made. I believe I
      have seen this even getting the wadl, which would have no parameters at all.

      Frequency of hitting this is high enough to be annoying but low enough to
      make reproducing it hard. :(

      Wild guess, but is there any chance this could be related to new
      authorization stuff?

      Philip


      On Mon, Jan 30, 2023 at 8:02 PM Rob Casey (via IRIS) <
      webservices-bounce<at>lists.ds.iris.edu> wrote:


      Hi John-

      KY has little in the way of BH and HH channels, which you can see in this
      query:


      http://service.iris.edu/fdsnws/station/1/query?network=KY&format=text&level=channel

      In fact, if you apply a channel filter to look for just those types, you
      will only find HH channels for CUSSO, so that will be all you can hope to
      extract. We can look at the time extent for data availability here:


      http://service.iris.edu/fdsnws/availability/1/extent?network=KY&format=text&cha=HH*

      Aside from only seeing three channels, you should be able to get data from
      them. We have still not sorted out why we get this error via ObsPy, but we
      think that something built into the ObsPy client is expecting a Gzipped
      return and getting something else instead. This could be a mis-marked
      content type header from the sending service. It's possible that what is
      being sent back is some kind of error message, but I don't recall that we
      have that content figured out.

      We have seen this problem before with some of our internal routines, and
      it specifically points to ObsPy and its current interaction with our
      station service. It's still on our problem list.

      Have you tried different time lengths with this specific query, or
      narrowed it to individual channels? Sorry that I don't have a definitive
      answer for this one. We'll try to solve it when we can spare people cycles.

      -Rob


      On Jan 28, 2023, at 10:46 AM, John West (via IRIS) <
      webservices-bounce<at>lists.ds.iris.edu> wrote:

      Hi! I'm using Obspy to call the stations web service to get BH* and HH*
      channels for a given network and 2-hour time window. Code snippet:
      Channels = DCclient.get_stations(startbefore=FileStart, endafter=FileEnd,
      network=N.code, channel=CHANNELS, level='channel')

      I get the following error:
      could not retrieve channels for datacenter IRIS: KY Not a gzipped file
      (b'a\r') Traceback (most recent call last): File
      "/home/adept/code/Channels.py", line 213, in FDSNchannels Channels =
      DCclient.get_stations(startbefore=FileStart, endafter=FileEnd, File
      "/usr/local/lib/python3.8/dist-packages/obspy/clients/fdsn/client.py", line
      725, in get_stations data_stream = self._download(url) File
      "/usr/local/lib/python3.8/dist-packages/obspy/clients/fdsn/client.py", line
      1393, in _download code, data = download_url( File
      "/usr/local/lib/python3.8/dist-packages/obspy/clients/fdsn/client.py", line
      1819, in download_url data = io.BytesIO(f.read()) File
      "/usr/lib/python3.8/gzip.py", line 292, in read return
      self._buffer.read(size) File "/usr/lib/python3.8/gzip.py", line 479, in
      read if not self._read_gzip_header(): File "/usr/lib/python3.8/gzip.py",
      line 427, in _read_gzip_header raise BadGzipFile('Not a gzipped file (%r)'
      % magic) gzip.BadGzipFile: Not a gzipped file (b'a\r')

      I don't know if this is related to the issue reported by Philip Crotwell
      last week. The code works with other data centers, and used to work with
      the IRIS service.

      Thanks!
      -- John

      ----------------------
      Web Services
      Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
      Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu

      Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
      Update subscription preferences at http://ds.iris.edu/account/profile/



      ----------------------
      Web Services
      Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
      Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu

      Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
      Update subscription preferences at http://ds.iris.edu/account/profile/


      • Rob Casey
        2023-02-06 13:44:48

        Hi Philip.


        Frequency of hitting this is high enough to be annoying but low enough to make reproducing it hard. :(

        Indeed.

        Wild guess, but is there any chance this could be related to new authorization stuff?

        No. I think it relates to changes introduced when we upgraded to Java 11. The timing is quite coincidental. The new authorization stuff mostly involved work at a level above the REST services. We retained the existing auth system in REST. Also, the auth stuff won't trigger on a vanilla /query endpoint.


        -Rob



17:14:54 v.22510d55