Thread: Re: intermittent problems with station service

https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465*issuecomment-2138050112__;Iw!!IKRxdwAv5BmarQ!ZzykRuyHMXnnNf_WNg2VpUZ5sHbDDQG6nc-PMssbhuFDx-mEWpN_Hg1VesVNSqhgrrkpCndDM1PEUDwJMYhfVVdfswAz$

cheers,
Tobias


On 29/05/2024 20:21, John West (via SAGE) wrote:

We might be dealing with two different issues, but perhaps we should

solve one at a time.

Well, OK, but the original issue was primarily about the XML problem. I
occasionally see the malformed gzip issue, but get the XML error
multiple times per day.

I'll look into turning off gzip in the obspy client, thanks!
-- John


On Wed, May 29, 2024 at 12:09 PM Rob Casey <rob.casey<at>earthscope.org
<rob.casey<at>earthscope.org>> wrote:



Hi John-

I have a couple of questions here: Is the gzip problem and the

malformed XML problem the same issue? Or are we dealing with two
different issues?


We might be dealing with two different issues, but perhaps we should
solve one at a time.

I'm using the obspy client, which handles the requesting in the

background. Is there a way to append headers to that client, or
would I have to rewrite my code to do that?


Not speaking from experience, but I note that the Accept-encoding
header addition is played out in obspy.clients.fdsn.client. I don't
see a way to externally configure this, so to test this, you'd need
to modify the code so that 'use_gzip' is set to false by default.
If that isn't enough, then you'd want to assert "gzip;q=0" where the
Accept-encoding header is written in that code.

Naturally, having the server on our end functioning suffiently well
with gzip turned off would be preferable. I am just not sure yet
what the side effects might be in making this change, including data
transfer performance.

-Rob

Thanks!
-- John


On Wed, May 29, 2024 at 11:45 AM Rob Casey

<rob.casey<at>earthscope.org <rob.casey<at>earthscope.org>> wrote:

Hi John and Philip-

Gillian alerted me to your posts. This has been a long

nagging problem that didn't present obvious solutions over the past
couple of years. We have had little to not staff time that we have
been able to place on this issue, either as we are deep in
infrastructure development in our new center.

That being said, one of our staffers, Dan, took a crack

at trying to find a pattern and look for a way to circumvent, if not
solve. He got to the stage where we realized that there were not a
specific set of distributed servers that were causing the problem,
it could come from any of the systems, just at indeterminate times.
Something with the fdsnws-station web service call causes it to go
to a multipart content mode that inserts byte size values and seems
to trigger gzip output. (thx. Tobias)

There is a client-based way that Dan suggested could get

around the issue, and that is to have the client request to not
receive gzipped returns. He describes it like so:

-------------------------------------------------------------------------

You can explicitly say "don't send gzip" by adding q=0

like this:

Accept-Encoding:gzip;q=0

With the following in my Python code:

headers = {"Accept-Encoding":"gzip;q=0"} # asks

server not to use gzip Transfer-Encoding

r = requests.get(url, headers=headers)

I get zero issues from station service.

-------------------------------------------------------------------------

So, you can try this out and see whether you get improved

results. In fact, it would be great to get outside evaulation of
this. To be sustainable, though, we need a server-side solution, so
Dan also suggested that there is a way to configure Tomcat to not
opt for gzip output. This is something that we should be able to
try out in beta and see if that doesn't break something else.

Regards,

-Rob

On May 29, 2024, at 10:23 AM, John West (via SAGE)

<webservices-bounce<at>lists.ds.iris.edu
<webservices-bounce<at>lists.ds.iris.edu>> wrote:

Tobias is apparently having trouble posting to the mail list.

He has opened an issue on github to track this problem:

(

https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$>
)

-- John


On Wed, May 29, 2024 at 10:59 AM Tobias Megies

<tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>> wrote:

Hi John,

yes, that's what I meant. My best guess was that the service

discovery

was creating the issue, because it's the only time we do

multiple

requests in parallel.
But yes, I can see that something also sometimes goes wrong in

"normal"

requests after the client initialization. Seems like a server

issue to

me but not 100% sure yet.

Can you maybe mention that github ticket I made on the ML

(

https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$>
), my mails don't go through,

my old IRIS account has my old mail sender address and I made

anew

account on EarthScope but no luck with that either.


cheers,
Tobias


On 29/05/2024 17:38, John D. West wrote:

Hi, Tobias.

I changed line 21 of my sample code to: DCclient =

Client('IRIS',

_discover_services=False)

Is that what you had in mind? It doesn't make any difference,

the code

still generates errors on random networks.

Thanks!
-- John


On Wed, May 29, 2024 at 9:29 AM Tobias Megies

<tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>

<tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>>>

wrote:

Hi John,

a quick fix is this:

client = Client(_discover_services=False)

This skips service discovery (parsing the server side

WADLs) and makes

it so custom parameters Earthscope has can't be used, but

it should be

fine unless you do some real fancy requests.

cheers,
Tobias

On 29/05/2024 17:26, Tobias Megies wrote:

I opened an issue on github to track the issue:

https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$>
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$

cheers,
Tobias


On 29/05/2024 04:19, John West (via SAGE) wrote:

Hi!

I'm calling the station service using the

obspy client.get_stations()

method, and getting randomly distributed badly formed

XML.

Typical error message is something like: "Start tag

expected,

'<' not

found, line 1, column 1 (<string>, line 1)"

I've also had a few errors with: "Not a gzipped file

(b'a\r')"

The attached sample Python code sometimes runs

without error, and

sometimes generates multiple errors during a run. The

errors are

NOT

in the same places, they appear for different

networks on each run.

Any ideas?

Thanks!
-- John

--
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>

<tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>>

WWW:

https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$>
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$

Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck

Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970

Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München

Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205

--
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:

https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$

Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck

Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970

Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München

Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205

----------------------
Web Services
Topic home:

https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlv0ti8Wwg$
<
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlv0ti8Wwg$>
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu <mailto:
webservices-unsubscribe<at>lists.ds.iris.edu>

Sent from the IRIS Message Center

(

https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlvVI56oCQ$
<
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlvVI56oCQ$>
)

Update subscription preferences at

https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUltH577DAA$
<
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUltH577DAA$

----------------------
Web Services
Topic home:

https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLtKquQ_Q$
<
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLtKquQ_Q$>
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu <mailto:
webservices-unsubscribe<at>lists.ds.iris.edu>

Sent from the IRIS Message Center

(

https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YJT1c67lQ$
<
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YJT1c67lQ$>
)

Update subscription preferences at

https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLsKF4oPA$
<
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLsKF4oPA$

----------------------
Web Services
Topic home:

https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ZzykRuyHMXnnNf_WNg2VpUZ5sHbDDQG6nc-PMssbhuFDx-mEWpN_Hg1VesVNSqhgrrkpCndDM1PEUDwJMYhfVRmKmXan$
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu

Sent from the IRIS Message Center (

https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ZzykRuyHMXnnNf_WNg2VpUZ5sHbDDQG6nc-PMssbhuFDx-mEWpN_Hg1VesVNSqhgrrkpCndDM1PEUDwJMYhfVfquSI3V$
)

Update subscription preferences at

https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ZzykRuyHMXnnNf_WNg2VpUZ5sHbDDQG6nc-PMssbhuFDx-mEWpN_Hg1VesVNSqhgrrkpCndDM1PEUDwJMYhfVYM97ReI$

--
EMail: tobias.megies<at>lmu.de
WWW:
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!ZzykRuyHMXnnNf_WNg2VpUZ5sHbDDQG6nc-PMssbhuFDx-mEWpN_Hg1VesVNSqhgrrkpCndDM1PEUDwJMYhfVeUvNoAZ$

Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck

Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970

Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München

Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205