John West
Tobias is apparently having trouble posting to the mail list. He has opened
an issue on github to track this problem:
(
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$
)
-- John
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de> wrote:
an issue on github to track this problem:
(
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$
)
-- John
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de> wrote:
Hi John,
yes, that's what I meant. My best guess was that the service discovery
was creating the issue, because it's the only time we do multiple
requests in parallel.
But yes, I can see that something also sometimes goes wrong in "normal"
requests after the client initialization. Seems like a server issue to
me but not 100% sure yet.
Can you maybe mention that github ticket I made on the ML
(
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$
), my mails don't go through,
my old IRIS account has my old mail sender address and I made anew
account on EarthScope but no luck with that either.
cheers,
Tobias
On 29/05/2024 17:38, John D. West wrote:
Hi, Tobias.makes
I changed line 21 of my sample code to: DCclient = Client('IRIS',
_discover_services=False)
Is that what you had in mind? It doesn't make any difference, the code
still generates errors on random networks.
Thanks!
-- John
On Wed, May 29, 2024 at 9:29 AM Tobias Megies <tobias.megies<at>lmu.de
<tobias.megies<at>lmu.de>> wrote:
Hi John,
a quick fix is this:
client = Client(_discover_services=False)
This skips service discovery (parsing the server side WADLs) and
it so custom parameters Earthscope has can't be used, but it shouldbe
fine unless you do some real fancy requests.https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$
cheers,
Tobias
On 29/05/2024 17:26, Tobias Megies wrote:
I opened an issue on github to track the issue:
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$
run.cheers,obspy client.get_stations()
Tobias
On 29/05/2024 04:19, John West (via SAGE) wrote:
Hi!
I'm calling the station service using the
'<' notmethod, and getting randomly distributed badly formed XML.
Typical error message is something like: "Start tag expected,
NOTfound, line 1, column 1 (<string>, line 1)"
I've also had a few errors with: "Not a gzipped file (b'a\r')"
The attached sample Python code sometimes runs without error, and
sometimes generates multiple errors during a run. The errors are
in the same places, they appear for different networks on each
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$--Any ideas?
Thanks!
-- John
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
Geophysikalisches Observatorium--
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
EMail: tobias.megies<at>lmu.de
WWW:
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$
Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
-
Hi John and Philip-
Gillian alerted me to your posts. This has been a long nagging problem that didn't present obvious solutions over the past couple of years. We have had little to not staff time that we have been able to place on this issue, either as we are deep in infrastructure development in our new center.
That being said, one of our staffers, Dan, took a crack at trying to find a pattern and look for a way to circumvent, if not solve. He got to the stage where we realized that there were not a specific set of distributed servers that were causing the problem, it could come from any of the systems, just at indeterminate times. Something with the fdsnws-station web service call causes it to go to a multipart content mode that inserts byte size values and seems to trigger gzip output. (thx. Tobias)
There is a client-based way that Dan suggested could get around the issue, and that is to have the client request to not receive gzipped returns. He describes it like so:
-------------------------------------------------------------------------
You can explicitly say "don't send gzip" by adding q=0 like this:
Accept-Encoding:gzip;q=0
With the following in my Python code:
headers = {"Accept-Encoding":"gzip;q=0"} # asks server not to use gzip Transfer-Encoding
r = requests.get(url, headers=headers)
I get zero issues from station service.
-------------------------------------------------------------------------
So, you can try this out and see whether you get improved results. In fact, it would be great to get outside evaulation of this. To be sustainable, though, we need a server-side solution, so Dan also suggested that there is a way to configure Tomcat to not opt for gzip output. This is something that we should be able to try out in beta and see if that doesn't break something else.
Regards,
-Rob
On May 29, 2024, at 10:23 AM, John West (via SAGE) <webservices-bounce<at>lists.ds.iris.edu> wrote:
Tobias is apparently having trouble posting to the mail list. He has opened an issue on github to track this problem:
(https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$ )
-- John
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de> wrote:
Hi John,
yes, that's what I meant. My best guess was that the service discovery
was creating the issue, because it's the only time we do multiple
requests in parallel.
But yes, I can see that something also sometimes goes wrong in "normal"
requests after the client initialization. Seems like a server issue to
me but not 100% sure yet.
Can you maybe mention that github ticket I made on the ML
(https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$ ), my mails don't go through,
my old IRIS account has my old mail sender address and I made anew
account on EarthScope but no luck with that either.
cheers,
Tobias
On 29/05/2024 17:38, John D. West wrote:
Hi, Tobias.
--
I changed line 21 of my sample code to: DCclient = Client('IRIS',
_discover_services=False)
Is that what you had in mind? It doesn't make any difference, the code
still generates errors on random networks.
Thanks!
-- John
On Wed, May 29, 2024 at 9:29 AM Tobias Megies <tobias.megies<at>lmu.de
<tobias.megies<at>lmu.de>> wrote:
Hi John,
a quick fix is this:
client = Client(_discover_services=False)
This skips service discovery (parsing the server side WADLs) and makes
it so custom parameters Earthscope has can't be used, but it should be
fine unless you do some real fancy requests.
cheers,
Tobias
On 29/05/2024 17:26, Tobias Megies wrote:
I opened an issue on github to track the issue:
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$ https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$>
cheers,
obspy client.get_stations()
Tobias
On 29/05/2024 04:19, John West (via SAGE) wrote:
Hi!
I'm calling the station service using the
'<' notmethod, and getting randomly distributed badly formed XML.
Typical error message is something like: "Start tag expected,
NOTfound, line 1, column 1 (<string>, line 1)"
I've also had a few errors with: "Not a gzipped file (b'a\r')"
The attached sample Python code sometimes runs without error, and
sometimes generates multiple errors during a run. The errors are
--in the same places, they appear for different networks on each run.
Any ideas?
Thanks!
-- John
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$ https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$>
Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
EMail: tobias.megies<at>lmu.de
WWW: https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$
Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ | Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
-
Also, as a PSA, we are moving away from the old IRIS mailing lists and encouraging everyone to bring their questions to the EarthScope help desk at:
data-help<at>earthscope.org <data-help<at>earthscope.org>
This is a curated help desk location where we can have multiple eyes on the issues you bring forward and we will address them as promptly as possible.
Thank you for your continued interest in our services.
-Rob
On May 29, 2024, at 10:46 AM, Rob Casey (via SAGE) <webservices-bounce<at>lists.ds.iris.edu> wrote:
Hi John and Philip-
Gillian alerted me to your posts. This has been a long nagging problem that didn't present obvious solutions over the past couple of years. We have had little to not staff time that we have been able to place on this issue, either as we are deep in infrastructure development in our new center.
That being said, one of our staffers, Dan, took a crack at trying to find a pattern and look for a way to circumvent, if not solve. He got to the stage where we realized that there were not a specific set of distributed servers that were causing the problem, it could come from any of the systems, just at indeterminate times. Something with the fdsnws-station web service call causes it to go to a multipart content mode that inserts byte size values and seems to trigger gzip output. (thx. Tobias)
There is a client-based way that Dan suggested could get around the issue, and that is to have the client request to not receive gzipped returns. He describes it like so:
-------------------------------------------------------------------------
You can explicitly say "don't send gzip" by adding q=0 like this:
Accept-Encoding:gzip;q=0
With the following in my Python code:
headers = {"Accept-Encoding":"gzip;q=0"} # asks server not to use gzip Transfer-Encoding
r = requests.get(url, headers=headers)
I get zero issues from station service.
-------------------------------------------------------------------------
So, you can try this out and see whether you get improved results. In fact, it would be great to get outside evaulation of this. To be sustainable, though, we need a server-side solution, so Dan also suggested that there is a way to configure Tomcat to not opt for gzip output. This is something that we should be able to try out in beta and see if that doesn't break something else.
Regards,
-Rob
On May 29, 2024, at 10:23 AM, John West (via SAGE) <webservices-bounce<at>lists.ds.iris.edu> wrote:
----------------------
Tobias is apparently having trouble posting to the mail list. He has opened an issue on github to track this problem:
(https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$ )
-- John
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de> wrote:
Hi John,
yes, that's what I meant. My best guess was that the service discovery
was creating the issue, because it's the only time we do multiple
requests in parallel.
But yes, I can see that something also sometimes goes wrong in "normal"
requests after the client initialization. Seems like a server issue to
me but not 100% sure yet.
Can you maybe mention that github ticket I made on the ML
(https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$ ), my mails don't go through,
my old IRIS account has my old mail sender address and I made anew
account on EarthScope but no luck with that either.
cheers,
Tobias
On 29/05/2024 17:38, John D. West wrote:
Hi, Tobias.
--
I changed line 21 of my sample code to: DCclient = Client('IRIS',
_discover_services=False)
Is that what you had in mind? It doesn't make any difference, the code
still generates errors on random networks.
Thanks!
-- John
On Wed, May 29, 2024 at 9:29 AM Tobias Megies <tobias.megies<at>lmu.de
<tobias.megies<at>lmu.de>> wrote:
Hi John,
a quick fix is this:
client = Client(_discover_services=False)
This skips service discovery (parsing the server side WADLs) and makes
it so custom parameters Earthscope has can't be used, but it should be
fine unless you do some real fancy requests.
cheers,
Tobias
On 29/05/2024 17:26, Tobias Megies wrote:
I opened an issue on github to track the issue:
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$ https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$>
cheers,
obspy client.get_stations()
Tobias
On 29/05/2024 04:19, John West (via SAGE) wrote:
Hi!
I'm calling the station service using the
'<' notmethod, and getting randomly distributed badly formed XML.
Typical error message is something like: "Start tag expected,
NOTfound, line 1, column 1 (<string>, line 1)"
I've also had a few errors with: "Not a gzipped file (b'a\r')"
The attached sample Python code sometimes runs without error, and
sometimes generates multiple errors during a run. The errors are
--in the same places, they appear for different networks on each run.
Any ideas?
Thanks!
-- John
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$ https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$>
Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
EMail: tobias.megies<at>lmu.de
WWW: https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$
Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ | Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ | Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
-
Thanks, Rob! I *thought* I had seen an email about the help desk, but when
I started searching for web services help it wasn't mentioned in my search
results.
-- John
On Wed, May 29, 2024 at 11:57 AM Rob Casey (via SAGE) <
webservices-bounce<at>lists.ds.iris.edu> wrote:
Also, as a PSA, we are moving away from the old IRIS mailing lists and
encouraging everyone to bring their questions to the EarthScope help desk
at:
data-help<at>earthscope.org
This is a curated help desk location where we can have multiple eyes on
the issues you bring forward and we will address them as promptly as
possible.
Thank you for your continued interest in our services.
-Rob
On May 29, 2024, at 10:46 AM, Rob Casey (via SAGE) <
webservices-bounce<at>lists.ds.iris.edu> wrote:
Hi John and Philip-
Gillian alerted me to your posts. This has been a long nagging problem
that didn't present obvious solutions over the past couple of years. We
have had little to not staff time that we have been able to place on this
issue, either as we are deep in infrastructure development in our new
center.
That being said, one of our staffers, Dan, took a crack at trying to find
a pattern and look for a way to circumvent, if not solve. He got to the
stage where we realized that there were not a specific set of distributed
servers that were causing the problem, it could come from any of the
systems, just at indeterminate times. Something with the fdsnws-station
web service call causes it to go to a multipart content mode that inserts
byte size values and seems to trigger gzip output. (thx. Tobias)
There is a client-based way that Dan suggested could get around the issue,
and that is to have the client request to not receive gzipped returns. He
describes it like so:
-------------------------------------------------------------------------
You can explicitly say "don't send gzip" by adding q=0 like this:
Accept-Encoding:gzip;q=0
With the following in my Python code:
headers = {"Accept-Encoding":"gzip;q=0"} # asks server not to use gzip
Transfer-Encoding
r = requests.get(url, headers=headers)
I get zero issues from station service.
-------------------------------------------------------------------------
So, you can try this out and see whether you get improved results. In
fact, it would be great to get outside evaulation of this. To be
sustainable, though, we need a server-side solution, so Dan also suggested
that there is a way to configure Tomcat to not opt for gzip output. This
is something that we should be able to try out in beta and see if that
doesn't break something else.
Regards,
-Rob
On May 29, 2024, at 10:23 AM, John West (via SAGE) <
webservices-bounce<at>lists.ds.iris.edu> wrote:
Tobias is apparently having trouble posting to the mail list. He has
opened an issue on github to track this problem:
(
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$
)
-- John
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de>
wrote:
Hi John,
yes, that's what I meant. My best guess was that the service discovery
was creating the issue, because it's the only time we do multiple
requests in parallel.
But yes, I can see that something also sometimes goes wrong in "normal"
requests after the client initialization. Seems like a server issue to
me but not 100% sure yet.
Can you maybe mention that github ticket I made on the ML
(
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$
), my mails don't go through,
my old IRIS account has my old mail sender address and I made anew
account on EarthScope but no luck with that either.
cheers,
Tobias
On 29/05/2024 17:38, John D. West wrote:
Hi, Tobias.
I changed line 21 of my sample code to: DCclient = Client('IRIS',
_discover_services=False)
Is that what you had in mind? It doesn't make any difference, the code
still generates errors on random networks.
Thanks!
-- John
On Wed, May 29, 2024 at 9:29 AM Tobias Megies <tobias.megies<at>lmu.de
<tobias.megies<at>lmu.de>> wrote:
Hi John,
a quick fix is this:
client = Client(_discover_services=False)
This skips service discovery (parsing the server side WADLs) and makes
it so custom parameters Earthscope has can't be used, but it should be
fine unless you do some real fancy requests.
cheers,
Tobias
On 29/05/2024 17:26, Tobias Megies wrote:
I opened an issue on github to track the issue:
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$%3E
cheers,
Tobias
On 29/05/2024 04:19, John West (via SAGE) wrote:
Hi!
I'm calling the station service using the
obspy client.get_stations()
method, and getting randomly distributed badly formed XML.
Typical error message is something like: "Start tag expected,
'<' not
found, line 1, column 1 (<string>, line 1)"
I've also had a few errors with: "Not a gzipped file (b'a\r')"
The attached sample Python code sometimes runs without error, and
sometimes generates multiple errors during a run. The errors are
NOT
in the same places, they appear for different networks on each run.
Any ideas?
Thanks!
-- John
--
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$%3E
Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
--
EMail: tobias.megies<at>lmu.de
WWW:
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$
Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!fMdbCYcWRo4li8OOs029nTneQiRsn-cxHfUxgaU5kp8Zwt3XphdaC83YyyHZn4o0AnkutkNC1dlCI5UDJP3jbPlc20vq2EvIQiSjfAo$>
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!fMdbCYcWRo4li8OOs029nTneQiRsn-cxHfUxgaU5kp8Zwt3XphdaC83YyyHZn4o0AnkutkNC1dlCI5UDJP3jbPlc20vq2EvIOrHDGDc$>
)
Update subscription preferences at http://ds.iris.edu/account/profile/
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!fMdbCYcWRo4li8OOs029nTneQiRsn-cxHfUxgaU5kp8Zwt3XphdaC83YyyHZn4o0AnkutkNC1dlCI5UDJP3jbPlc20vq2EvIYRdLehQ$>
----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!fMdbCYcWRo4li8OOs029nTneQiRsn-cxHfUxgaU5kp8Zwt3XphdaC83YyyHZn4o0AnkutkNC1dlCI5UDJP3jbPlc20vq2EvIQiSjfAo$>
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!fMdbCYcWRo4li8OOs029nTneQiRsn-cxHfUxgaU5kp8Zwt3XphdaC83YyyHZn4o0AnkutkNC1dlCI5UDJP3jbPlc20vq2EvIOrHDGDc$>
)
Update subscription preferences at http://ds.iris.edu/account/profile/
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!fMdbCYcWRo4li8OOs029nTneQiRsn-cxHfUxgaU5kp8Zwt3XphdaC83YyyHZn4o0AnkutkNC1dlCI5UDJP3jbPlc20vq2EvIYRdLehQ$>
----------------------
Web Services
Topic home:
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!fMdbCYcWRo4li8OOs029nTneQiRsn-cxHfUxgaU5kp8Zwt3XphdaC83YyyHZn4o0AnkutkNC1dlCI5UDJP3jbPlc20vq2EvIQiSjfAo$
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!fMdbCYcWRo4li8OOs029nTneQiRsn-cxHfUxgaU5kp8Zwt3XphdaC83YyyHZn4o0AnkutkNC1dlCI5UDJP3jbPlc20vq2EvIOrHDGDc$
)
Update subscription preferences at
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!fMdbCYcWRo4li8OOs029nTneQiRsn-cxHfUxgaU5kp8Zwt3XphdaC83YyyHZn4o0AnkutkNC1dlCI5UDJP3jbPlc20vq2EvIYRdLehQ$
-
-
Hi, Rob!
I have a couple of questions here: Is the gzip problem and the malformed
XML problem the same issue? Or are we dealing with two different issues?
I'm using the obspy client, which handles the requesting in the background.
Is there a way to append headers to that client, or would I have to rewrite
my code to do that?
Thanks!
-- John
On Wed, May 29, 2024 at 11:45 AM Rob Casey <rob.casey<at>earthscope.org> wrote:
Hi John and Philip-
Gillian alerted me to your posts. This has been a long nagging
problem that didn't present obvious solutions over the past couple of
years. We have had little to not staff time that we have been able to
place on this issue, either as we are deep in infrastructure development in
our new center.
That being said, one of our staffers, Dan, took a crack at trying
to find a pattern and look for a way to circumvent, if not solve. He got
to the stage where we realized that there were not a specific set of
distributed servers that were causing the problem, it could come from any
of the systems, just at indeterminate times. Something with the
fdsnws-station web service call causes it to go to a multipart content mode
that inserts byte size values and seems to trigger gzip output. (thx.
Tobias)
There is a client-based way that Dan suggested could get around
the issue, and that is to have the client request to not receive gzipped
returns. He describes it like so:
-------------------------------------------------------------------------
You can explicitly say "don't send gzip" by adding q=0 like this:
Accept-Encoding:gzip;q=0
With the following in my Python code:
headers = {"Accept-Encoding":"gzip;q=0"} # asks server not
to use gzip Transfer-Encoding
r = requests.get(url, headers=headers)
I get zero issues from station service.
-------------------------------------------------------------------------
So, you can try this out and see whether you get improved
results. In fact, it would be great to get outside evaulation of this. To
be sustainable, though, we need a server-side solution, so Dan also
suggested that there is a way to configure Tomcat to not opt for gzip
output. This is something that we should be able to try out in beta and
see if that doesn't break something else.
Regards,
-Rob
On May 29, 2024, at 10:23 AM, John West (via SAGE) <
webservices-bounce<at>lists.ds.iris.edu> wrote:
Tobias is apparently having trouble posting to the mail list. He has
opened an issue on github to track this problem:
(
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$
)
-- John
wrote:
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de>
Hi John,
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$
yes, that's what I meant. My best guess was that the service discovery
was creating the issue, because it's the only time we do multiple
requests in parallel.
But yes, I can see that something also sometimes goes wrong in "normal"
requests after the client initialization. Seems like a server issue to
me but not 100% sure yet.
Can you maybe mention that github ticket I made on the ML
(
), my mails don't go through,
my old IRIS account has my old mail sender address and I made anew
makes
account on EarthScope but no luck with that either.
cheers,
Tobias
On 29/05/2024 17:38, John D. West wrote:
Hi, Tobias.
I changed line 21 of my sample code to: DCclient = Client('IRIS',
_discover_services=False)
Is that what you had in mind? It doesn't make any difference, the code
still generates errors on random networks.
Thanks!
-- John
On Wed, May 29, 2024 at 9:29 AM Tobias Megies <tobias.megies<at>lmu.de
<tobias.megies<at>lmu.de>> wrote:
Hi John,
a quick fix is this:
client = Client(_discover_services=False)
This skips service discovery (parsing the server side WADLs) and
should beit so custom parameters Earthscope has can't be used, but it
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$fine unless you do some real fancy requests.
cheers,
Tobias
On 29/05/2024 17:26, Tobias Megies wrote:
I opened an issue on github to track the issue:
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$
andcheers,
obspy client.get_stations()
Tobias
On 29/05/2024 04:19, John West (via SAGE) wrote:
Hi!
I'm calling the station service using the
'<' notmethod, and getting randomly distributed badly formed XML.
Typical error message is something like: "Start tag expected,
found, line 1, column 1 (<string>, line 1)"
I've also had a few errors with: "Not a gzipped file (b'a\r')"
The attached sample Python code sometimes runs without error,
aresometimes generates multiple errors during a run. The errors
run.NOT
in the same places, they appear for different networks on each
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
--Any ideas?
Thanks!
-- John
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$Geophysikalisches Observatorium
--
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
EMail: tobias.megies<at>lmu.de
WWW:
Geophysikalisches Observatorium
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlv0ti8Wwg$
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
----------------------
Web Services
Topic home:
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlvVI56oCQ$
)
Update subscription preferences at
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUltH577DAA$
-
Hi John-
I have a couple of questions here: Is the gzip problem and the malformed XML problem the same issue? Or are we dealing with two different issues?
We might be dealing with two different issues, but perhaps we should solve one at a time.
I'm using the obspy client, which handles the requesting in the background. Is there a way to append headers to that client, or would I have to rewrite my code to do that?
Not speaking from experience, but I note that the Accept-encoding header addition is played out in obspy.clients.fdsn.client. I don't see a way to externally configure this, so to test this, you'd need to modify the code so that 'use_gzip' is set to false by default. If that isn't enough, then you'd want to assert "gzip;q=0" where the Accept-encoding header is written in that code.
Naturally, having the server on our end functioning suffiently well with gzip turned off would be preferable. I am just not sure yet what the side effects might be in making this change, including data transfer performance.
-Rob
Thanks!
-- John
On Wed, May 29, 2024 at 11:45 AM Rob Casey <rob.casey<at>earthscope.org> wrote:
Hi John and Philip-
Gillian alerted me to your posts. This has been a long nagging problem that didn't present obvious solutions over the past couple of years. We have had little to not staff time that we have been able to place on this issue, either as we are deep in infrastructure development in our new center.
That being said, one of our staffers, Dan, took a crack at trying to find a pattern and look for a way to circumvent, if not solve. He got to the stage where we realized that there were not a specific set of distributed servers that were causing the problem, it could come from any of the systems, just at indeterminate times. Something with the fdsnws-station web service call causes it to go to a multipart content mode that inserts byte size values and seems to trigger gzip output. (thx. Tobias)
There is a client-based way that Dan suggested could get around the issue, and that is to have the client request to not receive gzipped returns. He describes it like so:
-------------------------------------------------------------------------
You can explicitly say "don't send gzip" by adding q=0 like this:
Accept-Encoding:gzip;q=0
With the following in my Python code:
headers = {"Accept-Encoding":"gzip;q=0"} # asks server not to use gzip Transfer-Encoding
r = requests.get(url, headers=headers)
I get zero issues from station service.
-------------------------------------------------------------------------
So, you can try this out and see whether you get improved results. In fact, it would be great to get outside evaulation of this. To be sustainable, though, we need a server-side solution, so Dan also suggested that there is a way to configure Tomcat to not opt for gzip output. This is something that we should be able to try out in beta and see if that doesn't break something else.
Regards,
-Rob
On May 29, 2024, at 10:23 AM, John West (via SAGE) <webservices-bounce<at>lists.ds.iris.edu> wrote:
----------------------
Tobias is apparently having trouble posting to the mail list. He has opened an issue on github to track this problem:
(https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$ )
-- John
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de> wrote:
Hi John,
yes, that's what I meant. My best guess was that the service discovery
was creating the issue, because it's the only time we do multiple
requests in parallel.
But yes, I can see that something also sometimes goes wrong in "normal"
requests after the client initialization. Seems like a server issue to
me but not 100% sure yet.
Can you maybe mention that github ticket I made on the ML
(https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$ ), my mails don't go through,
my old IRIS account has my old mail sender address and I made anew
account on EarthScope but no luck with that either.
cheers,
Tobias
On 29/05/2024 17:38, John D. West wrote:
Hi, Tobias.
--
I changed line 21 of my sample code to: DCclient = Client('IRIS',
_discover_services=False)
Is that what you had in mind? It doesn't make any difference, the code
still generates errors on random networks.
Thanks!
-- John
On Wed, May 29, 2024 at 9:29 AM Tobias Megies <tobias.megies<at>lmu.de
<tobias.megies<at>lmu.de>> wrote:
Hi John,
a quick fix is this:
client = Client(_discover_services=False)
This skips service discovery (parsing the server side WADLs) and makes
it so custom parameters Earthscope has can't be used, but it should be
fine unless you do some real fancy requests.
cheers,
Tobias
On 29/05/2024 17:26, Tobias Megies wrote:
I opened an issue on github to track the issue:
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$ https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$>
cheers,
obspy client.get_stations()
Tobias
On 29/05/2024 04:19, John West (via SAGE) wrote:
Hi!
I'm calling the station service using the
'<' notmethod, and getting randomly distributed badly formed XML.
Typical error message is something like: "Start tag expected,
NOTfound, line 1, column 1 (<string>, line 1)"
I've also had a few errors with: "Not a gzipped file (b'a\r')"
The attached sample Python code sometimes runs without error, and
sometimes generates multiple errors during a run. The errors are
--in the same places, they appear for different networks on each run.
Any ideas?
Thanks!
-- John
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$ https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$>
Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
EMail: tobias.megies<at>lmu.de
WWW: https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$
Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
----------------------
Web Services
Topic home: https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlv0ti8Wwg$ | Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlvVI56oCQ$ )
Update subscription preferences at https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUltH577DAA$
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ | Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
-
We might be dealing with two different issues, but perhaps we should solve
one at a time.
Well, OK, but the original issue was primarily about the XML problem. I
occasionally see the malformed gzip issue, but get the XML error multiple
times per day.
I'll look into turning off gzip in the obspy client, thanks!
-- John
On Wed, May 29, 2024 at 12:09 PM Rob Casey <rob.casey<at>earthscope.org> wrote:
Hi John-
I have a couple of questions here: Is the gzip problem and the malformed
XML problem the same issue? Or are we dealing with two different issues?
We might be dealing with two different issues, but perhaps we should solve
one at a time.
I'm using the obspy client, which handles the requesting in the
background. Is there a way to append headers to that client, or would I
have to rewrite my code to do that?
Not speaking from experience, but I note that the Accept-encoding header
addition is played out in obspy.clients.fdsn.client. I don't see a way to
externally configure this, so to test this, you'd need to modify the code
so that 'use_gzip' is set to false by default. If that isn't enough, then
you'd want to assert "gzip;q=0" where the Accept-encoding header is written
in that code.
Naturally, having the server on our end functioning suffiently well with
gzip turned off would be preferable. I am just not sure yet what the side
effects might be in making this change, including data transfer performance.
-Rob
Thanks!
wrote:
-- John
On Wed, May 29, 2024 at 11:45 AM Rob Casey <rob.casey<at>earthscope.org>
Hi John and Philip-
problem that didn't present obvious solutions over the past couple of
Gillian alerted me to your posts. This has been a long nagging
years. We have had little to not staff time that we have been able to
place on this issue, either as we are deep in infrastructure development in
our new center.
That being said, one of our staffers, Dan, took a crack at
trying to find a pattern and look for a way to circumvent, if not solve.
He got to the stage where we realized that there were not a specific set of
distributed servers that were causing the problem, it could come from any
of the systems, just at indeterminate times. Something with the
fdsnws-station web service call causes it to go to a multipart content mode
that inserts byte size values and seems to trigger gzip output. (thx.
Tobias)
There is a client-based way that Dan suggested could get around
the issue, and that is to have the client request to not receive gzipped
returns. He describes it like so:
-------------------------------------------------------------------------
not to use gzip Transfer-Encoding
You can explicitly say "don't send gzip" by adding q=0 like this:
Accept-Encoding:gzip;q=0
With the following in my Python code:
headers = {"Accept-Encoding":"gzip;q=0"} # asks server
r = requests.get(url, headers=headers)
results. In fact, it would be great to get outside evaulation of this. To
I get zero issues from station service.
-------------------------------------------------------------------------
So, you can try this out and see whether you get improved
be sustainable, though, we need a server-side solution, so Dan also
suggested that there is a way to configure Tomcat to not opt for gzip
output. This is something that we should be able to try out in beta and
see if that doesn't break something else.
Regards,
webservices-bounce<at>lists.ds.iris.edu> wrote:
-Rob
On May 29, 2024, at 10:23 AM, John West (via SAGE) <
opened an issue on github to track this problem:Tobias is apparently having trouble posting to the mail list. He has
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$(
)
wrote:-- John
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de>
"normal"Hi John,
yes, that's what I meant. My best guess was that the service discovery
was creating the issue, because it's the only time we do multiple
requests in parallel.
But yes, I can see that something also sometimes goes wrong in
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$requests after the client initialization. Seems like a server issue to
me but not 100% sure yet.
Can you maybe mention that github ticket I made on the ML
(
), my mails don't go through,
codemy old IRIS account has my old mail sender address and I made anew
account on EarthScope but no luck with that either.
cheers,
Tobias
On 29/05/2024 17:38, John D. West wrote:
Hi, Tobias.
I changed line 21 of my sample code to: DCclient = Client('IRIS',
_discover_services=False)
Is that what you had in mind? It doesn't make any difference, the
makesstill generates errors on random networks.
Thanks!
-- John
On Wed, May 29, 2024 at 9:29 AM Tobias Megies <tobias.megies<at>lmu.de
<tobias.megies<at>lmu.de>> wrote:
Hi John,
a quick fix is this:
client = Client(_discover_services=False)
This skips service discovery (parsing the server side WADLs) and
should beit so custom parameters Earthscope has can't be used, but it
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$fine unless you do some real fancy requests.
cheers,
Tobias
On 29/05/2024 17:26, Tobias Megies wrote:
I opened an issue on github to track the issue:
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$
(b'a\r')"cheers,
obspy client.get_stations()
Tobias
On 29/05/2024 04:19, John West (via SAGE) wrote:
Hi!
I'm calling the station service using the
'<' notmethod, and getting randomly distributed badly formed XML.
Typical error message is something like: "Start tag expected,
found, line 1, column 1 (<string>, line 1)"
I've also had a few errors with: "Not a gzipped file
error, andThe attached sample Python code sometimes runs without
aresometimes generates multiple errors during a run. The errors
each run.NOT
in the same places, they appear for different networks on
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
--Any ideas?
Thanks!
-- John
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$Geophysikalisches Observatorium
--
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
EMail: tobias.megies<at>lmu.de
WWW:
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlv0ti8Wwg$Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
----------------------
Web Services
Topic home:
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlvVI56oCQ$Sent from the IRIS Message Center (
)
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUltH577DAA$Update subscription preferences at
----------------------
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLtKquQ_Q$
Web Services
Topic home:
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YJT1c67lQ$
)
Update subscription preferences at
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLsKF4oPA$
-
Hi all
I just did some testing using my seisFile java library (so no obspy
dependency) and have gotten xml parse errors even with the http encoding
header set to be "identity" which I think turns off all compression. Output
xml for network 8A (I think) had the first line as "2000". All 98 previous
networks downloaded parsable stationxml for stations without error.
Rerunning, I hit the same thing, this time for network 2P, again with
"2000" as the first line. Running a third time, I got all the way to XA
before it crashed. See output below with http headers. All three failures
show that same first line of "2000" which seems really odd.
I think at least we can say this isn't an obspy issue.
Philip
Network: XA PANDA -- San Juan, Argentina Experiment (PANDA-ARGENTINA)
http://service.iris.edu:80/fdsnws/station/1/query?endafter=1987-01-01T00:00:00.000Z&level=station&network=XA&startbefore=1988-12-31T23:59:59.999Z
200
Unable to parse as staxml:
HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers:
Access-Control-Allow-Origin,Access-Control-Allow-Credentials
Vary:
origin,access-control-request-method,access-control-request-headers,accept-encoding
Date: Wed, 29 May 2024 18:44:05 GMT
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/xml
Connection: close
Date: Wed, 29 May 2024 18:44:05 GMT
Connection: close
2000
<?xml version="1.0" encoding="ISO-8859-1"?>
<FDSNStationXML xmlns="http://www.fdsn.org/xml/station/1" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance" xmlns:iris="
http://www.fdsn.org/xml/station/1/iris" xsi:schemaLocation="
http://www.fdsn.org/xml/station/1
http://www.fdsn.org/xml/station/fdsn-station-1.1.xsd" schemaVersion="1.1">
<Source>IRIS-DMC</Source>
<Sender>IRIS-DMC</Sender>
<Module>IRIS WEB SERVICE: fdsnws-station | version: 1.1.52</Module>
<ModuleURI>
http://service.iris.edu/fdsnws/station/1/query?endafter=1987-01-01T00:00:00.000Z&;level=station&network=XA&startbefore=1988-12-31T23:59:59.999Z
</ModuleURI>
<Created>2024-05-29T18:44:06.2266</Created>
<Network code="XA" startDate="1987-01-01T00:00:00.0000"
endDate="1988-12-31T23:59:59.9999" restrictedStatus="open">
<Description>PANDA -- San Juan, Argentina Experiment
(PANDA-ARGENTINA)</Description>
<TotalNumberStations>80</TotalNumberStations>
<SelectedNumberStations>80</SelectedNumberStations>
<Station code="I01H" startDate="1987-08-25T00:00:00.0000"
endDate="1988-12-30T23:59:59.0000" restrictedStatus="open"
iris:alternateNetworkCodes="_CERI-TEMPORARY,.UNRESTRICTED">
On Wed, May 29, 2024 at 2:25 PM John West (via SAGE) <
webservices-bounce<at>lists.ds.iris.edu> wrote:
We might be dealing with two different issues, but perhaps we should
solve one at a time.
Well, OK, but the original issue was primarily about the XML problem. I
occasionally see the malformed gzip issue, but get the XML error multiple
times per day.
I'll look into turning off gzip in the obspy client, thanks!
-- John
On Wed, May 29, 2024 at 12:09 PM Rob Casey <rob.casey<at>earthscope.org>
wrote:
Hi John-
----------------------
I have a couple of questions here: Is the gzip problem and the
malformed XML problem the same issue? Or are we dealing with two different
issues?
We might be dealing with two different issues, but perhaps we should
solve one at a time.
I'm using the obspy client, which handles the requesting in the
background. Is there a way to append headers to that client, or would I
have to rewrite my code to do that?
Not speaking from experience, but I note that the Accept-encoding header
addition is played out in obspy.clients.fdsn.client. I don't see a way to
externally configure this, so to test this, you'd need to modify the code
so that 'use_gzip' is set to false by default. If that isn't enough, then
you'd want to assert "gzip;q=0" where the Accept-encoding header is written
in that code.
Naturally, having the server on our end functioning suffiently well with
gzip turned off would be preferable. I am just not sure yet what the side
effects might be in making this change, including data transfer performance.
-Rob
Thanks!
wrote:
-- John
On Wed, May 29, 2024 at 11:45 AM Rob Casey <rob.casey<at>earthscope.org>
Hi John and Philip-
problem that didn't present obvious solutions over the past couple of
Gillian alerted me to your posts. This has been a long nagging
years. We have had little to not staff time that we have been able to
place on this issue, either as we are deep in infrastructure development in
our new center.
That being said, one of our staffers, Dan, took a crack at
trying to find a pattern and look for a way to circumvent, if not solve.
He got to the stage where we realized that there were not a specific set of
distributed servers that were causing the problem, it could come from any
of the systems, just at indeterminate times. Something with the
fdsnws-station web service call causes it to go to a multipart content mode
that inserts byte size values and seems to trigger gzip output. (thx.
Tobias)
There is a client-based way that Dan suggested could get around
the issue, and that is to have the client request to not receive gzipped
returns. He describes it like so:
-------------------------------------------------------------------------
You can explicitly say "don't send gzip" by adding q=0 like
this:
Accept-Encoding:gzip;q=0
not to use gzip Transfer-Encoding
With the following in my Python code:
headers = {"Accept-Encoding":"gzip;q=0"} # asks server
r = requests.get(url, headers=headers)
-------------------------------------------------------------------------
I get zero issues from station service.
So, you can try this out and see whether you get improved
results. In fact, it would be great to get outside evaulation of this. To
be sustainable, though, we need a server-side solution, so Dan also
suggested that there is a way to configure Tomcat to not opt for gzip
output. This is something that we should be able to try out in beta and
see if that doesn't break something else.
Regards,
webservices-bounce<at>lists.ds.iris.edu> wrote:
-Rob
On May 29, 2024, at 10:23 AM, John West (via SAGE) <
opened an issue on github to track this problem:Tobias is apparently having trouble posting to the mail list. He has
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$(
)
wrote:-- John
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de>
discoveryHi John,
yes, that's what I meant. My best guess was that the service
"normal"was creating the issue, because it's the only time we do multiple
requests in parallel.
But yes, I can see that something also sometimes goes wrong in
torequests after the client initialization. Seems like a server issue
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$me but not 100% sure yet.
Can you maybe mention that github ticket I made on the ML
(
), my mails don't go through,
codemy old IRIS account has my old mail sender address and I made anew
account on EarthScope but no luck with that either.
cheers,
Tobias
On 29/05/2024 17:38, John D. West wrote:
Hi, Tobias.
I changed line 21 of my sample code to: DCclient = Client('IRIS',
_discover_services=False)
Is that what you had in mind? It doesn't make any difference, the
and makesstill generates errors on random networks.
Thanks!
-- John
On Wed, May 29, 2024 at 9:29 AM Tobias Megies <tobias.megies<at>lmu.de
<tobias.megies<at>lmu.de>> wrote:
Hi John,
a quick fix is this:
client = Client(_discover_services=False)
This skips service discovery (parsing the server side WADLs)
should beit so custom parameters Earthscope has can't be used, but it
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$fine unless you do some real fancy requests.
cheers,
Tobias
On 29/05/2024 17:26, Tobias Megies wrote:
I opened an issue on github to track the issue:
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$
expected,cheers,
obspy client.get_stations()
Tobias
On 29/05/2024 04:19, John West (via SAGE) wrote:
Hi!
I'm calling the station service using the
method, and getting randomly distributed badly formed XML.
Typical error message is something like: "Start tag
(b'a\r')"'<' not
found, line 1, column 1 (<string>, line 1)"
I've also had a few errors with: "Not a gzipped file
error, andThe attached sample Python code sometimes runs without
errors aresometimes generates multiple errors during a run. The
each run.NOT
in the same places, they appear for different networks on
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
--Any ideas?
Thanks!
-- John
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$Geophysikalisches Observatorium
--
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
EMail: tobias.megies<at>lmu.de
WWW:
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlv0ti8Wwg$Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
----------------------
Web Services
Topic home:
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlvVI56oCQ$Sent from the IRIS Message Center (
)
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUltH577DAA$Update subscription preferences at
----------------------
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLtKquQ_Q$
Web Services
Topic home:
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YJT1c67lQ$
)
Update subscription preferences at
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLsKF4oPA$
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
-
That's very interesting, Philip.
It turns out that the problem only occurs randomly, so I can deal with it
in my code by implementing retry counters and just retry on failure. Maybe
not elegant, but it works.
-- John
On Wed, May 29, 2024 at 12:51 PM Philip Crotwell <crotwell<at>seis.sc.edu>
wrote:
Hi all
I just did some testing using my seisFile java library (so no obspy
dependency) and have gotten xml parse errors even with the http encoding
header set to be "identity" which I think turns off all compression. Output
xml for network 8A (I think) had the first line as "2000". All 98 previous
networks downloaded parsable stationxml for stations without error.
Rerunning, I hit the same thing, this time for network 2P, again with
"2000" as the first line. Running a third time, I got all the way to XA
before it crashed. See output below with http headers. All three failures
show that same first line of "2000" which seems really odd.
I think at least we can say this isn't an obspy issue.
Philip
Network: XA PANDA -- San Juan, Argentina Experiment (PANDA-ARGENTINA)
http://service.iris.edu:80/fdsnws/station/1/query?endafter=1987-01-01T00:00:00.000Z&level=station&network=XA&startbefore=1988-12-31T23:59:59.999Z
https://urldefense.com/v3/__http://service.iris.edu:80/fdsnws/station/1/query?endafter=1987-01-01T00:00:00.000Z&level=station&network=XA&startbefore=1988-12-31T23:59:59.999Z__;!!IKRxdwAv5BmarQ!Z4EukG5rK6TtAPPEwMWqOJ4WYURuk4aaXt5SswnEH4n3IZKsM0cv-TWJNO0LmsPeGGLbVRIcdE2hWA-nr9Mz-T8a$>
200
Unable to parse as staxml:
HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers:
Access-Control-Allow-Origin,Access-Control-Allow-Credentials
Vary:
origin,access-control-request-method,access-control-request-headers,accept-encoding
Date: Wed, 29 May 2024 18:44:05 GMT
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/xml
Connection: close
Date: Wed, 29 May 2024 18:44:05 GMT
Connection: close
2000
<?xml version="1.0" encoding="ISO-8859-1"?>
<FDSNStationXML xmlns="http://www.fdsn.org/xml/station/1
https://urldefense.com/v3/__http://www.fdsn.org/xml/station/1__;!!IKRxdwAv5BmarQ!Z4EukG5rK6TtAPPEwMWqOJ4WYURuk4aaXt5SswnEH4n3IZKsM0cv-TWJNO0LmsPeGGLbVRIcdE2hWA-nr3S-9tfL$>"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
https://urldefense.com/v3/__http://www.w3.org/2001/XMLSchema-instance__;!!IKRxdwAv5BmarQ!Z4EukG5rK6TtAPPEwMWqOJ4WYURuk4aaXt5SswnEH4n3IZKsM0cv-TWJNO0LmsPeGGLbVRIcdE2hWA-nrx--gNQZ$>"
xmlns:iris="http://www.fdsn.org/xml/station/1/iris
https://urldefense.com/v3/__http://www.fdsn.org/xml/station/1/iris__;!!IKRxdwAv5BmarQ!Z4EukG5rK6TtAPPEwMWqOJ4WYURuk4aaXt5SswnEH4n3IZKsM0cv-TWJNO0LmsPeGGLbVRIcdE2hWA-nrwEZZ7dX$>"
xsi:schemaLocation="http://www.fdsn.org/xml/station/1
https://urldefense.com/v3/__http://www.fdsn.org/xml/station/1__;!!IKRxdwAv5BmarQ!Z4EukG5rK6TtAPPEwMWqOJ4WYURuk4aaXt5SswnEH4n3IZKsM0cv-TWJNO0LmsPeGGLbVRIcdE2hWA-nr3S-9tfL$>
http://www.fdsn.org/xml/station/fdsn-station-1.1.xsd
https://urldefense.com/v3/__http://www.fdsn.org/xml/station/fdsn-station-1.1.xsd__;!!IKRxdwAv5BmarQ!Z4EukG5rK6TtAPPEwMWqOJ4WYURuk4aaXt5SswnEH4n3IZKsM0cv-TWJNO0LmsPeGGLbVRIcdE2hWA-nr6RF1fSh$>"
schemaVersion="1.1">
<Source>IRIS-DMC</Source>
<Sender>IRIS-DMC</Sender>
<Module>IRIS WEB SERVICE: fdsnws-station | version: 1.1.52</Module>
<ModuleURI>
http://service.iris.edu/fdsnws/station/1/query?endafter=1987-01-01T00:00:00.000Z&;level=station&network=XA&startbefore=1988-12-31T23:59:59.999Z
https://urldefense.com/v3/__http://service.iris.edu/fdsnws/station/1/query?endafter=1987-01-01T00:00:00.000Z&;level=station&network=XA&startbefore=1988-12-31T23:59:59.999Z__;!!IKRxdwAv5BmarQ!Z4EukG5rK6TtAPPEwMWqOJ4WYURuk4aaXt5SswnEH4n3IZKsM0cv-TWJNO0LmsPeGGLbVRIcdE2hWA-nr0nIzt3h$>
</ModuleURI>
<Created>2024-05-29T18:44:06.2266</Created>
<Network code="XA" startDate="1987-01-01T00:00:00.0000"
endDate="1988-12-31T23:59:59.9999" restrictedStatus="open">
<Description>PANDA -- San Juan, Argentina Experiment
(PANDA-ARGENTINA)</Description>
<TotalNumberStations>80</TotalNumberStations>
<SelectedNumberStations>80</SelectedNumberStations>
<Station code="I01H" startDate="1987-08-25T00:00:00.0000"
endDate="1988-12-30T23:59:59.0000" restrictedStatus="open"
iris:alternateNetworkCodes="_CERI-TEMPORARY,.UNRESTRICTED">
On Wed, May 29, 2024 at 2:25 PM John West (via SAGE) <
webservices-bounce<at>lists.ds.iris.edu> wrote:
We might be dealing with two different issues, but perhaps we should
solve one at a time.
Well, OK, but the original issue was primarily about the XML problem. I
occasionally see the malformed gzip issue, but get the XML error multiple
times per day.
I'll look into turning off gzip in the obspy client, thanks!
-- John
On Wed, May 29, 2024 at 12:09 PM Rob Casey <rob.casey<at>earthscope.org>
wrote:
Hi John-
----------------------
I have a couple of questions here: Is the gzip problem and the
malformed XML problem the same issue? Or are we dealing with two different
issues?
We might be dealing with two different issues, but perhaps we should
solve one at a time.
I'm using the obspy client, which handles the requesting in the
background. Is there a way to append headers to that client, or would I
have to rewrite my code to do that?
Not speaking from experience, but I note that the Accept-encoding header
addition is played out in obspy.clients.fdsn.client. I don't see a way to
externally configure this, so to test this, you'd need to modify the code
so that 'use_gzip' is set to false by default. If that isn't enough, then
you'd want to assert "gzip;q=0" where the Accept-encoding header is written
in that code.
Naturally, having the server on our end functioning suffiently well with
gzip turned off would be preferable. I am just not sure yet what the side
effects might be in making this change, including data transfer performance.
-Rob
Thanks!
wrote:
-- John
On Wed, May 29, 2024 at 11:45 AM Rob Casey <rob.casey<at>earthscope.org>
Hi John and Philip-
nagging problem that didn't present obvious solutions over the past couple
Gillian alerted me to your posts. This has been a long
of years. We have had little to not staff time that we have been able to
place on this issue, either as we are deep in infrastructure development in
our new center.
That being said, one of our staffers, Dan, took a crack at
trying to find a pattern and look for a way to circumvent, if not solve.
He got to the stage where we realized that there were not a specific set of
distributed servers that were causing the problem, it could come from any
of the systems, just at indeterminate times. Something with the
fdsnws-station web service call causes it to go to a multipart content mode
that inserts byte size values and seems to trigger gzip output. (thx.
Tobias)
There is a client-based way that Dan suggested could get
around the issue, and that is to have the client request to not receive
gzipped returns. He describes it like so:
-------------------------------------------------------------------------
You can explicitly say "don't send gzip" by adding q=0 like
this:
Accept-Encoding:gzip;q=0
not to use gzip Transfer-Encoding
With the following in my Python code:
headers = {"Accept-Encoding":"gzip;q=0"} # asks server
r = requests.get(url, headers=headers)
-------------------------------------------------------------------------
I get zero issues from station service.
So, you can try this out and see whether you get improved
results. In fact, it would be great to get outside evaulation of this. To
be sustainable, though, we need a server-side solution, so Dan also
suggested that there is a way to configure Tomcat to not opt for gzip
output. This is something that we should be able to try out in beta and
see if that doesn't break something else.
Regards,
webservices-bounce<at>lists.ds.iris.edu> wrote:
-Rob
On May 29, 2024, at 10:23 AM, John West (via SAGE) <
opened an issue on github to track this problem:Tobias is apparently having trouble posting to the mail list. He has
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$(
)
wrote:-- John
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de>
discoveryHi John,
yes, that's what I meant. My best guess was that the service
"normal"was creating the issue, because it's the only time we do multiple
requests in parallel.
But yes, I can see that something also sometimes goes wrong in
torequests after the client initialization. Seems like a server issue
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$me but not 100% sure yet.
Can you maybe mention that github ticket I made on the ML
(
), my mails don't go through,
codemy old IRIS account has my old mail sender address and I made anew
account on EarthScope but no luck with that either.
cheers,
Tobias
On 29/05/2024 17:38, John D. West wrote:
Hi, Tobias.
I changed line 21 of my sample code to: DCclient = Client('IRIS',
_discover_services=False)
Is that what you had in mind? It doesn't make any difference, the
tobias.megies<at>lmu.destill generates errors on random networks.
Thanks!
-- John
On Wed, May 29, 2024 at 9:29 AM Tobias Megies <
and makes<tobias.megies<at>lmu.de>> wrote:
Hi John,
a quick fix is this:
client = Client(_discover_services=False)
This skips service discovery (parsing the server side WADLs)
should beit so custom parameters Earthscope has can't be used, but it
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$fine unless you do some real fancy requests.
cheers,
Tobias
On 29/05/2024 17:26, Tobias Megies wrote:
I opened an issue on github to track the issue:
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$
expected,cheers,
obspy client.get_stations()
Tobias
On 29/05/2024 04:19, John West (via SAGE) wrote:
Hi!
I'm calling the station service using the
method, and getting randomly distributed badly formed XML.
Typical error message is something like: "Start tag
(b'a\r')"'<' not
found, line 1, column 1 (<string>, line 1)"
I've also had a few errors with: "Not a gzipped file
error, andThe attached sample Python code sometimes runs without
errors aresometimes generates multiple errors during a run. The
each run.NOT
in the same places, they appear for different networks on
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
--Any ideas?
Thanks!
-- John
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$Geophysikalisches Observatorium
--
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
EMail: tobias.megies<at>lmu.de
WWW:
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlv0ti8Wwg$Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
----------------------
Web Services
Topic home:
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlvVI56oCQ$Sent from the IRIS Message Center (
)
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUltH577DAA$Update subscription preferences at
----------------------
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLtKquQ_Q$
Web Services
Topic home:
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YJT1c67lQ$
)
Update subscription preferences at
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLsKF4oPA$
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!Z4EukG5rK6TtAPPEwMWqOJ4WYURuk4aaXt5SswnEH4n3IZKsM0cv-TWJNO0LmsPeGGLbVRIcdE2hWA-nr9syrOpj$>
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!Z4EukG5rK6TtAPPEwMWqOJ4WYURuk4aaXt5SswnEH4n3IZKsM0cv-TWJNO0LmsPeGGLbVRIcdE2hWA-nr77wkVyv$>
)
Update subscription preferences at http://ds.iris.edu/account/profile/
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!Z4EukG5rK6TtAPPEwMWqOJ4WYURuk4aaXt5SswnEH4n3IZKsM0cv-TWJNO0LmsPeGGLbVRIcdE2hWA-nr8WvQM4L$>
-
Tobias on obspy ticket noted that it looks like the returned data is using
chunked transfer, but failing to set the
Transfer-Encoding: chunked
header. I believe the 2000 is the chunk size. See
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
Philip
On Wed, May 29, 2024 at 2:56 PM Philip Crotwell (via SAGE) <
webservices-bounce<at>lists.ds.iris.edu> wrote:
Hi all
I just did some testing using my seisFile java library (so no obspy
dependency) and have gotten xml parse errors even with the http encoding
header set to be "identity" which I think turns off all compression. Output
xml for network 8A (I think) had the first line as "2000". All 98 previous
networks downloaded parsable stationxml for stations without error.
Rerunning, I hit the same thing, this time for network 2P, again with
"2000" as the first line. Running a third time, I got all the way to XA
before it crashed. See output below with http headers. All three failures
show that same first line of "2000" which seems really odd.
I think at least we can say this isn't an obspy issue.
Philip
Network: XA PANDA -- San Juan, Argentina Experiment (PANDA-ARGENTINA)
http://service.iris.edu:80/fdsnws/station/1/query?endafter=1987-01-01T00:00:00.000Z&level=station&network=XA&startbefore=1988-12-31T23:59:59.999Z
200
Unable to parse as staxml:
HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers:
Access-Control-Allow-Origin,Access-Control-Allow-Credentials
Vary:
origin,access-control-request-method,access-control-request-headers,accept-encoding
Date: Wed, 29 May 2024 18:44:05 GMT
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/xml
Connection: close
Date: Wed, 29 May 2024 18:44:05 GMT
Connection: close
2000
<?xml version="1.0" encoding="ISO-8859-1"?>
<FDSNStationXML xmlns="http://www.fdsn.org/xml/station/1" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance" xmlns:iris="
http://www.fdsn.org/xml/station/1/iris" xsi:schemaLocation="
http://www.fdsn.org/xml/station/1
http://www.fdsn.org/xml/station/fdsn-station-1.1.xsd" schemaVersion="1.1">
<Source>IRIS-DMC</Source>
<Sender>IRIS-DMC</Sender>
<Module>IRIS WEB SERVICE: fdsnws-station | version: 1.1.52</Module>
<ModuleURI>
http://service.iris.edu/fdsnws/station/1/query?endafter=1987-01-01T00:00:00.000Z&;level=station&network=XA&startbefore=1988-12-31T23:59:59.999Z
</ModuleURI>
<Created>2024-05-29T18:44:06.2266</Created>
<Network code="XA" startDate="1987-01-01T00:00:00.0000"
endDate="1988-12-31T23:59:59.9999" restrictedStatus="open">
<Description>PANDA -- San Juan, Argentina Experiment
(PANDA-ARGENTINA)</Description>
<TotalNumberStations>80</TotalNumberStations>
<SelectedNumberStations>80</SelectedNumberStations>
<Station code="I01H" startDate="1987-08-25T00:00:00.0000"
endDate="1988-12-30T23:59:59.0000" restrictedStatus="open"
iris:alternateNetworkCodes="_CERI-TEMPORARY,.UNRESTRICTED">
On Wed, May 29, 2024 at 2:25 PM John West (via SAGE) <
webservices-bounce<at>lists.ds.iris.edu> wrote:
----------------------We might be dealing with two different issues, but perhaps we should
solve one at a time.
Well, OK, but the original issue was primarily about the XML problem. I
occasionally see the malformed gzip issue, but get the XML error multiple
times per day.
I'll look into turning off gzip in the obspy client, thanks!
-- John
On Wed, May 29, 2024 at 12:09 PM Rob Casey <rob.casey<at>earthscope.org>
wrote:
Hi John-
----------------------
I have a couple of questions here: Is the gzip problem and the
malformed XML problem the same issue? Or are we dealing with two different
issues?
We might be dealing with two different issues, but perhaps we should
solve one at a time.
I'm using the obspy client, which handles the requesting in the
background. Is there a way to append headers to that client, or would I
have to rewrite my code to do that?
Not speaking from experience, but I note that the Accept-encoding header
addition is played out in obspy.clients.fdsn.client. I don't see a way to
externally configure this, so to test this, you'd need to modify the code
so that 'use_gzip' is set to false by default. If that isn't enough, then
you'd want to assert "gzip;q=0" where the Accept-encoding header is written
in that code.
Naturally, having the server on our end functioning suffiently well with
gzip turned off would be preferable. I am just not sure yet what the side
effects might be in making this change, including data transfer performance.
-Rob
Thanks!
wrote:
-- John
On Wed, May 29, 2024 at 11:45 AM Rob Casey <rob.casey<at>earthscope.org>
Hi John and Philip-
nagging problem that didn't present obvious solutions over the past couple
Gillian alerted me to your posts. This has been a long
of years. We have had little to not staff time that we have been able to
place on this issue, either as we are deep in infrastructure development in
our new center.
That being said, one of our staffers, Dan, took a crack at
trying to find a pattern and look for a way to circumvent, if not solve.
He got to the stage where we realized that there were not a specific set of
distributed servers that were causing the problem, it could come from any
of the systems, just at indeterminate times. Something with the
fdsnws-station web service call causes it to go to a multipart content mode
that inserts byte size values and seems to trigger gzip output. (thx.
Tobias)
There is a client-based way that Dan suggested could get
around the issue, and that is to have the client request to not receive
gzipped returns. He describes it like so:
-------------------------------------------------------------------------
You can explicitly say "don't send gzip" by adding q=0 like
this:
Accept-Encoding:gzip;q=0
not to use gzip Transfer-Encoding
With the following in my Python code:
headers = {"Accept-Encoding":"gzip;q=0"} # asks server
r = requests.get(url, headers=headers)
-------------------------------------------------------------------------
I get zero issues from station service.
So, you can try this out and see whether you get improved
results. In fact, it would be great to get outside evaulation of this. To
be sustainable, though, we need a server-side solution, so Dan also
suggested that there is a way to configure Tomcat to not opt for gzip
output. This is something that we should be able to try out in beta and
see if that doesn't break something else.
Regards,
webservices-bounce<at>lists.ds.iris.edu> wrote:
-Rob
On May 29, 2024, at 10:23 AM, John West (via SAGE) <
opened an issue on github to track this problem:Tobias is apparently having trouble posting to the mail list. He has
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$(
)
wrote:-- John
On Wed, May 29, 2024 at 10:59 AM Tobias Megies <tobias.megies<at>lmu.de>
discoveryHi John,
yes, that's what I meant. My best guess was that the service
"normal"was creating the issue, because it's the only time we do multiple
requests in parallel.
But yes, I can see that something also sometimes goes wrong in
torequests after the client initialization. Seems like a server issue
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnNNFvzDT$me but not 100% sure yet.
Can you maybe mention that github ticket I made on the ML
(
), my mails don't go through,
codemy old IRIS account has my old mail sender address and I made anew
account on EarthScope but no luck with that either.
cheers,
Tobias
On 29/05/2024 17:38, John D. West wrote:
Hi, Tobias.
I changed line 21 of my sample code to: DCclient = Client('IRIS',
_discover_services=False)
Is that what you had in mind? It doesn't make any difference, the
tobias.megies<at>lmu.destill generates errors on random networks.
Thanks!
-- John
On Wed, May 29, 2024 at 9:29 AM Tobias Megies <
and makes<tobias.megies<at>lmu.de>> wrote:
Hi John,
a quick fix is this:
client = Client(_discover_services=False)
This skips service discovery (parsing the server side WADLs)
should beit so custom parameters Earthscope has can't be used, but it
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$fine unless you do some real fancy requests.
cheers,
Tobias
On 29/05/2024 17:26, Tobias Megies wrote:
I opened an issue on github to track the issue:
<
https://urldefense.com/v3/__https://github.com/obspy/obspy/issues/3465__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpakDxgJgS$
expected,cheers,
obspy client.get_stations()
Tobias
On 29/05/2024 04:19, John West (via SAGE) wrote:
Hi!
I'm calling the station service using the
method, and getting randomly distributed badly formed XML.
Typical error message is something like: "Start tag
(b'a\r')"'<' not
found, line 1, column 1 (<string>, line 1)"
I've also had a few errors with: "Not a gzipped file
error, andThe attached sample Python code sometimes runs without
errors aresometimes generates multiple errors during a run. The
each run.NOT
in the same places, they appear for different networks on
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
--Any ideas?
Thanks!
-- John
EMail: tobias.megies<at>lmu.de <tobias.megies<at>lmu.de>
WWW:
<
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!YbAqxDBWoNH9JDXLefMCt-N51qul5iRqXOqZKQeXCts8UL-jBXiT5LuOqCUjiwxkfp-C3ui9CONDDvIMI3gpahg3U8Zc$
https://urldefense.com/v3/__https://www.geophysik.uni-muenchen.de__;!!IKRxdwAv5BmarQ!by8qJhcaNx0UirQy7fS_eIIjHB8u0U2EWHK7pbf4dlNmM_b6xEZRmMWLnzNsh2K773S2lj2fV1OqRhKvC-GfnHnkp9Zx$Geophysikalisches Observatorium
--
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
EMail: tobias.megies<at>lmu.de
WWW:
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlv0ti8Wwg$Geophysikalisches Observatorium
Ludwigshöhe 8
82256 Fürstenfeldbruck
Tel: +49 (0) 89 2180-73981
Fax: +49 (0) 89 2180-73970
Ludwig-Maximilians-Universität
Department für Geo- und Umweltwissenschaften
Sektion Geophysik
Theresienstrasse 41/IV
80333 München
Tel: +49 (0) 89 2180-4237
Fax: +49 (0) 89 2180-4205
----------------------
Web Services
Topic home:
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUlvVI56oCQ$Sent from the IRIS Message Center (
)
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ZKNqX6iReO0OgYUlhqyEpRvYxt3YTtGoFI06JkMiDNgcbH3GUwMXA-YyByyq8-DY4vCSNSBFl7kPa8CKUltH577DAA$Update subscription preferences at
----------------------
https://urldefense.com/v3/__http://ds.iris.edu/message-center/topic/webservices/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLtKquQ_Q$
Web Services
Topic home:
| Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (
https://urldefense.com/v3/__http://ds.iris.edu/message-center/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YJT1c67lQ$
)
Update subscription preferences at
https://urldefense.com/v3/__http://ds.iris.edu/account/profile/__;!!IKRxdwAv5BmarQ!ff2GOGJ6NGGxWxHIQrfg_7RJBjGmFvZT5n8_7EuLAk23whvuOJPScoj2zvQYJffy2ZAQ0hjTfN845lny5YLsKF4oPA$
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
-
-
-
-
-
Rob Casey
Philip Crotwell
