Thread: RE: More Stations service problems

Hello Andres,

Thank you for the link. I think you are right. I will check our python code and see if I can uncover a cause in the client, or any other information.

Regards,
Matthew

-----Original Message-----
From: Andres Heinloo <andres<at>gfz-potsdam.de>
Sent: Thursday, 9 February 2023 12:31 PM
To: Matthew Hoyles <Matthew.Hoyles<at>ga.gov.au>; webservices<at>lists.ds.iris.edu; Matthew Hoyles (via IRIS) <webservices-bounce<at>lists.ds.iris.edu>
Subject: Re: [IRIS][webservices] More Stations service problems

Hi,

IMHO everything points to chunked encoding used wrongly somehow.

See: https://en.wikipedia.org/wiki/Chunked_transfer_encoding#Format

Regards,
Andres



On Wed, 08 Feb 2023 23:32:02 -0000
"Matthew Hoyles (via IRIS)" <webservices-bounce<at>lists.ds.iris.edu>
wrote:

Hello Rob,

A little extra information: the line breaks surrounding the inserted
hex numbers are <cr> <nl> (Windows style), not <nl> (Unix/Linux
style) used for the rest of the XML. This might indicate a process
running on a Windows machine is doing the inserting.

Regards,
Matthew

Matthew Hoyles
DevOps Engineer, Nuclear Monitoring  |  Information Services Place,
Space and Communities Division  |  GEOSCIENCE AUSTRALIA

-----Original Message-----
From: webservices-bounce<at>lists.ds.iris.edu
<webservices-bounce<at>lists.ds.iris.edu>
Sent: Thursday, 9 February 2023 8:49 AM
To: Web Services <webservices<at>lists.ds.iris.edu>
Subject: Re: [IRIS][webservices] More Stations service problems


Hi Sebastian-

Thank you for reporting this!

One of our staff also saw these 2000's pop up. I thought that it
might have been the intermixing of curl download counter output getting
worked into the stream, but here you are using Python and I am assuming
that there is not additional output being generated from your client.

Very good analysis that this number is actually 0x2000, which would
be 8192 or 8K bytes. Very odd that this happens at all, but happens
seemingly randomly. Given this information, I wonder if the developers
I have CC'ed have any theories as to why 8K is being sent with the
payload parts?

Regards,

-Rob

On Feb 8, 2023, at 7:21 AM, Sebastian Heimann (via IRIS)
<webservices-bounce<at>lists.ds.iris.edu> wrote:

Retrying, sorry my previous mail got incorrectly formatted probably
because of the included PGP signature.


Hi!

I think I am running into the same type of problems as has been
reported previously. I find that about 1 in 30 requests returns
corrupt StationXML. Looks like there is e.g. a stray debug print
statement in the server code, printing the size of the following block
as a hex value.

Here's an excerpt from one of the corrupt results (the `2000` lines
are the problem):

```
2000
<?xml version="1.0" encoding="ISO-8859-1"?>

<FDSNStationXML xmlns="http://www.fdsn.org/xml/station/1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:iris="http://www.fdsn.org/xml/station/1/iris"
xsi:schemaLocation="http://www.fdsn.org/xml/station/1
http://www.fdsn.org/xml/station/fdsn-station-1.1.xsd"
schemaVersion="1.1">
<Source>IRIS-DMC</Source>

...

<NumeratorCoefficient
i="12">-9.78385E-7</NumeratorCoefficient>
<NumeratorCoefficient
i="13">3.18969E-5</NumeratorCoefficient>
<NumeratorCoefficient
i="14">1.68242E-4</NumeratorCoefficient>
<Nume
2000
ratorCoefficient i="15">-3.49811E-4</NumeratorCoefficient>
<NumeratorCoefficient
i="16">-5.52227E-4</NumeratorCoefficient>
<NumeratorCoefficient
i="17">7.31389E-4</NumeratorCoefficient>
<NumeratorCoefficient
i="18">-0.00108921</NumeratorCoefficient>
<NumeratorCoefficient
i="19">7.87631E-4</NumeratorCoefficient>

...

```

All blocks have size 0x2000 except for the last one.

Additionally the end a superfluous `0` is printed, I guess the size
of
the print buffer is printed under some conditions to the response.

```
</Response>
</Channel>
</Station>
</Network>
</FDSNStationXML>
0
```

Strange that this only happens on about every 30th request.
Repeating
the request with the same parameters usually returns a correct
result.

Hope this helps to find the problem.

Best wishes
Sebastian




PS: here is some hotfix code for the client side to clean the
possibly
corrupt StationXML after retrieval:


import io

def fix_iris_bug(data):
fin = io.BytesIO(data)
fout = io.BytesIO()
start = True
buggy = False
while True:
line = fin.readline()
if not line:
break

if start:
try:
iexpect = int(line, 16) + len(line)
buggy = True
ipos = 0
except ValueError:
pass

start = False

else:
ipos += len(line)
if buggy and ipos >= iexpect:
iexpect = int(line, 16) + len(line)
ipos = 0
else:
fout.write(line)

return fout.getvalue()



----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu

Sent from the IRIS Message Center
(http://ds.iris.edu/message-center/)
Update subscription preferences at
http://ds.iris.edu/account/profile/

----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu

Sent from the IRIS Message Center
(http://ds.iris.edu/message-center/)
Update subscription preferences at
http://ds.iris.edu/account/profile/

Geoscience Australia Disclaimer: This e-mail (and files transmitted
with it) is intended only for the person or entity to which it is
addressed. If you are not the intended recipient, then you have
received this e-mail by mistake and any use, dissemination,
forwarding, printing or copying of this e-mail and its file
attachments is prohibited. The security of emails transmitted cannot
be guaranteed; by forwarding or replying to this email, you
acknowledge and accept these risks.


----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu

Sent from the IRIS Message Center
(http://ds.iris.edu/message-center/)
Update subscription preferences at
http://ds.iris.edu/account/profile/

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.