Thread: RE: More Stations service problems

Hello Rob,

A little extra information: the line breaks surrounding the inserted hex numbers are <cr> <nl> (Windows style), not <nl> (Unix/Linux style) used for the rest of the XML. This might indicate a process running on a Windows machine is doing the inserting.

Regards,
Matthew

Matthew Hoyles
DevOps Engineer, Nuclear Monitoring  |  Information Services
Place, Space and Communities Division  |  GEOSCIENCE AUSTRALIA

-----Original Message-----
From: webservices-bounce<at>lists.ds.iris.edu <webservices-bounce<at>lists.ds.iris.edu>
Sent: Thursday, 9 February 2023 8:49 AM
To: Web Services <webservices<at>lists.ds.iris.edu>
Subject: Re: [IRIS][webservices] More Stations service problems


Hi Sebastian-

Thank you for reporting this!

One of our staff also saw these 2000's pop up. I thought that it might have been the intermixing of curl download counter output getting worked into the stream, but here you are using Python and I am assuming that there is not additional output being generated from your client.

Very good analysis that this number is actually 0x2000, which would be 8192 or 8K bytes. Very odd that this happens at all, but happens seemingly randomly. Given this information, I wonder if the developers I have CC'ed have any theories as to why 8K is being sent with the payload parts?

Regards,

-Rob

On Feb 8, 2023, at 7:21 AM, Sebastian Heimann (via IRIS) <webservices-bounce<at>lists.ds.iris.edu> wrote:

Retrying, sorry my previous mail got incorrectly formatted probably
because of the included PGP signature.


Hi!

I think I am running into the same type of problems as has been
reported previously. I find that about 1 in 30 requests returns
corrupt StationXML. Looks like there is e.g. a stray debug print
statement in the server code, printing the size of the following block as a hex value.

Here's an excerpt from one of the corrupt results (the `2000` lines
are the problem):

```
2000
<?xml version="1.0" encoding="ISO-8859-1"?>

<FDSNStationXML xmlns="http://www.fdsn.org/xml/station/1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:iris="http://www.fdsn.org/xml/station/1/iris"
xsi:schemaLocation="http://www.fdsn.org/xml/station/1
http://www.fdsn.org/xml/station/fdsn-station-1.1.xsd" schemaVersion="1.1">
<Source>IRIS-DMC</Source>

...

<NumeratorCoefficient i="12">-9.78385E-7</NumeratorCoefficient>
<NumeratorCoefficient i="13">3.18969E-5</NumeratorCoefficient>
<NumeratorCoefficient i="14">1.68242E-4</NumeratorCoefficient>
<Nume
2000
ratorCoefficient i="15">-3.49811E-4</NumeratorCoefficient>
<NumeratorCoefficient i="16">-5.52227E-4</NumeratorCoefficient>
<NumeratorCoefficient i="17">7.31389E-4</NumeratorCoefficient>
<NumeratorCoefficient i="18">-0.00108921</NumeratorCoefficient>
<NumeratorCoefficient
i="19">7.87631E-4</NumeratorCoefficient>

...

```

All blocks have size 0x2000 except for the last one.

Additionally the end a superfluous `0` is printed, I guess the size of
the print buffer is printed under some conditions to the response.

```
</Response>
</Channel>
</Station>
</Network>
</FDSNStationXML>
0
```

Strange that this only happens on about every 30th request. Repeating
the request with the same parameters usually returns a correct result.

Hope this helps to find the problem.

Best wishes
Sebastian




PS: here is some hotfix code for the client side to clean the possibly
corrupt StationXML after retrieval:


import io

def fix_iris_bug(data):
fin = io.BytesIO(data)
fout = io.BytesIO()
start = True
buggy = False
while True:
line = fin.readline()
if not line:
break

if start:
try:
iexpect = int(line, 16) + len(line)
buggy = True
ipos = 0
except ValueError:
pass

start = False

else:
ipos += len(line)
if buggy and ipos >= iexpect:
iexpect = int(line, 16) + len(line)
ipos = 0
else:
fout.write(line)

return fout.getvalue()



----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu

Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/

----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ | Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu

Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.