[webservices] Fwd: dataselect wrong data returned to multithreaded client

Thu Jul 19 11:17:21 PDT 2012

Hi

I don't think I can tell if it was a collision or not. All I know is
that one request in one thread got data for a different station. It is
true that another thread on my end asked for data from that other
station at a similar time, but I am not sure if it was actually
coincidental or not. Because I was doing a test using many threads on
my end (and hence many http connections), I assumed that it was caused
by that, but I actually don't know. Given I can't seem to reproduce
it, I am not sure what to say.

I am leaving the "is the answer from the same station as the request"
check in my code, so hopefully I will notice if it happens again.

Debugging stuff that you can't reproduce is somewhere between really,
really hard and impossible!

thanks,
Philip

On Thu, Jul 19, 2012 at 1:53 PM, Bruce Weertman
<bruce at iris.washington.edu> wrote:
> Philip:
>
> Just to get back to you - and everybody else on the list.
>
> Are you saying is  thread-1 and thread-2 are each making requests, but
> thread-1 gets the data that thread-2 made.
>
> Or something along that line?
>
> Possibilities:
>
> (1) Collision of token values. Token is generated from the request via a
> standard hashing algorithm.
>
> (2) In the java servlet one request gets confused with another request. This
> could happen if there was a mistake
> where the wrong scope was used for variables. For example improperly using
> variables from the servlet instance and not off
> of the stack or accidentally using a global (ie static) variable. Code
> review looks OK. But that's just code review.
>
> (3) Some sort of weird networking issue related to reusing http
> connections??
>
> I wrote a script to scan the cache (had 34,000 items in it) looking for a
> mis-match between the data and the requests,
> but I didn't find any. A necessary but not sufficient test of the cache
> working properly.
>
>
> Are you saying that you saw these requests collide
> http://www.iris.edu/ws/dataselect/query?net=II&sta=BORG&loc=00&cha=BHZ&start=2009-05-16T01:08:44&end=2009-05-16T01:26:26
> http://www.iris.edu/ws/dataselect/query?net=II&sta=ARU&loc=00&cha=BHZ&start=2009-05-16T01:08:44&end=2009-05-16T01:26:26
>
> I don't see that:
>
> geodude:tmp bruce$ wget -O aru.dat
> "http://www.iris.edu/ws/dataselect/query?net=II&sta=ARU&loc=00&cha=BHZ&start=2009-05-16T01:08:44&end=2009-05-16T01:26:26"
> --2012-07-19 10:19:45--
> http://www.iris.edu/ws/dataselect/query?net=II&sta=ARU&loc=00&cha=BHZ&start=2009-05-16T01:08:44&end=2009-05-16T01:26:26
> Resolving www.iris.edu... 128.95.166.129
> Connecting to www.iris.edu|128.95.166.129|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 28672 (28K) [application/vnd.fdsn.mseed]
> Saving to: `aru.dat'
>
> 100%[==================================================================================================================================================================>]
> 28,672      --.-K/s   in 0.002s
>
> 2012-07-19 10:19:45 (11.3 MB/s) - `aru.dat' saved [28672/28672]
>
> geodude:tmp bruce$ msi aru.dat
> II_ARU_00_BHZ, 000001, M, 4096, 2894 samples, 19.99993515 Hz,
> 2009,136,01:08:44.015300
> II_ARU_00_BHZ, 000000, M, 4096, 3104 samples, 19.99993515 Hz,
> 2009,136,01:11:08.715900
> II_ARU_00_BHZ, 000000, M, 4096, 3446 samples, 19.99993515 Hz,
> 2009,136,01:13:43.916400
> II_ARU_00_BHZ, 000000, M, 4096, 3520 samples, 19.99993515 Hz,
> 2009,136,01:16:36.217000
> II_ARU_00_BHZ, 000000, M, 4096, 3712 samples, 19.99993515 Hz,
> 2009,136,01:19:32.217500
> II_ARU_00_BHZ, 000000, M, 4096, 3712 samples, 19.99993515 Hz,
> 2009,136,01:22:37.818100
> II_ARU_00_BHZ, 000001, M, 4096, 852 samples, 19.99993515 Hz,
> 2009,136,01:25:43.418700
>
>
> geodude:tmp bruce$ wget -O borg.dat
> "http://www.iris.edu/ws/dataselect/query?net=II&sta=BORG&loc=00&cha=BHZ&start=2009-05-16T01:08:44&end=2009-05-16T01:26:26"
> --2012-07-19 10:18:51--
> http://www.iris.edu/ws/dataselect/query?net=II&sta=BORG&loc=00&cha=BHZ&start=2009-05-16T01:08:44&end=2009-05-16T01:26:26
> Resolving www.iris.edu... 128.95.166.129
> Connecting to www.iris.edu|128.95.166.129|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 45056 (44K) [application/vnd.fdsn.mseed]
> Saving to: `borg.dat'
>
> 100%[==================================================================================================================================================================>]
> 45,056      --.-K/s   in 0.004s
>
> geodude:tmp bruce$ msi borg.dat
> II_BORG_00_BHZ, 000001, M, 4096, 762 samples, 20 Hz,
> 2009,136,01:08:44.049900
> II_BORG_00_BHZ, 000000, M, 4096, 2204 samples, 20 Hz,
> 2009,136,01:09:22.149900
> II_BORG_00_BHZ, 000000, M, 4096, 2120 samples, 20 Hz,
> 2009,136,01:11:12.349900
> II_BORG_00_BHZ, 000000, M, 4096, 2024 samples, 20 Hz,
> 2009,136,01:12:58.349900
> II_BORG_00_BHZ, 000000, M, 4096, 2170 samples, 20 Hz,
> 2009,136,01:14:39.549900
> II_BORG_00_BHZ, 000000, M, 4096, 2122 samples, 20 Hz,
> 2009,136,01:16:28.049900
> II_BORG_00_BHZ, 000000, M, 4096, 2100 samples, 20 Hz,
> 2009,136,01:18:14.149900
> II_BORG_00_BHZ, 000000, M, 4096, 2142 samples, 20 Hz,
> 2009,136,01:19:59.149900
> II_BORG_00_BHZ, 000000, M, 4096, 2172 samples, 20 Hz,
> 2009,136,01:21:46.249900
> II_BORG_00_BHZ, 000000, M, 4096, 2138 samples, 20 Hz,
> 2009,136,01:23:34.849900
> II_BORG_00_BHZ, 000001, M, 4096, 1286 samples, 20 Hz,
> 2009,136,01:25:21.749900
>
> Any more information would be appreciated.
>
> Thanks,
> -Bruce
>
>
> Begin forwarded message:
>
>
>
> On 2012/07/16 22:29, Philip Crotwell wrote:
>
> Hi
>
> I have been doing some tests with the dataselect web service from a
> multithreaded client (SOD) and found that occasionally I would get
> data records back in one thread that were not the channel asked for.
> Luckily I had a check in place from some earlier issues that compared
> the channel codes in the request with those in the result. Here is an
> example. It appears that the thread that requested data from BORG got
> data from ARU.
>
>
> 2012-07-16 15:51:26,794 - Channel id in returned seismogram doesn not
> match channelid in request.
> req=II.1986-01-01T00:00:00.000GMT.BORG.00.BHZ.2003-08-29T00:00:00.000GMT
> seis=II.1986-01-01T00:00:00.000GMT.ARU.00.BHZ.2003-08-29T00:00:00.000GMT
>
> The request, if it helps, was
> net=II&sta=BORG&loc=00&cha=BHZ&start=2009-05-16T01:08:44&end=2009-05-16T01:26:26
>
> This is a bit scary as I suspect many clients do not check the
> returned data to make sure it actually corresponds to their original
> request. I have not seen this with a single threaded client, so
> perhaps that is a clue.
>
> I will try to do some more testing and see if I can provide any more
> details.
>
> thanks,
> Philip
> _______________________________________________
> webservices mailing list
>
> webservices at iris.washington.edu
> http://www.iris.washington.edu/mailman/listinfo/webservices
>
>
>
>
>
> --------------------------------------
> Bruce R Weertman
> Software Engineer
> IRIS DMC
> 1408 NE 45h St, Suite 201
> Seattle, WA 98105
>
> bruce at iris.washington.edu
> (206) 547-0393
>
>
> _______________________________________________
> webservices mailing list
> webservices at iris.washington.edu
> http://www.iris.washington.edu/mailman/listinfo/webservices
>