Thread: Fwd: MUSTANG pdf problem

Hi Kyle and Carl,

We have tracked down the problem to the rate that your script is making
requests of the web service. The load balancer has set limits to the number
of connections per minute that can be handled, and this is sometimes being
exceeded. The simplest solution might be to slow the script down slightly
by using pause or sleep functions. We don’t have a better solution on our
end at this time.

cheers,
Gillian

On May 6, 2019, at 5:11 PM, Carl Tape <ctape<at>alaska.edu> wrote:

Thanks, Gillian. With the MUSTANG-generated spectrograms, we can compare
ours vs yours. So we're now in better position to document the issue, the
next time it arises. Great to hear that you have log files, too.

Carl

On Tue, May 7, 2019 at 6:30 AM Gillian Sharer <gillian<at>iris.washington.edu>
wrote:

Hi Kyle,

Thank you for the additional information. I assume you didn’t see any
errors when you ran your script (such as http failed connection or timeout
messages). We were not able to retrieve the log files for your requests but
we have increased the number of days that we retain logs before deleting
them. When you encounter this problem again, please let us know so that we
can check for errors in the logs. We have not been able to replicate this
behavior yet, but we’ll keep looking for a solution.

Best regards,
Gillian

On May 3, 2019, at 5:31 PM, Kyle Smith <todakozhi<at>gmail.com> wrote:

The data is being downloaded one day at a time and a MATLAB program is
doing it with the urlread function. My program loops over many days to get
the data. In the past I have checked data completeness manually for one
day and it works fine one day but there are issues for another day.

Thanks,

Kyle

On Fri, May 3, 2019 at 4:04 PM Carl Tape <ctape<at>alaska.edu> wrote:

Dear Kyle and Gillian,

I'm shifting this discussion to mustang-users, since my understanding is
that this is the correct place for it (not mustang-qa, which is an
announcement board from the DMC to users).

I have attached Kyle's spectrograms here. Note that the missing data do
not correspond to an epoch change. There are other cases where there is a
gap in horizontal but not in vertical, even though the 3-component
waveforms are all complete.

I can reiterate that the number of PSDs that Kyle receives from MUSTANG
appears to be highly time-dependent, something that would be great to
resolve. Usually we detect a swatch of missing PSDs within his home-made
spectrograms, then he retries the download another time and everything is
there. He's been using PSDs for about 4 years now, and we're grateful for
this data product!

Thanks,
Carl

--------------------------------------------------------------
Carl Tape
Associate Professor
Geophysical Institute (office 413D)
University of Alaska Fairbanks
Email: ctape<at>alaska.edu
Web: https://sites.google.com/alaska.edu/carltape
--------------------------------------------------------------

---------- Forwarded message ---------
From: Gillian Sharer <gillian<at>iris.washington.edu>
Date: Sat, May 4, 2019 at 4:53 AM
Subject: Re: [IRIS][mustang-qa] MUSTANG pdf problem
To: MUSTANG QA <mustang-qa<at>lists.ds.iris.edu>


Hi Kyle,

The MUSTANG PDF webservice should not have missing data (except for a
recently identified issue that affects days that have a metadata-epoch
change; this is in the process of being fixed) and data availability should
not change depending on the day. The fact that you are observing this is
concerning to us.

Can you give me more information about how you make requests from the
noise-pdf service to create the spectrogram? There was a problem that
occurred between March-18 to April-19-2019 that potentially affected
multiple-day PDF requests, but it should not have affected single day
requests.

Best regards,
Gillian

On May 3, 2019, at 12:34 PM, Kyle Smith <todakozhi<at>gmail.com> wrote:

Hi MUSTANG QA folks:

I am having trouble downloading some data from the SALMON projects (ZE
network) from MUSTANG pdf webservice:

http://services.iris.edu/mustang/noise-pdf/1/

The data is used to create a spectrogram with a resolution of 1 day. My
example is shown in the attached file WFLW_pdf_spectrogram.png. However,
sometimes time periods can be missing. Data availability can change
depending on the day. I compared my downloads to the new MUSTANG
spectrogram tool in:

http://services.iris.edu/mustang/noise-spectrogram/1/

and got a complete set of data shown in
WFLW_Z_Spectrogram_IRIS_MUSTANG.png. I also experience the same issue with
TA data. My downloaded data was from a few weeks ago but today I checked
the data again and it is available.

My questions are:

Why does the MUSTANG pdf webservice have missing data when there is not
supposed to be?
Why does data availability change depending on the day?
Why does the MUSTANG spectrogram tool have a more complete set of data?

Thank You,

Kyle Smith
<WFLW_Z_Spectrogram_IRIS_MUSTANG.png><WFLW_pdf_spectrogram.png>

---------------------------------------------------
Gillian Sharer
Deputy Director, Quality Assurance, IRIS DMC
gillian<at>iris.washington.edu
206-547-0393
---------------------------------------------------






----------------------
MUSTANG Users
Topic home: http://ds.iris.edu/message-center/topic/mustang-users/ |
Unsubscribe: mustang-users-unsubscribe<at>lists.ds.iris.edu

Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/