[webservices] includeavailability

[Chad Trabant]

Hi Joachim,

> On Nov 27, 2014, at 7:43 AM, Joachim Saul <saul at gfz-potsdam.de> wrote:
> 
> Hi Chad
> 
> Chad Trabant wrote on 11/26/14 21:25:
>> The data availability sub-system at the DMC is updated every 12 hours
>> for archive data.  Data is added to the archive (i.e. from the real time
>> collection system) on a regular basis and is usually all in place ~18
>> hours after arriving, often sooner, but it varies depending on system
>> load.  The data availability you see our our fdsnws-station service is
>> determined by the combination of those activities.
>> 
>> To report the near real-time data as available when setting
>> matchtimeseries=TRUE, our service assumes that data that has been
>> archived within the last 36 hours is still flowing into the data center.
>>  This is documented in the Data Availability section of the Help for
>> the service: http://service.iris.edu/fdsnws/station/docs/1/help/
> 
> Indeed, thanks for pointing that out. It is now clear that this behaviour is not a mistake, so I can work around it in my client.
> 
> On the other hand, quoting from the documentation:
> 
> "Extents are not modified in real-time. The archive will likely be out of sync by up to a day, meaning:
> 
>    The service assumes that if channel data was archived within the last 36 hours, then data for the last 36 hours is available."
> 
> Based on these assumptions and considering that any end time within the last 36 hours is equivalent to "this stream is probably producing near-real-time data now", wouldn't it be safe to set the end time of the availability time span to something very far in the future? I do understand that it may look odd to claim availability of not yet recorded data... but a similar trick is already used for the end times in other contexts, like e.g. "2500-12-31T23:59:59" as the "end time" for the IU network. One might also leave the end time unset to indicate "open end".

Not a bad idea.  But then we could not communicate what we are certain to be a valid latest time, i.e. the archive holdings.

> This would eliminate the need for additional logic at the client code. Implementation of such a logic requires some knowledge of the operational internals of the particular server, which may also be subject to changes.

If the station service is used by a client to prepare for a time series request, which I believe is very common, the 'matchtimeseries' solves the issue.  In effect, our fdsnws-station service is already doing the time window matching logic when using that option, the client does not need to do this again.  In fact, the client does not even need to request the availability information, just set matchtimeseries=true and assume whatever comes back intersects with data availably (this is exactly what our internal data extraction routine is doing).

Of course there are other use cases for data availability where you might need to know the actual date-times.  In my opinion, your proposed change would be handier for some but at the cost of others.  If you described what you were doing we can keep it in mind for consideration.

Chad