Thread: New ws-bulkdataselect revision released

Started: 2011-04-20 23:30:11
Last activity: 2011-08-10 01:35:05
Topics: Web Services
Chad Trabant
2011-04-20 23:30:11

Hello webservice users,

The DMC has updated it's ws-bulkdataselect service: http://www.iris.edu/ws/bulkdataselect/

The new version adds two optional query parameters:

minimumlength
Enforce minimum segment length (seconds).
Only time-series segments of this length or longer will be returned.

longestonly
If specified, only the longest continuous segment for each channel is
returned.

These new parameters can be used to reduce or avoid receiving data containing gaps.

The command line FetchBulkData client script that uses this service to request data has been updated with options to specify these new parameters, the latest client scripts are available here:
http://www.iris.edu/ws/wsclients/

Usage details: like the quality parameter, the two new parameters may be specified in the main selection list or separately as post parameters.

For example, this selection will return only the longest data, that is at least one day (86400 seconds) long:

minimumlength 86400.0
longestonly
TA A25A -- BHZ 2010-084T00:00:00 2010-091T00:00:00
TA A25A -- BHE 2010-084T00:00:00 2010-091T00:00:00

regards
Web services team.


  • Philip Crotwell
    2011-04-22 16:51:30
    Hi Chad

    I guess it is only natural that as these services are new, there will
    be some changes. It looks like this one is backwards compatible, but
    the previous one to the station web service wasn't and would cause
    problems for any other clients out there.

    Have you all given any thought to a longer term versioning for these
    systems? Here is the problem I worry about. I put code into a software
    package to access these web services. My code is obviously tuned to
    the way the web service works at the time I write and test it. Later,
    after I have made a release, IRIS makes a non-backwards compatible
    change. Then, my client breaks and it looks to the rest of the world
    like I have written buggy code and it can't be fixed without a new
    release of my client. I also have little to no forewarning that the
    change is coming, and therefore have little ability to be proactive in
    what I do.

    If you change the IRIS web page, people just notice the change and
    figure things out. They may like or dislike the change, but they can
    adapt. Web services are fundamentally different than web pages. A
    small change that would cause no problem for a person can easily cause
    a web service client to completely fail.

    I guess it feels to me like once you put out a web service that will
    be called by third party clients, you have some obligation to either
    make only backward compatible changes, or to run the old version and
    the new version in parallel for some time to allow an orderly
    transition. Given the development cycle for clients, my guess is this
    might be closer to years than weeks. Although the IDL revision for
    fissures/dhi was never implemented, we addressed this issue in DHI by
    encoding the IDL version number into the name service lookup system,
    so an IDL 1.0 client would be able to find IDL1.0 servers and would
    not accidentally try to connect to IDL 2.0 servers. Obviously web
    services will use different mechanisms, but if they are both going to
    be used and to evolve at the same time, then you have to provide some
    means for the communications protocol contract to remain stable.
    Otherwise every change, even small ones, can and will royally screw
    your users.

    This versioning could easily be accomplished in several ways, either
    putting a version number in the url, say
    http://www.iris.edu/ws/1.0/bulkdataselect/
    or as a parameter
    http://www.iris.edu/ws/bulkdataselect/?version=1.0
    The former has they advantage of you leaving the old server up and
    just adding new ones, while the second allows you to be fancy in your
    implementation to tailor the output to the client. I guess I prefer
    the stability of the first, but as a client writer I can deal with
    either.

    thanks,
    Philip

    On Wed, Apr 20, 2011 at 7:30 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:

    Hello webservice users,

    The DMC has updated it's ws-bulkdataselect service:
    http://www.iris.edu/ws/bulkdataselect/

    The new version adds two optional query parameters:

    minimumlength
    Enforce minimum segment length (seconds).
    Only time-series segments of this length or longer will be returned.

    longestonly
    If specified, only the longest continuous segment for each channel is
    returned.

    These new parameters can be used to reduce or avoid receiving data
    containing gaps.
    The command line FetchBulkData client script that uses this service to
    request data has been updated with options to specify these new parameters,
    the latest client scripts are available here:
    http://www.iris.edu/ws/wsclients/
    Usage details: like the quality parameter, the two new parameters may be
    specified in the main selection list or separately as post parameters.
    For example, this selection will return only the longest data, that is at
    least one day (86400 seconds) long:

    minimumlength 86400.0
    longestonly
    TA A25A -- BHZ 2010-084T00:00:00 2010-091T00:00:00
    TA A25A -- BHE 2010-084T00:00:00 2010-091T00:00:00
    regards
    Web services team.


    _______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices




    • Chad Trabant
      2011-04-22 17:45:26

      Hi Philip,

      We have discussed how to deal with versioned services/results including both of the suggestions you make below. We landed at using different base URLs for major changes in API or content, either adding "v2" or changing the service name.

      There is a balance to be struck between having numerous versions and too little versioning. I disagree that "every change, even small ones, can and will royally screw your users". Adding a parameter to a service and nothing else changes does not, in my opinion, warrant a completely new URL. Likewise, adding a completely new element to XML results without changing the structure of the existing elements does also not warrant a 99% parallel service, wouldn't be very eXtensible if adding details broke a parser.

      For the recent update here was the logic. The change was relatively disruptive, we added a whole new <Network> level to the StationXML. StationXML is not quite settled but is getting very close, so we don't expect this to happen much more often if at all. The ws-station service is relatively new, most folks are using it with the Fetch* scripts we provide which were not effected because of the stream processing of XML (i.e. you are one of the few that are parsing it directly) so that limits the disruption. In short, the impact of changing the results versus changing the URL in this case was not worth having a new service. If, say, a year from now we need to do that big of a change and the adoption of ws-station is high we would probably make a new version.

      StationXML will continue to evolve to match the needs of SCEDC, NCEDC, IRIS and USGS (and maybe the FDSN in the future). At this point we do not expect large structural changes, but we do expect additions which should not break any real XML parser.

      Chad

      On Apr 22, 2011, at 6:51 AM, Philip Crotwell wrote:

      Hi Chad

      I guess it is only natural that as these services are new, there will
      be some changes. It looks like this one is backwards compatible, but
      the previous one to the station web service wasn't and would cause
      problems for any other clients out there.

      Have you all given any thought to a longer term versioning for these
      systems? Here is the problem I worry about. I put code into a software
      package to access these web services. My code is obviously tuned to
      the way the web service works at the time I write and test it. Later,
      after I have made a release, IRIS makes a non-backwards compatible
      change. Then, my client breaks and it looks to the rest of the world
      like I have written buggy code and it can't be fixed without a new
      release of my client. I also have little to no forewarning that the
      change is coming, and therefore have little ability to be proactive in
      what I do.

      If you change the IRIS web page, people just notice the change and
      figure things out. They may like or dislike the change, but they can
      adapt. Web services are fundamentally different than web pages. A
      small change that would cause no problem for a person can easily cause
      a web service client to completely fail.

      I guess it feels to me like once you put out a web service that will
      be called by third party clients, you have some obligation to either
      make only backward compatible changes, or to run the old version and
      the new version in parallel for some time to allow an orderly
      transition. Given the development cycle for clients, my guess is this
      might be closer to years than weeks. Although the IDL revision for
      fissures/dhi was never implemented, we addressed this issue in DHI by
      encoding the IDL version number into the name service lookup system,
      so an IDL 1.0 client would be able to find IDL1.0 servers and would
      not accidentally try to connect to IDL 2.0 servers. Obviously web
      services will use different mechanisms, but if they are both going to
      be used and to evolve at the same time, then you have to provide some
      means for the communications protocol contract to remain stable.
      Otherwise every change, even small ones, can and will royally screw
      your users.

      This versioning could easily be accomplished in several ways, either
      putting a version number in the url, say
      http://www.iris.edu/ws/1.0/bulkdataselect/
      or as a parameter
      http://www.iris.edu/ws/bulkdataselect/?version=1.0
      The former has they advantage of you leaving the old server up and
      just adding new ones, while the second allows you to be fancy in your
      implementation to tailor the output to the client. I guess I prefer
      the stability of the first, but as a client writer I can deal with
      either.

      thanks,
      Philip

      On Wed, Apr 20, 2011 at 7:30 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:

      Hello webservice users,

      The DMC has updated it's ws-bulkdataselect service:
      http://www.iris.edu/ws/bulkdataselect/

      The new version adds two optional query parameters:

      minimumlength
      Enforce minimum segment length (seconds).
      Only time-series segments of this length or longer will be returned.

      longestonly
      If specified, only the longest continuous segment for each channel is
      returned.

      These new parameters can be used to reduce or avoid receiving data
      containing gaps.
      The command line FetchBulkData client script that uses this service to
      request data has been updated with options to specify these new parameters,
      the latest client scripts are available here:
      http://www.iris.edu/ws/wsclients/
      Usage details: like the quality parameter, the two new parameters may be
      specified in the main selection list or separately as post parameters.
      For example, this selection will return only the longest data, that is at
      least one day (86400 seconds) long:

      minimumlength 86400.0
      longestonly
      TA A25A -- BHZ 2010-084T00:00:00 2010-091T00:00:00
      TA A25A -- BHE 2010-084T00:00:00 2010-091T00:00:00
      regards
      Web services team.


      _______________________________________________
      webservices mailing list
      webservices<at>iris.washington.edu
      http://www.iris.washington.edu/mailman/listinfo/webservices



      _______________________________________________
      webservices mailing list
      webservices<at>iris.washington.edu
      http://www.iris.washington.edu/mailman/listinfo/webservices



      • Philip Crotwell
        2011-04-22 21:43:23
        OK, sound like you have a plan for this. My concern was that the
        station change was done with little advance warning and no grace
        period. So, it broke my code. Luckily all the places it was used were
        local, so I was able to quickly update them. As you say, this is all
        new, so some instability is to be expected. And I am fine with
        backward compatible changes (adding optional url args or extra xml
        elements that can be ignored), those certainly don't need version
        changes. But a change, like the station one, that was small in some
        sense, yet it really altered the xml output in a pretty fundamental
        way. I like the new stationxml better, so I am not complaining about
        that. It is just that as a client writer, I am highly dependent on
        those servers and at the mercy of the decisions the dmc makes while at
        the same time having very little power or influence on how those
        choices are made. Had I made a software release the day before you
        pushed out the station change, I would have been SOL. And so I would
        have expected that type of change to happen in beta, but for there to
        be a grace period and some forewarning at least to the web services
        email list after you declared these services ready for prime time.

        At any rate, the new station ws is quite nice, so on the whole I am very happy.

        Philip


        On Fri, Apr 22, 2011 at 1:45 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:

        Hi Philip,

        We have discussed how to deal with versioned services/results including both of the suggestions you make below.  We landed at using different base URLs for major changes in API or content, either adding "v2" or changing the service name.

        There is a balance to be struck between having numerous versions and too little versioning.  I disagree that "every change, even small ones, can and will royally screw your users".  Adding a parameter to a service and nothing else changes does not, in my opinion, warrant a completely new URL.  Likewise, adding a completely new element to XML results without changing the structure of the existing elements does also not warrant a 99% parallel service, wouldn't be very eXtensible if adding details broke a parser.

        For the recent update here was the logic.  The change was relatively disruptive, we added a whole new <Network> level to the StationXML.  StationXML is not quite settled but is getting very close, so we don't expect this to happen much more often if at all.  The ws-station service is relatively new, most folks are using it with the Fetch* scripts we provide which were not effected because of the stream processing of XML (i.e. you are one of the few that are parsing it directly) so that limits the disruption.  In short, the impact of changing the results versus changing the URL in this case was not worth having a new service.  If, say, a year from now we need to do that big of a change and the adoption of ws-station is high we would probably make a new version.

        StationXML will continue to evolve to match the needs of SCEDC, NCEDC, IRIS and USGS (and maybe the FDSN in the future).  At this point we do not expect large structural changes, but we do expect additions which should not break any real XML parser.

        Chad

        On Apr 22, 2011, at 6:51 AM, Philip Crotwell wrote:

        Hi Chad

        I guess it is only natural that as these services are new, there will
        be some changes. It looks like this one is backwards compatible, but
        the previous one to the station web service wasn't and would cause
        problems for any other clients out there.

        Have you all given any thought to a longer term versioning for these
        systems? Here is the problem I worry about. I put code into a software
        package to access these web services. My code is obviously tuned to
        the way the web service works at the time I write and test it. Later,
        after I have made a release, IRIS makes a non-backwards compatible
        change. Then, my client breaks and it looks to the rest of the world
        like I have written buggy code and it can't be fixed without a new
        release of my client. I also have little to no forewarning that the
        change is coming, and therefore have little ability to be proactive in
        what I do.

        If you change the IRIS web page, people just notice the change and
        figure things out. They may like or dislike the change, but they can
        adapt. Web services are fundamentally different than web pages. A
        small change that would cause no problem for a person can easily cause
        a web service client to completely fail.

        I guess it feels to me like once you put out a web service that will
        be called by third party clients, you have some obligation to either
        make only backward compatible changes, or to run the old version and
        the new version in parallel for some time to allow an orderly
        transition. Given the development cycle for clients, my guess is this
        might be closer to years than weeks. Although the IDL revision for
        fissures/dhi was never implemented, we addressed this issue in DHI by
        encoding the IDL version number into the name service lookup system,
        so an IDL 1.0 client would be able to find IDL1.0 servers and would
        not accidentally try to connect to IDL 2.0 servers. Obviously web
        services will use different mechanisms, but if they are both going to
        be used and to evolve at the same time, then you have to provide some
        means for the communications protocol contract to remain stable.
        Otherwise every change, even small ones, can and will royally screw
        your users.

        This versioning could easily be accomplished in several ways, either
        putting a version number in the url, say
        http://www.iris.edu/ws/1.0/bulkdataselect/
        or as a parameter
        http://www.iris.edu/ws/bulkdataselect/?version=1.0
        The former has they advantage of you leaving the old server up and
        just adding new ones, while the second allows you to be fancy in your
        implementation to tailor the output to the client. I guess I prefer
        the stability of the first, but as a client writer I can deal with
        either.

        thanks,
        Philip

        On Wed, Apr 20, 2011 at 7:30 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:

        Hello webservice users,

        The DMC has updated it's ws-bulkdataselect service:
        http://www.iris.edu/ws/bulkdataselect/

        The new version adds two optional query parameters:

        minimumlength
        Enforce minimum segment length (seconds).
        Only time-series segments of this length or longer will be returned.

        longestonly
        If specified, only the longest continuous segment for each channel is
        returned.

        These new parameters can be used to reduce or avoid receiving data
        containing gaps.
        The command line FetchBulkData client script that uses this service to
        request data has been updated with options to specify these new parameters,
        the latest client scripts are available here:
        http://www.iris.edu/ws/wsclients/
        Usage details: like the quality parameter, the two new parameters may be
        specified in the main selection list or separately as post parameters.
        For example, this selection will return only the longest data, that is at
        least one day (86400 seconds) long:

        minimumlength 86400.0
        longestonly
        TA A25A -- BHZ 2010-084T00:00:00 2010-091T00:00:00
        TA A25A -- BHE 2010-084T00:00:00 2010-091T00:00:00
        regards
        Web services team.


        _______________________________________________
        webservices mailing list
        webservices<at>iris.washington.edu
        http://www.iris.washington.edu/mailman/listinfo/webservices



        _______________________________________________
        webservices mailing list
        webservices<at>iris.washington.edu
        http://www.iris.washington.edu/mailman/listinfo/webservices


        _______________________________________________
        webservices mailing list
        webservices<at>iris.washington.edu
        http://www.iris.washington.edu/mailman/listinfo/webservices



  • Chad Trabant
    2011-07-20 13:02:53

    Hello web service users,

    New versions of the DMC's web service Fetch scripts have been released and are available here:

    http://www.iris.edu/ws/wsclients/

    The FetchBulkData script has been made more robust by breaking large requests into groups and restarting the download of a group when the connection has been interrupted. These features combined allow FetchBulkData to better handle very large data requests and to recover automatically from network connectivity problems or service interruptions.

    The web page now also includes client usage documentation with examples of usage and description of key features.

    Each script has improved documentation, minor fixes and improved reporting that will help the DMC and users of these scripts diagnose problems.

    Users of these scripts are encouraged to keep their copy up to date.

    regards,
    web services team


    On Apr 20, 2011, at 4:30 PM, Chad Trabant wrote:


    Hello webservice users,

    The DMC has updated it's ws-bulkdataselect service: http://www.iris.edu/ws/bulkdataselect/

    The new version adds two optional query parameters:

    minimumlength
    Enforce minimum segment length (seconds).
    Only time-series segments of this length or longer will be returned.

    longestonly
    If specified, only the longest continuous segment for each channel is
    returned.

    These new parameters can be used to reduce or avoid receiving data containing gaps.

    The command line FetchBulkData client script that uses this service to request data has been updated with options to specify these new parameters, the latest client scripts are available here:
    http://www.iris.edu/ws/wsclients/

    Usage details: like the quality parameter, the two new parameters may be specified in the main selection list or separately as post parameters.

    For example, this selection will return only the longest data, that is at least one day (86400 seconds) long:

    minimumlength 86400.0
    longestonly
    TA A25A -- BHZ 2010-084T00:00:00 2010-091T00:00:00
    TA A25A -- BHE 2010-084T00:00:00 2010-091T00:00:00

    regards
    Web services team.



    • Chad Trabant
      2011-08-10 01:35:05

      Hello web service users,

      A new revision of the DMC's FetchBulkData script has been released and is available here:

      http://www.iris.edu/ws/wsclients/

      The FetchBulkData script has been updated to include options for selecting data by geographic region in to two ways:

      a) Users may specify a rectangular region using minimum and maximum latitude and/or longitude options
      b) Users may specify a circular region by defining a point and a maximum radius, optionally a minimum radius may be specified to selected a limited distance range.

      The documentation on the page above now includes an example of selecting data by region.

      This version also fixes a bug for selecting multiple channels of data with different time windows, like is possible with a BREQ_FAST or selection file.

      Users of this script are encouraged to keep their copy up to date.

      regards,
      web services team




07:49:19 v.01697673