Thread: StationEpoch duplicates

Started: 2011-11-14 18:05:12
Last activity: 2011-11-14 22:20:54
Topics: Web Services
John West
2011-11-14 18:05:12
Hi.

The query
http://www.iris.edu/ws/station/query?updatedafter=2011-06-06&level=sta&net=ND&timewindow=2010-01-01,2500-12-12
returns multiple duplicates of the <StationEpoch> sections for each
station.

Makes my sync to a local copy procedure very slow...

Thanks!

-- John

  • Alex Hutko
    2011-11-14 03:16:48
    Hi John,

    What happened is that you selected "station level" information using the -sta flag. If you didn't use that, it would default to channel level and you would see that WS is returning one line for each channel and omitting the channel name, so it looks like lots of duplicates. If you want only the earliest and latest station epochs, you will have to read them all to figure this out.

    Also just as an FYI since you are interested in the epochs, sometimes a very tiny gap in station epochs can exist (like one sample point e.g. 0.05sec) or metadata gets changed and what happens is that you will see 2 epochs where the end of the first epoch will equal the start of the second.

    epoch 1: 2002-03-17T00:00:00,2005-04-05T06:07:08
    epoch 2: 2005-04-05T06:07:08,2599-12-31T23:59:59

    Alex


    channel level:

    scoop synthetics/METADATA> FetchMetadata -ua 2011-06-06 -N ND -vvv -s 2010-01-01 -e 2500-12-12
    FetchMetadata (2011.172) at 2011-11-13 18:37:39
    == Data selections ==
    ND||||2010-01-01T00:00:00|2500-12-12T00:00:00
    Metadata URI: 'http://www.iris.edu/ws/station/query?level=chan&network=ND&timewindow=2010-01-01,2500-12-12&updatedafter=2011-06-06'
    Fetching metadata :: Received 108.1 KB
    Received 108.1 KB of metadata in 0.1 seconds (1.2 MB/s)
    Parsing XML metadata... Done (0.1 seconds)
    Processed metadata for 84 channel epochs in 0.2 seconds (488.3 KB/s)
    Writing metadata (84 channel epochs) file
    #net,sta,loc,chan,lat,lon,elev,depth,azimuth,SACdip,instrument,scale,scalefreq,scaleunits,samplerate,start,end
    ND,KOUNC,--,BHE,-20.5507,164.2861,88.0,0.0,90.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,BHN,-20.5507,164.2861,88.0,0.0,0.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,BHZ,-20.5507,164.2861,88.0,0.0,0.0,0,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,BNE,-20.5507,164.2861,88.0,0.0,90.0,90,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,BNN,-20.5507,164.2861,88.0,0.0,0.0,90,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,BNZ,-20.5507,164.2861,88.0,0.0,0.0,0,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,HHE,-20.5507,164.2861,88.0,0.0,90.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,HHN,-20.5507,164.2861,88.0,0.0,0.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,HHZ,-20.5507,164.2861,88.0,0.0,0.0,0,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,HNE,-20.5507,164.2861,88.0,0.0,90.0,90,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,HNN,-20.5507,164.2861,88.0,0.0,0.0,90,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,--,HNZ,-20.5507,164.2861,88.0,0.0,0.0,0,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,LIFNC,--,BHE,-20.7771,167.2402,33.0,0.0,90.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
    ND,LIFNC,--,BHN,-20.7771,167.2402,33.0,0.0,0.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
    ND,LIFNC,--,BHZ,-20.7771,167.2402,33.0,0.0,0.0,0,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
    ND,LIFNC,--,BNE,-20.7771,167.2402,33.0,0.0,90.0,90,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,20,2010-08-28T00:00:00,2599-12-31T23:59:59

    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    station level (-sta flag):

    scoop synthetics/METADATA> FetchMetadata -ua 2011-06-06 -N ND -vvv -s 2010-01-01 -e 2500-12-12 -sta
    FetchMetadata (2011.172) at 2011-11-13 18:36:27
    == Data selections ==
    ND||||2010-01-01T00:00:00|2500-12-12T00:00:00
    Metadata URI: 'http://www.iris.edu/ws/station/query?level=sta&network=ND&timewindow=2010-01-01,2500-12-12&updatedafter=2011-06-06'
    Fetching metadata :: Received 38.3 KB
    Received 38.3 KB of metadata in 0.1 seconds (492.5 KB/s)
    Parsing XML metadata... Done (0.0 seconds)
    Processed metadata for 84 station epochs in 0.1 seconds (323.3 KB/s)
    Writing metadata (84 station epochs) file
    #net,sta,lat,lon,elev,site,start,end
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
    ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
    ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
    ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
    ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59



    On Nov 13, 2011, at 4:05 PM, John D. West wrote:

    Hi.

    The query http://www.iris.edu/ws/station/query?updatedafter=2011-06-06&level=sta&net=ND&timewindow=2010-01-01,2500-12-12
    returns multiple duplicates of the <StationEpoch> sections for each station.

    Makes my sync to a local copy procedure very slow...

    Thanks!

    -- John
    _______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices


    • John West
      2011-11-14 22:20:54
      Hi, Alex.

      OK, I understand why I'm getting duplicates. I do think, however, that the
      service should only return distinct StationEpochs. If the service returns
      this level of duplicated information, it makes me wonder whether the
      level=sta format has any utility. Why even provide it?

      Thanks!

      -- John


      On Mon, Nov 14, 2011 at 1:16 PM, Alex <alex<at>iris.washington.edu> wrote:

      Hi John,

      What happened is that you selected "station level" information using the
      -sta flag. If you didn't use that, it would default to channel level and
      you would see that WS is returning one line for each channel and omitting
      the channel name, so it looks like lots of duplicates. If you want only
      the earliest and latest station epochs, you will have to read them all to
      figure this out.

      Also just as an FYI since you are interested in the epochs, sometimes a
      very tiny gap in station epochs can exist (like one sample point e.g.
      0.05sec) or metadata gets changed and what happens is that you will see 2
      epochs where the end of the first epoch will equal the start of the second.

      epoch 1: 2002-03-17T00:00:00,2005-04-05T06:07:08
      epoch 2: 2005-04-05T06:07:08,2599-12-31T23:59:59

      Alex


      channel level:

      scoop synthetics/METADATA> FetchMetadata -ua 2011-06-06 -N ND -vvv -s
      2010-01-01 -e 2500-12-12
      FetchMetadata (2011.172) at 2011-11-13 18:37:39
      == Data selections ==
      ND||||2010-01-01T00:00:00|2500-12-12T00:00:00
      Metadata URI: '
      http://www.iris.edu/ws/station/query?level=chan&network=ND&timewindow=2010-01-01,2500-12-12&updatedafter=2011-06-06
      '
      Fetching metadata :: Received 108.1 KB
      Received 108.1 KB of metadata in 0.1 seconds (1.2 MB/s)
      Parsing XML metadata... Done (0.1 seconds)
      Processed metadata for 84 channel epochs in 0.2 seconds (488.3 KB/s)
      Writing metadata (84 channel epochs) file

      #net,sta,loc,chan,lat,lon,elev,depth,azimuth,SACdip,instrument,scale,scalefreq,scaleunits,samplerate,start,end
      ND,KOUNC,--,BHE,-20.5507,164.2861,88.0,0.0,90.0,90,Trillium 120P; 120 s;
      1201 V/m/s-Kephren; gain
      1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,BHN,-20.5507,164.2861,88.0,0.0,0.0,90,Trillium 120P; 120 s;
      1201 V/m/s-Kephren; gain
      1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,BHZ,-20.5507,164.2861,88.0,0.0,0.0,0,Trillium 120P; 120 s;
      1201 V/m/s-Kephren; gain
      1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,BNE,-20.5507,164.2861,88.0,0.0,90.0,90,Episensor; DC to >200
      Hz; 20V differential full
      s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,BNN,-20.5507,164.2861,88.0,0.0,0.0,90,Episensor; DC to >200
      Hz; 20V differential full
      s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,BNZ,-20.5507,164.2861,88.0,0.0,0.0,0,Episensor; DC to >200 Hz;
      20V differential full
      s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,HHE,-20.5507,164.2861,88.0,0.0,90.0,90,Trillium 120P; 120 s;
      1201 V/m/s-Kephren; gain
      1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,HHN,-20.5507,164.2861,88.0,0.0,0.0,90,Trillium 120P; 120 s;
      1201 V/m/s-Kephren; gain
      1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,HHZ,-20.5507,164.2861,88.0,0.0,0.0,0,Trillium 120P; 120 s;
      1201 V/m/s-Kephren; gain
      1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,HNE,-20.5507,164.2861,88.0,0.0,90.0,90,Episensor; DC to >200
      Hz; 20V differential full
      s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,HNN,-20.5507,164.2861,88.0,0.0,0.0,90,Episensor; DC to >200
      Hz; 20V differential full
      s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,--,HNZ,-20.5507,164.2861,88.0,0.0,0.0,0,Episensor; DC to >200 Hz;
      20V differential full
      s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,LIFNC,--,BHE,-20.7771,167.2402,33.0,0.0,90.0,90,Trillium 120P; 120 s;
      1201 V/m/s-Kephren; gain
      1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
      ND,LIFNC,--,BHN,-20.7771,167.2402,33.0,0.0,0.0,90,Trillium 120P; 120 s;
      1201 V/m/s-Kephren; gain
      1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
      ND,LIFNC,--,BHZ,-20.7771,167.2402,33.0,0.0,0.0,0,Trillium 120P; 120 s;
      1201 V/m/s-Kephren; gain
      1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
      ND,LIFNC,--,BNE,-20.7771,167.2402,33.0,0.0,90.0,90,Episensor; DC to >200
      Hz; 20V differential full
      s,2.18979E8,0.05,M/S**2,20,2010-08-28T00:00:00,2599-12-31T23:59:59


      ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      station level (-sta flag):

      scoop synthetics/METADATA> FetchMetadata -ua 2011-06-06 -N ND -vvv -s
      2010-01-01 -e 2500-12-12 -sta
      FetchMetadata (2011.172) at 2011-11-13 18:36:27
      == Data selections ==
      ND||||2010-01-01T00:00:00|2500-12-12T00:00:00
      Metadata URI: '
      http://www.iris.edu/ws/station/query?level=sta&network=ND&timewindow=2010-01-01,2500-12-12&updatedafter=2011-06-06
      '
      Fetching metadata :: Received 38.3 KB
      Received 38.3 KB of metadata in 0.1 seconds (492.5 KB/s)
      Parsing XML metadata... Done (0.0 seconds)
      Processed metadata for 84 station epochs in 0.1 seconds (323.3 KB/s)
      Writing metadata (84 station epochs) file
      #net,sta,lat,lon,elev,site,start,end
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
      Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
      ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New
      Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
      ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New
      Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
      ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New
      Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
      ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New
      Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59



      On Nov 13, 2011, at 4:05 PM, John D. West wrote:

      Hi.

      The query
      http://www.iris.edu/ws/station/query?updatedafter=2011-06-06&level=sta&net=ND&timewindow=2010-01-01,2500-12-12
      returns multiple duplicates of the <StationEpoch> sections for each
      station.

      Makes my sync to a local copy procedure very slow...

      Thanks!

      -- John
      _______________________________________________
      webservices mailing list
      webservices<at>iris.washington.edu
      http://www.iris.washington.edu/mailman/listinfo/webservices




      • Philip Crotwell
        2011-11-14 17:23:15
        Hi

        This might be related to the duplicate <stationEpoch> issue I found on
        Oct 19. Yazan said that this was a bug and would be fixed in the next
        version.

        Philip

        ------------

        Thanks Philip

        This is a bug and will be addressed in the new release (1.2.6). The
        new release is scheduled for release very soon. Yazan

        On Oct 19, 2011, at 7:08 AM, Philip Crotwell wrote:

        Hi

        This URL generates a large number of duplicate <stationEpoch> elements
        for each station. Oddly, if anything, including a *, is put in the
        "sta" query item, then the duplicate stationEpoch elements are not
        there. Deleting the timewindow also removes the duplicate entries. Not
        sure, but wonder if it is picking up a stationEpoch for each channel?

        http://www.iris.edu/ws/station/query?net=II&timewindow=1997-06-07%2C2011-06-07&level=sta

        thanks,
        Philip

  • Chad Trabant
    2011-11-14 15:24:37

    Hi John,

    We have beta release ready to go out that fixes this bug and others. Unless remaining issues are found the beta will be released for external testing today.

    Chad

    On Nov 13, 2011, at 4:05 PM, John D. West wrote:

    Hi.

    The query http://www.iris.edu/ws/station/query?updatedafter=2011-06-06&level=sta&net=ND&timewindow=2010-01-01,2500-12-12
    returns multiple duplicates of the <StationEpoch> sections for each station.

    Makes my sync to a local copy procedure very slow...

    Thanks!

    -- John
    _______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices


Page built 22:33:02 | v.090361a2