NSF SAGE: Thread: StationEpoch duplicates

Started: 2011-11-14 18:05:12

Last activity: 2011-11-14 22:20:54

Topics: Web Services

John West

2011-11-14 18:05:12

Hi.

The query
http://www.iris.edu/ws/station/query?updatedafter=2011-06-06&level=sta&net=ND&timewindow=2010-01-01,2500-12-12
returns multiple duplicates of the <StationEpoch> sections for each
station.

Makes my sync to a local copy procedure very slow...

Thanks!

-- John

Alex Hutko

Re: StationEpoch duplicates

2011-11-14 03:16:48

Hi John,

What happened is that you selected "station level" information using the -sta flag. If you didn't use that, it would default to channel level and you would see that WS is returning one line for each channel and omitting the channel name, so it looks like lots of duplicates. If you want only the earliest and latest station epochs, you will have to read them all to figure this out.

Also just as an FYI since you are interested in the epochs, sometimes a very tiny gap in station epochs can exist (like one sample point e.g. 0.05sec) or metadata gets changed and what happens is that you will see 2 epochs where the end of the first epoch will equal the start of the second.

epoch 1: 2002-03-17T00:00:00,2005-04-05T06:07:08
epoch 2: 2005-04-05T06:07:08,2599-12-31T23:59:59

Alex

channel level:

scoop synthetics/METADATA> FetchMetadata -ua 2011-06-06 -N ND -vvv -s 2010-01-01 -e 2500-12-12
FetchMetadata (2011.172) at 2011-11-13 18:37:39
== Data selections ==
ND||||2010-01-01T00:00:00|2500-12-12T00:00:00
Metadata URI: 'http://www.iris.edu/ws/station/query?level=chan&network=ND&timewindow=2010-01-01,2500-12-12&updatedafter=2011-06-06'
Fetching metadata :: Received 108.1 KB
Received 108.1 KB of metadata in 0.1 seconds (1.2 MB/s)
Parsing XML metadata... Done (0.1 seconds)
Processed metadata for 84 channel epochs in 0.2 seconds (488.3 KB/s)
Writing metadata (84 channel epochs) file
#net,sta,loc,chan,lat,lon,elev,depth,azimuth,SACdip,instrument,scale,scalefreq,scaleunits,samplerate,start,end
ND,KOUNC,--,BHE,-20.5507,164.2861,88.0,0.0,90.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,BHN,-20.5507,164.2861,88.0,0.0,0.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,BHZ,-20.5507,164.2861,88.0,0.0,0.0,0,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,BNE,-20.5507,164.2861,88.0,0.0,90.0,90,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,BNN,-20.5507,164.2861,88.0,0.0,0.0,90,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,BNZ,-20.5507,164.2861,88.0,0.0,0.0,0,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,HHE,-20.5507,164.2861,88.0,0.0,90.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,HHN,-20.5507,164.2861,88.0,0.0,0.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,HHZ,-20.5507,164.2861,88.0,0.0,0.0,0,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,HNE,-20.5507,164.2861,88.0,0.0,90.0,90,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,HNN,-20.5507,164.2861,88.0,0.0,0.0,90,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,--,HNZ,-20.5507,164.2861,88.0,0.0,0.0,0,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,LIFNC,--,BHE,-20.7771,167.2402,33.0,0.0,90.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
ND,LIFNC,--,BHN,-20.7771,167.2402,33.0,0.0,0.0,90,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
ND,LIFNC,--,BHZ,-20.7771,167.2402,33.0,0.0,0.0,0,Trillium 120P; 120 s; 1201 V/m/s-Kephren; gain 1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
ND,LIFNC,--,BNE,-20.7771,167.2402,33.0,0.0,90.0,90,Episensor; DC to >200 Hz; 20V differential full s,2.18979E8,0.05,M/S**2,20,2010-08-28T00:00:00,2599-12-31T23:59:59

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
station level (-sta flag):

scoop synthetics/METADATA> FetchMetadata -ua 2011-06-06 -N ND -vvv -s 2010-01-01 -e 2500-12-12 -sta
FetchMetadata (2011.172) at 2011-11-13 18:36:27
== Data selections ==
ND||||2010-01-01T00:00:00|2500-12-12T00:00:00
Metadata URI: 'http://www.iris.edu/ws/station/query?level=sta&network=ND&timewindow=2010-01-01,2500-12-12&updatedafter=2011-06-06'
Fetching metadata :: Received 38.3 KB
Received 38.3 KB of metadata in 0.1 seconds (492.5 KB/s)
Parsing XML metadata... Done (0.0 seconds)
Processed metadata for 84 station epochs in 0.1 seconds (323.3 KB/s)
Writing metadata (84 station epochs) file
#net,sta,lat,lon,elev,site,start,end
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59

On Nov 13, 2011, at 4:05 PM, John D. West wrote:

Hi.

The query http://www.iris.edu/ws/station/query?updatedafter=2011-06-06&level=sta&net=ND&timewindow=2010-01-01,2500-12-12
returns multiple duplicates of the <StationEpoch> sections for each station.

Makes my sync to a local copy procedure very slow...

Thanks!

-- John
_______________________________________________
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices
- John West
  
  Re: StationEpoch duplicates
  
  2011-11-14 22:20:54
  
  Hi, Alex.
  
  OK, I understand why I'm getting duplicates. I do think, however, that the
  service should only return distinct StationEpochs. If the service returns
  this level of duplicated information, it makes me wonder whether the
  level=sta format has any utility. Why even provide it?
  
  Thanks!
  
  -- John
  
  On Mon, Nov 14, 2011 at 1:16 PM, Alex <alex<at>iris.washington.edu> wrote:
  
  Hi John,
  
  What happened is that you selected "station level" information using the
  -sta flag. If you didn't use that, it would default to channel level and
  you would see that WS is returning one line for each channel and omitting
  the channel name, so it looks like lots of duplicates. If you want only
  the earliest and latest station epochs, you will have to read them all to
  figure this out.
  
  Also just as an FYI since you are interested in the epochs, sometimes a
  very tiny gap in station epochs can exist (like one sample point e.g.
  0.05sec) or metadata gets changed and what happens is that you will see 2
  epochs where the end of the first epoch will equal the start of the second.
  
  epoch 1: 2002-03-17T00:00:00,2005-04-05T06:07:08
  epoch 2: 2005-04-05T06:07:08,2599-12-31T23:59:59
  
  Alex
  
  channel level:
  
  scoop synthetics/METADATA> FetchMetadata -ua 2011-06-06 -N ND -vvv -s
  2010-01-01 -e 2500-12-12
  FetchMetadata (2011.172) at 2011-11-13 18:37:39
  == Data selections ==
  ND||||2010-01-01T00:00:00|2500-12-12T00:00:00
  Metadata URI: '
  http://www.iris.edu/ws/station/query?level=chan&network=ND&timewindow=2010-01-01,2500-12-12&updatedafter=2011-06-06
  '
  Fetching metadata :: Received 108.1 KB
  Received 108.1 KB of metadata in 0.1 seconds (1.2 MB/s)
  Parsing XML metadata... Done (0.1 seconds)
  Processed metadata for 84 channel epochs in 0.2 seconds (488.3 KB/s)
  Writing metadata (84 channel epochs) file
  
  #net,sta,loc,chan,lat,lon,elev,depth,azimuth,SACdip,instrument,scale,scalefreq,scaleunits,samplerate,start,end
  ND,KOUNC,--,BHE,-20.5507,164.2861,88.0,0.0,90.0,90,Trillium 120P; 120 s;
  1201 V/m/s-Kephren; gain
  1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,BHN,-20.5507,164.2861,88.0,0.0,0.0,90,Trillium 120P; 120 s;
  1201 V/m/s-Kephren; gain
  1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,BHZ,-20.5507,164.2861,88.0,0.0,0.0,0,Trillium 120P; 120 s;
  1201 V/m/s-Kephren; gain
  1;,1.28257E11,0.05,M/S,20,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,BNE,-20.5507,164.2861,88.0,0.0,90.0,90,Episensor; DC to >200
  Hz; 20V differential full
  s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,BNN,-20.5507,164.2861,88.0,0.0,0.0,90,Episensor; DC to >200
  Hz; 20V differential full
  s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,BNZ,-20.5507,164.2861,88.0,0.0,0.0,0,Episensor; DC to >200 Hz;
  20V differential full
  s,2.18979E8,0.05,M/S**2,20,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,HHE,-20.5507,164.2861,88.0,0.0,90.0,90,Trillium 120P; 120 s;
  1201 V/m/s-Kephren; gain
  1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,HHN,-20.5507,164.2861,88.0,0.0,0.0,90,Trillium 120P; 120 s;
  1201 V/m/s-Kephren; gain
  1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,HHZ,-20.5507,164.2861,88.0,0.0,0.0,0,Trillium 120P; 120 s;
  1201 V/m/s-Kephren; gain
  1;,1.28257E11,0.05,M/S,100,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,HNE,-20.5507,164.2861,88.0,0.0,90.0,90,Episensor; DC to >200
  Hz; 20V differential full
  s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,HNN,-20.5507,164.2861,88.0,0.0,0.0,90,Episensor; DC to >200
  Hz; 20V differential full
  s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,--,HNZ,-20.5507,164.2861,88.0,0.0,0.0,0,Episensor; DC to >200 Hz;
  20V differential full
  s,2.18979E8,0.05,M/S**2,100,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,LIFNC,--,BHE,-20.7771,167.2402,33.0,0.0,90.0,90,Trillium 120P; 120 s;
  1201 V/m/s-Kephren; gain
  1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
  ND,LIFNC,--,BHN,-20.7771,167.2402,33.0,0.0,0.0,90,Trillium 120P; 120 s;
  1201 V/m/s-Kephren; gain
  1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
  ND,LIFNC,--,BHZ,-20.7771,167.2402,33.0,0.0,0.0,0,Trillium 120P; 120 s;
  1201 V/m/s-Kephren; gain
  1;,1.28257E11,0.05,M/S,20,2010-08-28T00:00:00,2599-12-31T23:59:59
  ND,LIFNC,--,BNE,-20.7771,167.2402,33.0,0.0,90.0,90,Episensor; DC to >200
  Hz; 20V differential full
  s,2.18979E8,0.05,M/S**2,20,2010-08-28T00:00:00,2599-12-31T23:59:59
  
  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  station level (-sta flag):
  
  scoop synthetics/METADATA> FetchMetadata -ua 2011-06-06 -N ND -vvv -s
  2010-01-01 -e 2500-12-12 -sta
  FetchMetadata (2011.172) at 2011-11-13 18:36:27
  == Data selections ==
  ND||||2010-01-01T00:00:00|2500-12-12T00:00:00
  Metadata URI: '
  http://www.iris.edu/ws/station/query?level=sta&network=ND&timewindow=2010-01-01,2500-12-12&updatedafter=2011-06-06
  '
  Fetching metadata :: Received 38.3 KB
  Received 38.3 KB of metadata in 0.1 seconds (492.5 KB/s)
  Parsing XML metadata... Done (0.0 seconds)
  Processed metadata for 84 station epochs in 0.1 seconds (323.3 KB/s)
  Writing metadata (84 station epochs) file
  #net,sta,lat,lon,elev,site,start,end
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,KOUNC,-20.5507,164.2861,88.0,Koumac; New
  Caledonia,2011-03-17T00:00:00,2599-12-31T23:59:59
  ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New
  Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
  ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New
  Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
  ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New
  Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
  ND,LIFNC,-20.7771,167.2402,33.0,Lifou; Loyalty Islands; New
  Caledonia,2010-08-28T00:00:00,2599-12-31T23:59:59
  
  On Nov 13, 2011, at 4:05 PM, John D. West wrote:
  
  Hi.
  
  The query
  http://www.iris.edu/ws/station/query?updatedafter=2011-06-06&level=sta&net=ND&timewindow=2010-01-01,2500-12-12
  returns multiple duplicates of the <StationEpoch> sections for each
  station.
  
  Makes my sync to a local copy procedure very slow...
  
  Thanks!
  
  -- John
  _______________________________________________
  webservices mailing list
  webservices<at>iris.washington.edu
  http://www.iris.washington.edu/mailman/listinfo/webservices
  - Philip Crotwell
    
    Re: StationEpoch duplicates
    
    2011-11-14 17:23:15
    
    Hi
    
    This might be related to the duplicate <stationEpoch> issue I found on
    Oct 19. Yazan said that this was a bug and would be fixed in the next
    version.
    
    Philip
    
    ------------
    
    Thanks Philip
    
    This is a bug and will be addressed in the new release (1.2.6). The
    new release is scheduled for release very soon. Yazan
    
    On Oct 19, 2011, at 7:08 AM, Philip Crotwell wrote:
    
    Hi
    
    This URL generates a large number of duplicate <stationEpoch> elements
    for each station. Oddly, if anything, including a *, is put in the
    "sta" query item, then the duplicate stationEpoch elements are not
    there. Deleting the timewindow also removes the duplicate entries. Not
    sure, but wonder if it is picking up a stationEpoch for each channel?
    
    http://www.iris.edu/ws/station/query?net=II&timewindow=1997-06-07%2C2011-06-07&level=sta
    
    thanks,
    Philip
Chad Trabant

Re: StationEpoch duplicates

2011-11-14 15:24:37

Hi John,

We have beta release ready to go out that fixes this bug and others. Unless remaining issues are found the beta will be released for external testing today.

Chad

On Nov 13, 2011, at 4:05 PM, John D. West wrote:

Hi.

The query http://www.iris.edu/ws/station/query?updatedafter=2011-06-06&level=sta&net=ND&timewindow=2010-01-01,2500-12-12
returns multiple duplicates of the <StationEpoch> sections for each station.

Makes my sync to a local copy procedure very slow...

Thanks!

-- John
_______________________________________________
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices

NSF SAGE ingests, curates, and distributes geoscience data

DATA AT NSF SAGE

DATA INGESTION

DATA ANALYTICS

DATA ACCESS

DOWNLOADABLE SOFTWARE

SUPPORT

NSF SAGE operates, provides, and maintains geoscience instrumentation

Permanent Networks

Portable Instrumentation

USArray

Community Engagement

Collaborations

New Directions

Past Projects

NSF SAGE provides a wide range of education, workforce, and outreach resources

LEARNING & TEACHING RESOURCES

LEARNING OPPORTUNITIES

PUBLIC OUTREACH

EXPLORE EARTHQUAKE DATA

ABOUT EarthScope

COMMUNITY

EVENTS

PUBLICATIONS

EarthScope ORGANIZATION

Thread: StationEpoch duplicates

Connect