[netCDFJava #XAS-459834]: Re: bufr reading

looking at this doc:

http://www.wmo.int/pages/prog/www/WMOCodes/Guides/BUFRCREX/Layer3-English-only.doc

3.1.2.3 and 3.2.2.3 goes into some detail.

"Now, consider the effect of replication on increment descriptors. Regulation 
94.5.4.3
states: “Time or location increment descriptors, from classes 04 to 07 
inclusive, may be
associated with replication descriptors in the following way: when an increment 
descriptor
immediately precedes a replication descriptor, or is separated from it by one 
or more
operator descriptors from Table C this shall infer that all such increments be 
applied for
each replication; the application of the increments shall have effect from the 
beginning of
each defined replication, including the first.”

my understanding is that the rules are:

1. a coordinate element is "in effect" until superseded or by another element 
of the same type, in the same dataset. There is also some mechanisms to 
"cancel" the coordinate.
2. A new dataset starts this over. In the CDM, each dataset is one "obs" in the 
sequence.
3. a "coordinate increment" adds to the existing coordinate of the same type, 
rather than replaces.
4. for replication, if there is a coordinate increment in effect, it is added 
for each replication.

let me know if you think this set of rules is correct. i generally proceed by 
trial and error, getting real examples to work correctly. you have to check as 
you add new files.


> Using a sequence, how do we determine when to stop applying an
> increment to a set of a data. I'm using the BUFR DART data dump below as
> an example.
> 
> Message 0 header=964IOZK01 AMMC 130013
> Sequence obs count=0
> var='time' units='secs since 2009-07-12T00:13:13Z' : 0.0
> var='Buoy/platform identifier' units='null' : 55013.0
> var='Platform transmitter ID' units='CodeTable 0-1-52' : 1.0
> var='Deep-ocean tsunameter type' units='CodeTable 0-2-47' : 1.0
> var='Year' units='Year' : 2009.0
> var='Month' units='Month' : 7.0
> var='Day' units='Day' : 12.0
> var='Hour' units='Hour' : 0.0
> var='Minute' units='Minute' : 13.0
> var='Second' units='Second' : 13.0
> var='sampling interval (time)' units='Second' : 900.0
> var='sample averaging period' units='Second' : 15.0
> var='Number of samples' units='null' : 1.0
> Sequence seq1 count=0
> var='Quality information' units='CodeTable 0-33-2' : Data not
> suspect
> var='Year' units='Year' : 2009.0
> var='Month' units='Month' : 7.0
> var='Day' units='Day' : 12.0
> var='Hour' units='Hour' : 18.0
> var='Minute' units='Minute' : 15.0
> var='Second' units='Second' : 0.0
> var='Battery voltage' units='V' : 14.699999809265137
> var='Battery voltage-1' units='V' : 4.099999904632568
> var='Battery voltage (large range)' units='V' : 26.0
> var='BPR transmission count' units='null' : 1.0
> var='Time increment' units='Minute' : -15.0
> var='Short time increment' units='Minute' : 15.0
> var='Water column height' units='m' :
> 4919.7451171875,4919.68310546875,4919.6240234375,4919.5703125
> 
> We have a time increment which should be applied to the replicated
> water column height measurements. In this case, it's obvious that we
> apply the increments to the rest of the variables in the sequence.
> However, we can't always make this assumption. What if there were other
> variables that followed the replicated water column measurements. See
> example below. How can we determine if 'Some Variable' is part of a
> single replication and should have the increments applied or is a
> separate replication with no increments applied? Would there be a way to
> determine which variables get the increments applied and which ones
> don't within the sequence?
> 
> var='Time increment' units='Minute' : -15.0
> var='Short time increment' units='Minute' : 15.0
> var='Water column height' units='m' :
> 4919.7451171875,4919.68310546875,4919.6240234375,4919.5703125
> var='Some Variable' units='m' :
> 4919.7451171875,4919.68310546875,4919.6240234375,4919.5703125
> var='Variable 1' units='V' : 14.699999809265137
> var='Variable 2' units='V' : 4.099999904632568
> var='Variable 3' units='V' : 26.0
> 
> Thanks,
> Michelle
> 
> 
> 
> 
> 
> 
> ----- Original Message -----
> From: Unidata netCDF Java Support <address@hidden>
> Date: Thursday, January 28, 2010 7:43 pm
> Subject: [netCDFJava #XAS-459834]: Re: bufr reading
> > A Sequence contains a BUFR "deferred replication", ie a variable-
> > length number of replicated fields. It may or may not have a
> > coordinate increment.
> >
> > > Hi John,
> > >
> > > From a BUFR perspective, is it safe to assume that a
> > 'DataType.SEQUENCE'> will always contain replicated data and an
> > increment?>
> > > Thanks,
> > > Michelle
> > >
> > > ----- Original Message -----
> > > From: John Caron <address@hidden>
> > > Date: Thursday, January 21, 2010 5:20 pm
> > > Subject: Re: bufr reading
> > > > address@hidden wrote:
> > > > > Thanks for the link. Although I'm sure you've already seen this
> > > > > document, I thought I would pass it along anyway -
> > > > > http://www.wmo.ch/pages/prog/www/WDM/Guides/Guide-binary-
> > 1B.html.> > >
> > > > > The example I sent you is a DART/Tsunameter message.
> > > > >
> > > > > I believe there was some miscommunication on my part. What I was
> > > > trying> to ask - is there a method in your BUFR libraries that
> > will> > allow me to
> > > > > easily apply the time increments to the appropriate data.
> > I've been
> > > > > trying to figure out a way to use the SequenceDS object
> > created in
> > > > > processBufrMessageDataset in the BufrDataDump program you sent
> > > > me. I
> > > > > need to convert the data into a format (platform id, observation
> > > > date,> location, depth, value) that can be ingested into the OSMC.
> > > > For this
> > > > > particular message, we're only interested in the water column
> > > > height and
> > > > > the observations should look like:
> > > > >
> > > > > platform id   observation date        location
> > > >         depth   value
> > > > > 55013         7/12/2009 18:15:00      I get the location from
> > > > metadata                0
> > > > > 4919.7451171875
> > > > > 55013         7/12/2009 18:30:00
> > > >                         0       4919.68310546875
> > > > > 55013         7/12/2009 18:45:00
> > > >                         0       4919.6240234375
> > > > > 55013         7/12/2009 19:00:00
> > > >                         0       4919.5703125
> > > > >
> > > > > The problem I'm having is that for the DART BUFR messages (or
> > any> > > message where the following occurs) the increments occur in
> > > > conjunction> with the replication. Using the SequenceDs object,
> > > > > I'm not sure how to determine which data to apply the
> > increments.> > Do you
> > > > > have any suggestions and/or recommendations?
> > > > >
> > > > > Thanks!!
> > > > > Michelle
> > > > >
> > > > looking more closely at the BUFR.ORIG file you sent, there are
> > > > actually
> > > > two kinds of messages in there, Tsunameter data (template 3-6-30)
> > > > and
> > > > Surface Data (template 3-6-28), heres a dump of the two along with
> > > > an
> > > > example:
> > > >
> > > > 1)
> > > >
> > > > BUFR edition 4 time= Sat Jul 11 18:13:13 MDT 2009
> > > > wmoHeader=964IOZK01
> > > > AMMC 130013 hash=[0xd1c00f4d] listHash=[0xffffc63d] (-14787)
> > > >   Category= 31.7.112=Oceanographic data / Tsunameter data
> > > >   Center= 1.0 (Melbourne)
> > > >   Table= 0.13.0
> > > >    Table B= wmoTable= wmo.v13.composite localTable= none
> > mode=wmoOnly> >    Table D= wmoTable=
> > > > resource:/resources/bufrTables/wmo/TableD-121509.csv
> > localTable= none
> > > >  DDS nsubsets=6 type=0xc0 isObs=true isCompressed=true
> > > >  startPos=31 len=140 endPos=171 dataStart=73 dataLen=94
> > dataEnd=167> >    3-6-30  : (Sequence for representation of DART
> > buoy standard
> > > > hourly
> > > > report)
> > > >      3-6-27  : (Sequence for representation of DART buoy
> > > > identification  transmitter ID  type of tsunameter and the time
> > the> > message is transmitted to the ground system)
> > > >        0-1-5   : Buoy/platform identifier
> > > >        0-1-52  : Platform transmitter ID
> > > >        0-2-47  : Deep-ocean tsunameter type
> > > >        3-1-11  :
> > > >          0-4-1   : Year
> > > >          0-4-2   : Month
> > > >          0-4-3   : Day
> > > >        3-1-13  :
> > > >          0-4-4   : Hour
> > > >          0-4-5   : Minute
> > > >          0-4-6   : Second
> > > >      3-6-29  : (Sequence for representation of tsunameter sampling
> > > > information for water column heights in the time series report)
> > > >        0-25-170: sampling interval (time)
> > > >        0-25-171: sample averaging period
> > > >        0-25-172: Number of samples
> > > >      1-11-000: replication
> > > >      0-31-1  : Delayed descriptor replication factor
> > > >      0-33-2  : Quality information
> > > >      3-1-11  :
> > > >        0-4-1   : Year
> > > >        0-4-2   : Month
> > > >        0-4-3   : Day
> > > >      3-1-13  :
> > > >        0-4-4   : Hour
> > > >        0-4-5   : Minute
> > > >        0-4-6   : Second
> > > >      0-25-25 : Battery voltage
> > > >      0-25-25 : Battery voltage
> > > >      0-25-26 : Battery voltage (large range)
> > > >      0-22-185: BPR transmission count
> > > >      0-4-15  : Time increment
> > > >      0-4-65  : Short time increment
> > > >      1-01-004: replication
> > > >      0-22-182: Water column height
> > > >
> > > >  CDM Nested Table=
> > > >    0-1-5: Buoy/platform identifier units=Numeric scale=0 refVal=0
> > > > nbits=17    0-1-52: Platform transmitter ID units=Code table enum
> > > > nbits=3    0-2-47: Deep-ocean tsunameter type units=Code table
> > enum> > nbits=7    0-4-1: Year units=Year scale=0 refVal=0 nbits=12
> > > >    0-4-2: Month units=Month scale=0 refVal=0 nbits=4
> > > >    0-4-3: Day units=Day scale=0 refVal=0 nbits=6
> > > >    0-4-4: Hour units=Hour scale=0 refVal=0 nbits=5
> > > >    0-4-5: Minute units=Minute scale=0 refVal=0 nbits=6
> > > >    0-4-6: Second units=Second scale=0 refVal=0 nbits=6
> > > >    0-25-170: sampling interval (time) units=Second scale=0
> > > > refVal=0
> > > > nbits=10
> > > >    0-25-171: sample averaging period units=Second scale=0 refVal=0
> > > > nbits=10    0-25-172: Number of samples units=Numeric scale=0
> > > > refVal=0 nbits=10
> > > >    1-11-0: Replication count=0 replicationCountSize=8
> > > >      0-33-2: Quality information units=Code table enum nbits=2
> > > >      0-4-1: Year units=Year scale=0 refVal=0 nbits=12
> > > >      0-4-2: Month units=Month scale=0 refVal=0 nbits=4
> > > >      0-4-3: Day units=Day scale=0 refVal=0 nbits=6
> > > >      0-4-4: Hour units=Hour scale=0 refVal=0 nbits=5
> > > >      0-4-5: Minute units=Minute scale=0 refVal=0 nbits=6
> > > >      0-4-6: Second units=Second scale=0 refVal=0 nbits=6
> > > >      0-25-25: Battery voltage units=V scale=1 refVal=0 nbits=9
> > > >      0-25-25: Battery voltage units=V scale=1 refVal=0 nbits=9
> > > >      0-25-26: Battery voltage (large range) units=V scale=1
> > > > refVal=0
> > > > nbits=12
> > > >      0-22-185: BPR transmission count units=Numeric scale=0
> > > > refVal=0
> > > > nbits=10
> > > >      0-4-15: Time increment units=Minute scale=0 refVal=-2048
> > > > nbits=12      0-4-65: Short time increment units=Minute scale=0
> > > > refVal=-128 nbits=8
> > > >      1-1-4: Replication count=4
> > > >        0-22-182: Water column height units=m scale=3 refVal=0
> > > > nbits=23
> > > > time =0 secs since 2009-07-12T00:13:13Z
> > > >  Buoy/platform identifier =55013
> > > >  Platform transmitter ID =1 CodeTable 0-1-52
> > > >  Deep-ocean tsunameter type =1 CodeTable 0-2-47
> > > >  Year =2009 Year
> > > >  Month =7 Month
> > > >  Day =12 Day
> > > >  Hour =0 Hour
> > > >  Minute =13 Minute
> > > >  Second =13 Second
> > > >  sampling interval (time) =900 Second
> > > >  sample averaging period =15 Second
> > > >  Number of samples =1
> > > >  seq1 =
> > > >    {
> > > >      Quality information =  "Data not suspect" CodeTable 0-33-2
> > > >      Year =2009 Year
> > > >      Month =7 Month
> > > >      Day =12 Day
> > > >      Hour =18 Hour
> > > >      Minute =15 Minute
> > > >      Second =0 Second
> > > >      Battery voltage =14.7 V
> > > >      Battery voltage-1 =4.1 V
> > > >      Battery voltage (large range) =26.0 V
> > > >      BPR transmission count =1
> > > >      Time increment =-15.0 Minute
> > > >      Short time increment =15.0 Minute
> > > >      Water column height =
> > > >        {4919.745, 4919.683, 4919.624, 4919.5703} m
> > > >    } obs.seq1(0)
> > > >
> > > > 2)
> > > >
> > > > BUFR edition 4 time= Sat Jul 11 18:19:18 MDT 2009
> > > > wmoHeader=699IOZK10
> > > > AMMC 130019 hash=[0xd1be6296] listHash=[0xffffc63b] (-14789)
> > > >   Category= 31.0.112=Oceanographic data / Surface observation
> > > >   Center= 1.0 (Melbourne)
> > > >   Table= 0.13.0
> > > >    Table B= wmoTable= wmo.v13.composite localTable= none
> > mode=wmoOnly> >    Table D= wmoTable=
> > > > resource:/resources/bufrTables/wmo/TableD-121509.csv
> > localTable= none
> > > >  DDS nsubsets=1 type=0x80 isObs=true isCompressed=false
> > > >  startPos=206 len=70 endPos=276 dataStart=248 dataLen=24
> > dataEnd=272> >    3-6-28  : (Sequence for representation of time of
> > observation> > and
> > > > DART buoy position daily report)
> > > >      3-6-27  : (Sequence for representation of DART buoy
> > > > identification  transmitter ID  type of tsunameter and the time
> > the> > message is transmitted to the ground system)
> > > >        0-1-5   : Buoy/platform identifier
> > > >        0-1-52  : Platform transmitter ID
> > > >        0-2-47  : Deep-ocean tsunameter type
> > > >        3-1-11  :
> > > >          0-4-1   : Year
> > > >          0-4-2   : Month
> > > >          0-4-3   : Day
> > > >        3-1-13  :
> > > >          0-4-4   : Hour
> > > >          0-4-5   : Minute
> > > >          0-4-6   : Second
> > > >      3-1-11  :
> > > >        0-4-1   : Year
> > > >        0-4-2   : Month
> > > >        0-4-3   : Day
> > > >      3-1-13  :
> > > >        0-4-4   : Hour
> > > >        0-4-5   : Minute
> > > >        0-4-6   : Second
> > > >      3-1-21  :
> > > >        0-5-1   : Latitude (high accuracy)
> > > >        0-6-1   : Longitude (high accuracy)
> > > >
> > > >  time =0 secs since 2009-07-12T00:19:18Z
> > > >  Buoy/platform identifier =56003
> > > >  Platform transmitter ID =0 CodeTable 0-1-52
> > > >  Deep-ocean tsunameter type =1 CodeTable 0-2-47
> > > >  Year =2009 Year
> > > >  Month =7 Month
> > > >  Day =12 Day
> > > >  Hour =0 Hour
> > > >  Minute =19 Minute
> > > >  Second =18 Second
> > > >  Year-1 =2009 Year
> > > >  Month-1 =7 Month
> > > >  Day-1 =12 Day
> > > >  Hour-1 =6 Hour
> > > >  Minute-1 =47 Minute
> > > >  Second-1 =40 Second
> > > >  Latitude (high accuracy) =-15.019232 degrees_north
> > > >  Longitude (high accuracy) =118.004105 degrees_east
> > > >
> > > >
> > > > the first one has the nested sequence, the second only seems to
> > > > have
> > > > lat/lon.
> > > >
> > > > the nested sequence is rather awkward to process, but you just
> > have> > to
> > > > follow it in and extract the data. Ive attached a quick and dirty
> > > > example program that should get you started. output looks like:
> > > >
> > > > Dump D:/work/michelle/TimeIncr.bufr
> > > > Message 0 header=964IOZK01 AMMC 130013
> > > > Extract{platformId=55013.0, year=2009, month=7, day=12, hour=18,
> > > > min=15,
> > > > sec=0, incr=-15, incrS=15, value=4919.745 4919.683 4919.624
> > > > 4919.5703 }
> > > > Extract{platformId=55013.0, year=2009, month=7, day=12, hour=19,
> > > > min=15,
> > > > sec=0, incr=-15, incrS=15, value=4919.5303 4919.4854 4919.4443
> > > > 4919.413 }
> > > > Extract{platformId=55013.0, year=2009, month=7, day=12, hour=20,
> > > > min=15,
> > > > sec=0, incr=-15, incrS=15, value=4919.3853 4919.3604 4919.3423
> > > > 4919.333 }
> > > > Extract{platformId=55013.0, year=2009, month=7, day=12, hour=21,
> > > > min=15,
> > > > sec=0, incr=-15, incrS=15, value=4919.3354 4919.3394 4919.352
> > > > 4919.37 }
> > > > Extract{platformId=55013.0, year=2009, month=7, day=12, hour=22,
> > > > min=15,
> > > > sec=0, incr=-15, incrS=15, value=4919.4043 4919.4424 4919.4834
> > > > 4919.5293 }
> > > > Extract{platformId=55013.0, year=2009, month=7, day=12, hour=23,
> > > > min=15,
> > > > sec=0, incr=-15, incrS=15, value=4919.579 4919.637 4919.6973
> > > > 4919.7603 }
> > > > ......
> > > >
> > > >
> > > > let me know how it goes
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: XAS-459834
> > Department: Support netCDF Java
> > Priority: Normal
> > Status: Open
> >
> >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: XAS-459834
Department: Support netCDF Java
Priority: Urgent
Status: Open