[GEMPAK #CSI-145319]: dcmetr and Antarctic AWS sites

Robert,

I don't know all the permutations of sed under solaris (there is /usr/bin/sed, 
/usr/ucb/sed,
and /usr/xpg4/bin/sed).

The translation could be accomplished with "tr" and "sed" together, but the 
following nawk
will work if you don't want another 3rd party sed to maintain:

cat obs200611132000 | nawk '{gsub("\r","\r\r"); print $0}' | dcmetr.......

The -m 72 bins data to 20 minute intervals, otherwise hourly bins are
used. There isn't currently a facility with dcmetr to not bin the times
(at that point, you'd probably do better by getting the actual obs rather than
a conversion of their obs back into metar).


Steve Chiswell
Unidata User Support

> Steve,
> 
> Thanks for all the help.  I now have the obs piped to a c-schell script via 
> the LDM.  They are`all decoding fine, except for the fact they are every 15 
> minutes, and dcmetr "rounds" the time to nearest 20 minutes, but that is no 
> big deal.
> 
> 
> -----Original Message-----
> From: Unidata GEMPAK Support [mailto:address@hidden]
> Sent: Mon 11/13/2006 5:33 PM
> To: Robert Mullenax
> Cc: address@hidden
> Subject: [GEMPAK #CSI-145319]: dcmetr and Antarctic AWS sites
> 
> > I went back and sourced Gemenviron again and made sure it found $GEMTBL 
> > correctly.  It ran without the metar.pack error, but still did not produce 
> > output.  I feel really stupid just about now..
> >
> >
> > > source ~gempak/GEMPAK/Gemenviron
> > > cat obs200611132000 | sed 's/\r/\r\r/g' | dcmetr -v 4 -d - -c 061113/2000 
> > > YYYYMMDDHH_test.gem
> > [6868] 061113/2117[DC 3]  Starting up. Version 5.9.3
> > [6868] 061113/2117[DCMETR 7]  DCMETR version:  3.3
> > [6868] 061113/2117[DC 2]  read 83/204799 bytes strt 0 newstrt 83
> > [6868] 061113/2117[DC 2]  read 0/204716 bytes strt 83 newstrt 83
> > [6868] 061113/2117[DC -9]  End of input data file.
> > [6868] 061113/2117[DC 5]  Normal termination.
> > [6868] 061113/2117[DC 2]  Number of bulletins read and processed: 0
> > [6868] 061113/2117[DC 6]  Shutting down.
> 
> Robert,
> 
> The file you sent me was 82 bytes as the attachment. There are 4 instances of
> "\r" in the file (see with od -c obs200611132000). That means that the sed 
> replacement of
> "\r" with "\r\r" should yeil 86 bytes of input:
> [2635166] 061113/1625[DCMETR 7]  DCMETR version:  3.3
> [2635166] 061113/1625[DC 2]  read 86/102399 bytes strt 0 newstrt 86
> [2635166] 061113/1625[DC 2]  read 0/102314 bytes strt 86 newstrt 86
> [2635166] 061113/1625[DC -9]  End of input data file.
> [2635166] 061113/1625[DC 5]  Normal termination.
> [2635166] 061113/1625[DC 2]  Number of bulletins read and processed: 1
> [2635166] 061113/1625[DC 6]  Shutting down.
> 
> I see 86 bytes of input.
> 
> You show only 82 and 83 bytes of input in the 2 sets of log messages
> you sent me which just doesn't match the file you sent me earlier.
> An "od -c" of the file you sent me piped to that sed command looks like:
> % cat obs200611132000 | sed 's/\r/\r\r/g' | od -c
> 0000000 001  \r  \r  \n   9   2   1  \r  \r  \n   S   A   A   A   1   0
> 0000020       S   S   C   C       1   3   2   0   0   0  \r  \r  \n   M
> 0000040   E   T   A   R       B   I   W   S       1   3   2   0   0   0
> 0000060   Z       A   U   T   O       1   9   0   4   3   G   5   1   K
> 0000100   T       M   1   2   /   M   2   2       R   M   K       A   0
> 0000120   1   =  \r  \r  \n 003
> 
> 
> Verify that you have that identical input.
> 
> Steve Chiswell
> Unidata User Support
> 
> >
> >
> >
> > -----Original Message-----
> > From: Unidata GEMPAK Support [mailto:address@hidden]
> > Sent: Mon 11/13/2006 3:02 PM
> > To: Robert Mullenax
> > Cc: address@hidden
> > Subject: [GEMPAK #CSI-145319]: dcmetr and Antarctic AWS sites
> >
> > >
> > > Steve,
> > >
> > > I just tried it with a a new file and this does not work for me.  I have 
> > > attached the file.
> > > Where did I go wrong?
> > >
> > >
> > > > cat obs200611132000 | sed 's/\r/\r\r/g' | dcmetr -v 4 -d - -c 
> > > > 061113/2000 YYYYMMDDHH_metar.gem
> > > [4322] 061113/2007[DC 3] Version 5.9.3
> > > [4322] 061113/2007[DCMETR 7] 3.3
> > > [4322] 061113/2007[IN -11] metar.pack
> >
> > Robert,
> >
> > The file was fine and decoded here. Your problem from the line above is 
> > that metar.pack
> > wasn't found, which suggests that your GEMTBL variable isn't pointing to the
> > correct place (or it's not set at all). Check your "source Gemenviron".
> >
> > Steve Chiswell
> > Unidata User Support
> >
> >
> >
> >
> >
> >
> >
> > > [4322] 061113/2007[DC 2] read 81/204799 bytes strt 0 newstrt 81
> > > [4322] 061113/2007[DC 2] read 0/204718 bytes strt 81 newstrt 81
> > > [4322] 061113/2007[DC -9]
> > > [4322] 061113/2007[DC 5]
> > > [4322] 061113/2007[DC 2] Number of bulletins read and processed: 0
> > > [4322] 061113/2007[DC 6]
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Unidata GEMPAK Support [mailto:address@hidden]
> > > Sent: Fri 11/10/2006 2:50 PM
> > > To: Robert Mullenax
> > > Cc: address@hidden
> > > Subject: [GEMPAK #CSI-145319]: dcmetr and Antarctic AWS sites
> > >
> > > Robert,
> > >
> > > As expected, the problem with your report is the unseen carriage return 
> > > characters.
> > > Your file uses a single \r\n sequence at each line, while the FOS 
> > > patterns expect
> > > \r\r\n. In particular, the start of a bulletin is requires to be SOH \r 
> > > \r \n
> > > and the end of a bulletin is required to be \r \r \n ETX. Without that 
> > > correct
> > > sequence, the decoder is just reading bytes looking for a valid bulletin.
> > >
> > > You can modify and decode your data through a pipe such as:
> > > cat BIWS2006111017 | sed 's/\r/\r\r/g' | dcmetr -v 4 -d - -c 061110/1700 
> > > YYYYMMDDHH_metar.gem
> > >
> > > Steve Chiswell
> > > Unidata User Support
> > >
> > >
> > > > Our mail server has not been sending attachments out lately  (I have no 
> > > > idea why)..if you don't get this maybe I could send you the AWS data 
> > > > via LDM for a short period of time?
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Unidata GEMPAK Support [mailto:address@hidden]
> > > > Sent: Fri 11/10/2006 10:40 AM
> > > > To: Robert Mullenax
> > > > Cc: address@hidden
> > > > Subject: [GEMPAK #CSI-145319]: dcmetr and Antarctic AWS sites
> > > >
> > > > Robert,
> > > >
> > > > Please send either as an attachment or post since when you cut and
> > > > paste, you are translating the control characters, and we need to
> > > > verify that you have them correct since I can get the bulletin to decode
> > > > when I put in the correct bulletin delimeters.
> > > >
> > > > Steve Chiswell
> > > > Unidata User Support
> > > >
> > > >
> > > > > I used FILE   -close.  I cut and pasted it into the e-mail.
> > > > >
> > > > > ^A^M
> > > > > 285^M
> > > > > SAAA10 SSCC 092345^M
> > > > > METAR BIWS 092345Z AUTO 26003KT M10/M18 RMK A01=^M
> > > > > ^C
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Unidata GEMPAK Support [mailto:address@hidden]
> > > > > Sent: Thu 11/9/2006 1:36 PM
> > > > > To: Robert Mullenax
> > > > > Cc: address@hidden
> > > > > Subject: [GEMPAK #CSI-145319]: dcmetr and Antarctic AWS sites
> > > > >
> > > > > Robert,
> > > > >
> > > > > It sounds like you do not have proper bulletin delimeters  so that 
> > > > > the decoder can find
> > > > > the start and end of the bulletin.
> > > > >
> > > > > Please send me the raw data from the pqact FILE command.
> > > > >
> > > > > Steve Chiswell
> > > > > Unidata User Supporrt
> > > > >
> > > > >
> > > > >
> > > > > > Data arrived at 1846 and the decoder started up, but no file was 
> > > > > > written and the log file shows it has closed without processing a 
> > > > > > bulletin:
> > > > > >
> > > > > > [5945] 061109/1646[DC 3]  Starting up. Version 5.9.3
> > > > > > [5945] 061109/1651[DC 2]  Interrupt Signal
> > > > > > [5945] 061109/1651[DC 5]  Normal termination.
> > > > > > [5945] 061109/1651[DC 2]  Number of bulletins read and processed: 0
> > > > > > [5945] 061109/1651[DC 6]  Shutting down.
> > > > > > [6084] 061109/1701[DC 3]  Starting up. Version 5.9.3
> > > > > > [6084] 061109/1701[DCMETR 7]  DCMETR version:  3.3
> > > > > > [6084] 061109/1701[DC 2]  read 79/204799 bytes strt 0 newstrt 79
> > > > > > [6084] 061109/1710[DC 2]  Interrupt Signal
> > > > > > [6084] 061109/1710[DC 5]  Normal termination.
> > > > > > [6084] 061109/1710[DC 2]  Number of bulletins read and processed: 0
> > > > > > [6084] 061109/1710[DC 6]  Shutting down.
> > > > > > [21063] 061109/1846[DC 3] Version 5.9.3
> > > > > > [21063] 061109/1846[DCMETR 7] 3.3
> > > > > > [21063] 061109/1846[DC 2] read 79/204799 bytes strt 0 newstrt 79
> > > > > > [21063] 061109/1856[DC -6]
> > > > > > [21063] 061109/1856[DC 5]
> > > > > > [21063] 061109/1856[DC 2] Number of bulletins read and processed: 0
> > > > > > [21063] 061109/1856[DC 6]
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Robert Mullenax
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Unidata GEMPAK Support [mailto:address@hidden]
> > > > > > Sent: Thu 11/9/2006 12:12 PM
> > > > > > To: Robert Mullenax
> > > > > > Cc: address@hidden
> > > > > > Subject: [GEMPAK #CSI-145319]: dcmetr and Antarctic AWS sites
> > > > > >
> > > > > > Robert,
> > > > > >
> > > > > > I clipped out your metar and sent it to dcmetr and it decoded:
> > > > > > [2273920] 061109/1104[DC 3]  Starting up. Version 5.9.4
> > > > > > [2273920] 061109/1104[DCMETR 7]  DCMETR version:  3.3
> > > > > > [2273920] 061109/1104[DC 2]  read 89/102399 bytes strt 0 newstrt 89
> > > > > > [2273920] 061109/1104[DC 2]  read 0/102311 bytes strt 89 newstrt 89
> > > > > > [2273920] 061109/1104[DC -9]  End of input data file.
> > > > > > [2273920] 061109/1104[DC 5]  Normal termination.
> > > > > > [2273920] 061109/1104[DC 2]  Number of bulletins read and 
> > > > > > processed: 1
> > > > > > [2273920] 061109/1104[DC 6]  Shutting down.
> > > > > >
> > > > > > GEMPAK-SFLIST>r
> > > > > > PARM = 
> > > > > > PMSL;ALTI;TMPC;DWPC;SKNT;DRCT;GUST;WNUM;CHC1;CHC2;CHC3;VSBY;P03D;P03I;
> > > > > > MSUN;SNOW;WEQS;P24I;TDXC;TDNC;P03C;CTYL;CTYM;CTYH;P06I;T6XC;T6NC;CEIL;
> > > > > > P01I
> > > > > >
> > > > > > STN    YYMMDD/HHMM      PMSL     ALTI     TMPC     DWPC     SKNT    
> > > > > >  DRCT
> > > > > > GUST     WNUM     CHC1     CHC2     CHC3     VSBY
> > > > > > P03D     P03I     MSUN     SNOW     WEQS     P24I
> > > > > > TDXC     TDNC     P03C     CTYL     CTYM     CTYH
> > > > > > P06I     T6XC     T6NC     CEIL     P01I
> > > > > > BIWS    061109/1700  -9999.00 -9999.00   -13.00   -20.00     3.00   
> > > > > > 210.00
> > > > > > -9999.00 -9999.00 -9999.00 -9999.00 -9999.00 -9999.00
> > > > > > -9999.00 -9999.00 -9999.00 -9999.00 -9999.00 -9999.00
> > > > > > -9999.00 -9999.00 -9999.00 -9999.00 -9999.00 -9999.00
> > > > > > -9999.00 -9999.00 -9999.00 -9999.00 -9999.00
> > > > > >
> > > > > > when you say nothing happened, I'm assuming that you mean you have 
> > > > > > no files named
> > > > > > YYYYMMDD_saot.gem in your surface directory? If you do, then you 
> > > > > > are probably just waiting for
> > > > > > the data to flush the write before you can see it (without any 
> > > > > > other data, the decoder will flush in 600 seconds by default but 
> > > > > > can be adjusted with -t).
> > > > > >
> > > > > > Otherwise, you might want to do some tests by cat'ing the FILE'd 
> > > > > > bulletin to the decoder
> > > > > > so you can test things out with -v 4. You can send me one of your 
> > > > > > metar bulletins for me to try if you
> > > > > > still have trouble.
> > > > > >
> > > > > > Steve Chiswell
> > > > > > Unidata User Support
> > > > > >
> > > > > > >
> > > > > > > The SPAWAR folks have deployed some weather stations around 
> > > > > > > McMurdo and are transmitting the obs in METAR format on the 
> > > > > > > Antarctic-IDD.  I am saving the obs in raw form, but it would be 
> > > > > > > ideal to use dcmetr to decode them.  Here is what the bulletin 
> > > > > > > looks like when saved to disk(the obs are generally every 15 
> > > > > > > minutes):
> > > > > > >
> > > > > > > ^A^M
> > > > > > > 721^M
> > > > > > > SAAA10 SSCC 091700^M
> > > > > > > METAR BIWS 091700Z AUTO 21003KT M13/M20 RMK A01=^M
> > > > > > > ^C
> > > > > > >
> > > > > > >
> > > > > > > Even though it has a header, it is not being sent through the NWS 
> > > > > > > Gateway.  I added an entry for BIWS in sfmetar_sa.tbl:
> > > > > > >
> > > > > > > BIWS      99999 BLACK ISLAND AWS(ANTARC)         -- NZ  7813  
> > > > > > > 16615   213
> > > > > > >
> > > > > > > and created a pqact entry as a test:
> > > > > > >
> > > > > > > EXP    ^USAP\.NZCM\.AWS\.BIWS\.(........).(..)(..)
> > > > > > > PIPE    decoders/dcmetr -v 2 -a 500 -m 72 -s sfmetar_sa.tbl
> > > > > > > -d data/gempak/logs/dcmetrt.log
> > > > > > > -e GEMTBL=/usr/gempak/GEMPAK5.9.3/gempak/tables
> > > > > > > data/gempak/surface/YYYYMMDD_saot.gem
> > > > > > >
> > > > > > >
> > > > > > > However, nothing gets decoded.  This is what dcmetrt.log says:
> > > > > > >
> > > > > > > [5945] 061109/1646[DC 3]  Starting up. Version 5.9.3
> > > > > > > [5945] 061109/1651[DC 2]  Interrupt Signal
> > > > > > > [5945] 061109/1651[DC 5]  Normal termination.
> > > > > > > [5945] 061109/1651[DC 2]  Number of bulletins read and processed: > > > > > > > 0
> > > > > > > [5945] 061109/1651[DC 6]  Shutting down.
> > > > > > > [6084] 061109/1701[DC 3]  Starting up. Version 5.9.3
> > > > > > > [6084] 061109/1701[DCMETR 7]  DCMETR version:  3.3
> > > > > > > [6084] 061109/1701[DC 2]  read 79/204799 bytes strt 0 newstrt 79
> > > > > > >
> > > > > > > It read the file (79 bytes is right), but then nothing happens.
> > > > > > >
> > > > > > > Is there any chance this can be successfully decoded?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Robert Mullenax
> > > > > > > NMSU/PSL/CSBF Meteoroology
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Ticket Details
> > > > > > ===================
> > > > > > Ticket ID: CSI-145319
> > > > > > Department: Support GEMPAK
> > > > > > Priority: Normal
> > > > > > Status: Closed
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > Ticket Details
> > > > > ===================
> > > > > Ticket ID: CSI-145319
> > > > > Department: Support GEMPAK
> > > > > Priority: Normal
> > > > > Status: Closed
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > Ticket Details
> > > > ===================
> > > > Ticket ID: CSI-145319
> > > > Department: Support GEMPAK
> > > > Priority: Normal
> > > > Status: Closed
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: CSI-145319
> > > Department: Support GEMPAK
> > > Priority: Normal
> > > Status: Closed
> > >
> > >
> > >
> > >
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: CSI-145319
> > Department: Support GEMPAK
> > Priority: Normal
> > Status: Closed
> >
> >
> >
> >
> 
> 
> Ticket Details
> ===================
> Ticket ID: CSI-145319
> Department: Support GEMPAK
> Priority: Normal
> Status: Closed
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: CSI-145319
Department: Support GEMPAK
Priority: Normal
Status: Closed