[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

query of community interest in decoder technology



Hi,

I just wanted to put out a general query to the community to see if you 
were looking for any additional decoders or would like to offer your
services to help generate additional decoders.

Due to the data needs of our Alaska Ocean Observing System, I've been
developing decoders for CMAN and SHEF encoded messages streaming on the 
IDD.  A recent meeting indicated that these decoders may be useful to 
other groups in the ocean observing and grid computing areas.

The CMAN decoder is fairly robust in that it functions much like the
buoy2nc decoder but still may need to have some bugs ironed out.

The SHEF decoder is really just starting up.  SHEF 2 has 3 main formats of
messages: .A .B .E

.A and .E are easy in that the message is very structured.
.B is a free format setup by the data provider

I have .A in a functional state, but I have broken the NetCDF storage
portion of the code.  I'm not a expert in debugging perl/NetCDF.  
However, the code currently writes data into our Postgresql database using
perl/DBI.  We went the perl/DBI route since we needed access to this 
information in to our system in the short term.

Brief explaination of SHEF:

.A is one or more parameters per specified time
.E is one parameter over a period of specified time
.B is free form, but we can attempt to build XML descriptions
  that attempt to parse the message into its parts.  Crosswalker
  from free-form to NetCDF or perl/DBI.

Current SHEF 2 specification:
  http://www.weather.gov/oh/hrl/shef/indexshef.htm

SHEF messages include data from various instruments:
USGS, BLM, RAWS and NWLON and doesn't stop there...

Building a decoder is one step of the process.  As I've worked with
the current decoders and these new ones, there are large hurdles at
times and may or may never be fully addressed.

Some of these issues are:
* Format is not strictly followed.  (EX: METAR)
* Time is not fully defined in the coded message and requires
  inferral by the bulletin time or using what shreds of time
  are in the coded message
* Time zone handling/issues - real time decoding vs historical
    reprocessing
* Metadata/station listings
* Overlapping networks (NDBC & CMAN & NOS; USGS/RAWS)
* Various degress of QA/QC
* Units! (touches on all coded messages)
* Many others...

I would be very interested in basic decoders that would allow other 
information such as satellite and radar to become available via NetCDF or
RDBMs so that it could be served via OGC/OpenDap protocols via a GIS 
engine such as Mapserver.

I'm interested in all feedback from the community and Unidata.  The 
attempt here is to energize the community rather than try and create
more work for Unidata.

I'm willing to become a focal point for the community as an interface to
Unidata.  What tasks may be involved is not fully understood at this time.
The Ocean Observing System is also undergoing some evoloution, it seems 
important now to try and forge some bridges between communities.  

Looking forward to your comments.
Rob
-- 
Rob Cermak : 907-474-7948 : FAX 907-474-7204 : PGP = 0x75869A6E
Alaska Ocean Observing System : Data Management 
School of Fisheries and Ocean Sciences : University of Alaska Fairbanks
address@hidden : address@hidden
address@hidden : address@hidden