[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: ldm decoder



On Wed, 7 Apr 2004, Doug Lindholm wrote:

> Hi Charlie,
>
> I'm curious about your comment on "turning these netcdf files into a
> stream". The NetCDF file format is designed to allow convenient data
> access via a programming API (e.g. Java, perl, C). Although I am not the
> authority, I don't think NetCDF is well suited for streaming. I think
> what you want to do (and maybe this is your plan) is modify the metar2nc
> decoder to spit out a stream of simplified data instead of creating the
> NetCDF file. Robb would be the best one to help you there.

Charlie,

Doug is correct in stating that netcdf is a local file storage format
that provides quick access to the data using a programming API. metar2nc
is a stream reader ( NOAAPORT stream ) used to create a netcdf file. More
comments below.



>
> Good luck,
> Doug
>
> Charles Moad wrote:
> >     I get identical output as well for all the mentioned feedtypes.  They
> > all say IDS|DDPLUS.
> >     You were right about the missing tab on the first line, so the syntax
> > passes now.  I am also seeing the netcdf files being created so all
> > seems to be working well in that aspect.
> >     The challenge we face now is turning these netcdf files into a stream.
> > The metar2nc script is not exactly a trivial, so any input as to how it
> > works might help.  We are currently using a c++ api to register a dquob
> > stream source, but I think we are going to try to wrap this api in perl
> > and modify the metar2nc script.

I don't know  what you mean by "register a dquob stream source"?
When I understand your requirements then maybe a solution. In a
nutshell, the metar data arrives in bulletins that contain multiple reports
that are usually sent multiple times from different distribution stations.
There are 5-25% of the reports that are mal-formed so they need to be
disgarded. Most of the surface data falls into this category.  Also the
other surface reports are cryptic too, some even arrive in multiple
sections that need to be combined to make the final report.

Robb...

> >     Thanks for you help so far.
> >
> > - Charlie
> >
> > On Tue, 2004-04-06 at 17:02, Tom Baltzer wrote:
> >
> >>Hi Charlie,
> >>
> >>Well that's very interesting.  My understanding is that DDPLUS is just a
> >>combination of DDS and PPS (Public Products Service) so if you're able to
> >>get DDPLUS I'd have thought you could get DDS...
> >>
> >>I just spoke with a Jeff Weber (a local expert) who tells me that IDS|DDPLUS
> >>is a single feed composed of DDS, PPS and IDS and you cannot separate these
> >>by feed type. And in fact when I run the two ldmadmin watch -f commands you
> >>show (in two separate windows) I get identical results. I even went so far
> >>as to try each (PPS, DDS, PPS and DDPLUS) individually in 4 windows and got
> >>identical results.  What result do you get when you run the ldmadmin watch
> >>with the IDS|DDS feedtype?
> >>
> >>As far as the error message you're getting from checking the pqact.conf
> >>file, I suspect that there is yet another TAB required.  Check that the
> >>white space between IDS|DDPLUS and ^S in your pqact.conf file is in fact a
> >>tab rather than a space (sorry I didn't clarify this earlier).
> >>
> >>Let me know how it goes and/or if I can provide any other help.
> >>
> >>Best regards,
> >>Tom.
> >>
> >>
> >>
> >>>-----Original Message-----
> >>>From: Charles W Moad [mailto:address@hidden]
> >>>Sent: Tuesday, April 06, 2004 1:23 PM
> >>>To: Tom Baltzer
> >>>Cc: 'Beth A Plale'; 'Charles W Moad'; 'Tom Baltzer'; 'Doug Lindholm'
> >>>Subject: RE: ldm decoder
> >>>
> >>>As a first step I am trying run the standard pqact | metar2nc and save
> >>>the data files.  When I run "ldmadmin pqactcheck" I get the following
> >>>error:
> >>>
> >>>---
> >>>Apr 06 19:20:17 pqact[26196]: feedtype error at line 121: unknown feed
> >>>name in feedtype expression: "IDS|DDPLUS ^S(A....|P....|XUS8.) ....
> >>>([0-3][0-9])"
> >>>---
> >>>
> >>>I don't think we get the DDS|IDS feed because it does not show up as a
> >>>valid feedtype for "ldmadmin watch -f "DDS\|IDS"", however "ldmadmin
> >>>watch -f "IDS\|DDPLUS"" does work.
> >>>
> >>>Any help would be greatly appreciated.
> >>>
> >>>Thanks,
> >>>   Charlie
> >>>
> >>>On Tue, 2004-04-06 at 12:43, Tom Baltzer wrote:
> >>>
> >>>>Hi Beth and Charlie,
> >>>>
> >>>>It sounds like you have found the Unidata netCDF decoder package.  In my
> >>>>short time around here I've discovered that there are more decoders than
> >>>>just those found in that package and have been working on documenting
> >>>
> >>>those
> >>>
> >>>>that I see used by LDM systems around Unidata here:
> >>>>http://my.unidata.ucar.edu/content/projects/LEAD/Decoders.htm
> >>>>
> >>>>The pqact.conf configuration lines for pulling out metar data and
> >>>
> >>>placing it
> >>>
> >>>>in netcdf files in the ~ldm/data/pub/decoded/netcdf/surface directory
> >>>
> >>>are
> >>>
> >>>>(note the first 5 lines are comments):
> >>>>
> >>>>#
> >>>># metar decoder, creates NetCDF files formatted
> >>>>#    data/decoded/netcdf/surface/yymmddhh_metar.nc
> >>>>#
> >>>># All aviation reports including metar tests
> >>>>DDS|IDS ^S(A....|P....|XUS8.) .... ([0-3][0-9])
> >>>>        PIPE    decoders/metar2nc
> >>>>        etc/metar.cdl
> >>>>        data/pub/decoded/netcdf/surface
> >>>>        (\2:yy)(\2:mm)
> >>>>
> >>>>Be advised that those are <tab>s at the beginning of lines 2-5 (very
> >>>>important!).  Also be advised that it appears the decoder expects the
> >>>>directory structure to exist and be writable before it will work.  I've
> >>>>submitted this to the decoder folks as a fix.
> >>>>
> >>>>As Doug mentioned the pqact application will stream the raw data reports
> >>>
> >>>to
> >>>
> >>>>stdin and the metar2nc perl script then takes that streaming data and
> >>>
> >>>writes
> >>>
> >>>>it to a netCDF file.  So if you wished to you could take the metar2nc
> >>>
> >>>script
> >>>
> >>>>and modify it to act as a dQUOB interface. Or if dQUOB takes stdin then
> >>>
> >>>you
> >>>
> >>>>could use it to help you parse the incoming metar data. My understanding
> >>>
> >>>is
> >>>
> >>>>that you all have been trying to access the LDM product queue to do your
> >>>>dQUOB work.  It seems that using pqact/decoder type functions would be
> >>>
> >>>much
> >>>
> >>>>easier and effectively the same.
> >>>>
> >>>>Well I hope this helps.  Feel free to contact me directly with questions
> >>>
> >>>or
> >>>
> >>>>comments.
> >>>>
> >>>>Cheers,
> >>>>Tom.
> >>>>
> >>>>Tom Baltzer
> >>>>Software Engineer
> >>>>UCAR Office of Programs
> >>>>Unidata Program Center
> >>>>P.O. Box 3000
> >>>>Boulder, CO 80307-3000
> >>>>address@hidden
> >>>>303-497-8637
> >>>>
> >>>>
> >>>>
> >>>>>-----Original Message-----
> >>>>>From: Doug Lindholm [mailto:address@hidden]
> >>>>>Sent: Friday, April 02, 2004 11:52 AM
> >>>>>To: Beth A Plale
> >>>>>Cc: Charles W Moad; Tom Baltzer
> >>>>>Subject: Re: ldm decoder
> >>>>>
> >>>>>Hi Beth,
> >>>>>
> >>>>>Tom is spearheading the LEAD related LDM and decoders setup here. I'm
> >>>>>CC-ing him.
> >>>>>
> >>>>>Do you want to work with data streamed from the LDM or decoded files?
> >>>
> >>>I
> >>>
> >>>>>suspect you want a decoded stream, though I don't know of any decoders
> >>>>>that do that (but then by breadth of decoder knowledge is limited).
> >>>>>
> >>>>>Since I don't understand the nature of dQUOB I'm not sure which data
> >>>>>product would provide the path of least resistance. METAR data is
> >>>
> >>>simple
> >>>
> >>>>>conceptually but the decoding of the raw reports isn't trivial. You
> >>>>>could modify the metar2nc decoder (perl script) to spit out whatever
> >>>>>dQUOB wants.
> >>>>>
> >>>>>BTW, the METAR data comes over the IDS|DDPLUS feed. I'll let Tom help
> >>>>>you figure out how to modify the pqact.conf file to redirect the METAR
> >>>>>data.
> >>>>>
> >>>>>Let me know if you have any questions. There are a lot of resources at
> >>>>>Unidata we could point you to.
> >>>>>
> >>>>>Cheers,
> >>>>>Doug
> >>>>>
> >>>>>Beth A Plale wrote:
> >>>>>
> >>>>>>Hi Doug,
> >>>>>>
> >>>>>>My student, Charlie, is looking at piping data to a process, as you
> >>>>>>described it last week, where it will be decoded then used in dQUOB.
> >>>>>>We're looking for a good data product to use, and by "good" I more
> >>>
> >>>mean
> >>>
> >>>>>>that we're getting it via our LDM peer, it is relatively simple, and
> >>>>>
> >>>>>that
> >>>>>
> >>>>>>there's a decoder for it.  What would you suggest?  The data
> >>>
> >>>products
> >>>
> >>>>>>we're receiving are below.
> >>>>>>
> >>>>>>thanks!
> >>>>>>
> >>>>>>beth
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>I don't see METAR data in the queue.  I am running "ldmadmin watch",
> >>>
> >>>and
> >>>
> >>>>>>>I only see CRAFT, NNEXRAD, IDS|DDPLUS, CONDUIT, HDS.
> >>>>>>>
> >>>>>>>- Charlie
> >>>>>>
> >>>>>>
> >>>>>>>>      I think I found the decoder on the unidata web site.  They
> >>>
> >>>have
> >>>
> >>>>>netcdf
> >>>>>
> >>>>>>>>decoders for GRIB, METAR, Upperair, Synoptic, Buoy, Ascii, and
> >>>>>>>>ldmConnect.  What data do you want me to pull from the ldm queue?
> >>>
> >>>All
> >>>
> >>>>>>>>of it?
> >>>>>>>>
> >>>>>>>>- Charlie
> >>>>>>>>
> >>>>>>>>
> >>>>>
> >>>>>--
> >>>>>*---------------------------------------------------------------------
> >>>
> >>>-*
> >>>
> >>>>>| Doug Lindholm, Software Engineer          |  E-mail: address@hidden
> >>>
> >>>|
> >>>
> >>>>>| Unidata Program Center                    |   Phone: 303-497-8659
> >>>
> >>>|
> >>>
> >>>>>| UCAR Office of Programs                   |
> >>>
> >>>|
> >>>
> >>>>>| P.O. Box 3000                             |     There's no place
> >>>
> >>>|
> >>>
> >>>>>| Boulder, Colorado 80307-3000              |        like $HOME
> >>>
> >>>|
> >>>
> >>>>>*---------------------------------------------------------------------
> >>>
> >>>-*
> >>>
>
> --
> *----------------------------------------------------------------------*
> | Doug Lindholm, Software Engineer          |  E-mail: address@hidden   |
> | Unidata Program Center                    |   Phone: 303-497-8659    |
> | UCAR Office of Programs                   |                          |
> | P.O. Box 3000                             |     There's no place     |
> | Boulder, Colorado 80307-3000              |        like $HOME        |
> *----------------------------------------------------------------------*
>

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================