[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: ldm decoder
- Subject: Re: ldm decoder
- Date: Tue, 20 Apr 2004 13:10:26 -0600 (MDT)
On Tue, 20 Apr 2004, Charles Moad wrote:
> Hi,
> We are still trying to figure out a solution for streaming the metars.
> Since I have had the metar2nc decoder running for a while now, I see
> it is saving a netcdf file for each hour in the day. Each appears to
> have 4-5 thousand metars. Is this correct? Also, at what point are the
Charlie,
There are about 3500 metar stations that send 1-3 regular reports / hour.
Also, the stations can send extra reports when the there is active
weather, these are tagged as SPECI reports. So 3-5k reports / hour is
correct.
> netcdf files closed? Do they get closed at the end of the hour they
> reflect?
metar2nc usually runs one process at a time per month and it usually has
more than one netcdf files open at once, depending on data arrivals.
The netcdf files are synched when another file is used, they are
close if there hasn't been any activity on the file for 20 minutes. The
reports can arrive as much as a month late, but the metar2nc only allows
for a couple days span, old data isn't worth much. Usually metar2nc keeps
1-5 files open because some of the data arrives hours late. Also, the
decoder can reopen a file to add more data. The data is packed into the
file as it arrives, I'm working on a netcdf file that will have one record
per station so all the data for one station can be accessed at once
instead of scanning the whole file.
Robb....
The modification dates on the files say otherwise.
>
> Thanks,
> Charlie
>
> Robb Kambic wrote:
> > On Wed, 7 Apr 2004, Doug Lindholm wrote:
> >
> >
> >>Hi Charlie,
> >>
> >>I'm curious about your comment on "turning these netcdf files into a
> >>stream". The NetCDF file format is designed to allow convenient data
> >>access via a programming API (e.g. Java, perl, C). Although I am not the
> >>authority, I don't think NetCDF is well suited for streaming. I think
> >>what you want to do (and maybe this is your plan) is modify the metar2nc
> >>decoder to spit out a stream of simplified data instead of creating the
> >>NetCDF file. Robb would be the best one to help you there.
> >
> >
> > Charlie,
> >
> > Doug is correct in stating that netcdf is a local file storage format
> > that provides quick access to the data using a programming API. metar2nc
> > is a stream reader ( NOAAPORT stream ) used to create a netcdf file. More
> > comments below.
> >
> >
> >
> >
> >>Good luck,
> >>Doug
> >>
> >>Charles Moad wrote:
> >>
> >>> I get identical output as well for all the mentioned feedtypes. They
> >>>all say IDS|DDPLUS.
> >>> You were right about the missing tab on the first line, so the syntax
> >>>passes now. I am also seeing the netcdf files being created so all
> >>>seems to be working well in that aspect.
> >>> The challenge we face now is turning these netcdf files into a stream.
> >>>The metar2nc script is not exactly a trivial, so any input as to how it
> >>>works might help. We are currently using a c++ api to register a dquob
> >>>stream source, but I think we are going to try to wrap this api in perl
> >>>and modify the metar2nc script.
> >
> >
> > I don't know what you mean by "register a dquob stream source"?
> > When I understand your requirements then maybe a solution. In a
> > nutshell, the metar data arrives in bulletins that contain multiple reports
> > that are usually sent multiple times from different distribution stations.
> > There are 5-25% of the reports that are mal-formed so they need to be
> > disgarded. Most of the surface data falls into this category. Also the
> > other surface reports are cryptic too, some even arrive in multiple
> > sections that need to be combined to make the final report.
> >
> > Robb...
> >
> >
> >>> Thanks for you help so far.
> >>>
> >>>- Charlie
> >>>
> >>>On Tue, 2004-04-06 at 17:02, Tom Baltzer wrote:
> >>>
> >>>
> >>>>Hi Charlie,
> >>>>
> >>>>Well that's very interesting. My understanding is that DDPLUS is just a
> >>>>combination of DDS and PPS (Public Products Service) so if you're able to
> >>>>get DDPLUS I'd have thought you could get DDS...
> >>>>
> >>>>I just spoke with a Jeff Weber (a local expert) who tells me that
> >>>>IDS|DDPLUS
> >>>>is a single feed composed of DDS, PPS and IDS and you cannot separate
> >>>>these
> >>>>by feed type. And in fact when I run the two ldmadmin watch -f commands
> >>>>you
> >>>>show (in two separate windows) I get identical results. I even went so far
> >>>>as to try each (PPS, DDS, PPS and DDPLUS) individually in 4 windows and
> >>>>got
> >>>>identical results. What result do you get when you run the ldmadmin watch
> >>>>with the IDS|DDS feedtype?
> >>>>
> >>>>As far as the error message you're getting from checking the pqact.conf
> >>>>file, I suspect that there is yet another TAB required. Check that the
> >>>>white space between IDS|DDPLUS and ^S in your pqact.conf file is in fact a
> >>>>tab rather than a space (sorry I didn't clarify this earlier).
> >>>>
> >>>>Let me know how it goes and/or if I can provide any other help.
> >>>>
> >>>>Best regards,
> >>>>Tom.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>-----Original Message-----
> >>>>>From: Charles W Moad [mailto:address@hidden]
> >>>>>Sent: Tuesday, April 06, 2004 1:23 PM
> >>>>>To: Tom Baltzer
> >>>>>Cc: 'Beth A Plale'; 'Charles W Moad'; 'Tom Baltzer'; 'Doug Lindholm'
> >>>>>Subject: RE: ldm decoder
> >>>>>
> >>>>>As a first step I am trying run the standard pqact | metar2nc and save
> >>>>>the data files. When I run "ldmadmin pqactcheck" I get the following
> >>>>>error:
> >>>>>
> >>>>>---
> >>>>>Apr 06 19:20:17 pqact[26196]: feedtype error at line 121: unknown feed
> >>>>>name in feedtype expression: "IDS|DDPLUS ^S(A....|P....|XUS8.) ....
> >>>>>([0-3][0-9])"
> >>>>>---
> >>>>>
> >>>>>I don't think we get the DDS|IDS feed because it does not show up as a
> >>>>>valid feedtype for "ldmadmin watch -f "DDS\|IDS"", however "ldmadmin
> >>>>>watch -f "IDS\|DDPLUS"" does work.
> >>>>>
> >>>>>Any help would be greatly appreciated.
> >>>>>
> >>>>>Thanks,
> >>>>> Charlie
> >>>>>
> >>>>>On Tue, 2004-04-06 at 12:43, Tom Baltzer wrote:
> >>>>>
> >>>>>
> >>>>>>Hi Beth and Charlie,
> >>>>>>
> >>>>>>It sounds like you have found the Unidata netCDF decoder package. In my
> >>>>>>short time around here I've discovered that there are more decoders than
> >>>>>>just those found in that package and have been working on documenting
> >>>>>
> >>>>>those
> >>>>>
> >>>>>
> >>>>>>that I see used by LDM systems around Unidata here:
> >>>>>>http://my.unidata.ucar.edu/content/projects/LEAD/Decoders.htm
> >>>>>>
> >>>>>>The pqact.conf configuration lines for pulling out metar data and
> >>>>>
> >>>>>placing it
> >>>>>
> >>>>>
> >>>>>>in netcdf files in the ~ldm/data/pub/decoded/netcdf/surface directory
> >>>>>
> >>>>>are
> >>>>>
> >>>>>
> >>>>>>(note the first 5 lines are comments):
> >>>>>>
> >>>>>>#
> >>>>>># metar decoder, creates NetCDF files formatted
> >>>>>># data/decoded/netcdf/surface/yymmddhh_metar.nc
> >>>>>>#
> >>>>>># All aviation reports including metar tests
> >>>>>>DDS|IDS ^S(A....|P....|XUS8.) .... ([0-3][0-9])
> >>>>>> PIPE decoders/metar2nc
> >>>>>> etc/metar.cdl
> >>>>>> data/pub/decoded/netcdf/surface
> >>>>>> (\2:yy)(\2:mm)
> >>>>>>
> >>>>>>Be advised that those are <tab>s at the beginning of lines 2-5 (very
> >>>>>>important!). Also be advised that it appears the decoder expects the
> >>>>>>directory structure to exist and be writable before it will work. I've
> >>>>>>submitted this to the decoder folks as a fix.
> >>>>>>
> >>>>>>As Doug mentioned the pqact application will stream the raw data reports
> >>>>>
> >>>>>to
> >>>>>
> >>>>>
> >>>>>>stdin and the metar2nc perl script then takes that streaming data and
> >>>>>
> >>>>>writes
> >>>>>
> >>>>>
> >>>>>>it to a netCDF file. So if you wished to you could take the metar2nc
> >>>>>
> >>>>>script
> >>>>>
> >>>>>
> >>>>>>and modify it to act as a dQUOB interface. Or if dQUOB takes stdin then
> >>>>>
> >>>>>you
> >>>>>
> >>>>>
> >>>>>>could use it to help you parse the incoming metar data. My understanding
> >>>>>
> >>>>>is
> >>>>>
> >>>>>
> >>>>>>that you all have been trying to access the LDM product queue to do your
> >>>>>>dQUOB work. It seems that using pqact/decoder type functions would be
> >>>>>
> >>>>>much
> >>>>>
> >>>>>
> >>>>>>easier and effectively the same.
> >>>>>>
> >>>>>>Well I hope this helps. Feel free to contact me directly with questions
> >>>>>
> >>>>>or
> >>>>>
> >>>>>
> >>>>>>comments.
> >>>>>>
> >>>>>>Cheers,
> >>>>>>Tom.
> >>>>>>
> >>>>>>Tom Baltzer
> >>>>>>Software Engineer
> >>>>>>UCAR Office of Programs
> >>>>>>Unidata Program Center
> >>>>>>P.O. Box 3000
> >>>>>>Boulder, CO 80307-3000
> >>>>>>address@hidden
> >>>>>>303-497-8637
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>-----Original Message-----
> >>>>>>>From: Doug Lindholm [mailto:address@hidden]
> >>>>>>>Sent: Friday, April 02, 2004 11:52 AM
> >>>>>>>To: Beth A Plale
> >>>>>>>Cc: Charles W Moad; Tom Baltzer
> >>>>>>>Subject: Re: ldm decoder
> >>>>>>>
> >>>>>>>Hi Beth,
> >>>>>>>
> >>>>>>>Tom is spearheading the LEAD related LDM and decoders setup here. I'm
> >>>>>>>CC-ing him.
> >>>>>>>
> >>>>>>>Do you want to work with data streamed from the LDM or decoded files?
> >>>>>
> >>>>>I
> >>>>>
> >>>>>
> >>>>>>>suspect you want a decoded stream, though I don't know of any decoders
> >>>>>>>that do that (but then by breadth of decoder knowledge is limited).
> >>>>>>>
> >>>>>>>Since I don't understand the nature of dQUOB I'm not sure which data
> >>>>>>>product would provide the path of least resistance. METAR data is
> >>>>>
> >>>>>simple
> >>>>>
> >>>>>
> >>>>>>>conceptually but the decoding of the raw reports isn't trivial. You
> >>>>>>>could modify the metar2nc decoder (perl script) to spit out whatever
> >>>>>>>dQUOB wants.
> >>>>>>>
> >>>>>>>BTW, the METAR data comes over the IDS|DDPLUS feed. I'll let Tom help
> >>>>>>>you figure out how to modify the pqact.conf file to redirect the METAR
> >>>>>>>data.
> >>>>>>>
> >>>>>>>Let me know if you have any questions. There are a lot of resources at
> >>>>>>>Unidata we could point you to.
> >>>>>>>
> >>>>>>>Cheers,
> >>>>>>>Doug
> >>>>>>>
> >>>>>>>Beth A Plale wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>>Hi Doug,
> >>>>>>>>
> >>>>>>>>My student, Charlie, is looking at piping data to a process, as you
> >>>>>>>>described it last week, where it will be decoded then used in dQUOB.
> >>>>>>>>We're looking for a good data product to use, and by "good" I more
> >>>>>
> >>>>>mean
> >>>>>
> >>>>>
> >>>>>>>>that we're getting it via our LDM peer, it is relatively simple, and
> >>>>>>>
> >>>>>>>that
> >>>>>>>
> >>>>>>>
> >>>>>>>>there's a decoder for it. What would you suggest? The data
> >>>>>
> >>>>>products
> >>>>>
> >>>>>
> >>>>>>>>we're receiving are below.
> >>>>>>>>
> >>>>>>>>thanks!
> >>>>>>>>
> >>>>>>>>beth
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>I don't see METAR data in the queue. I am running "ldmadmin watch",
> >>>>>
> >>>>>and
> >>>>>
> >>>>>
> >>>>>>>>>I only see CRAFT, NNEXRAD, IDS|DDPLUS, CONDUIT, HDS.
> >>>>>>>>>
> >>>>>>>>>- Charlie
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> I think I found the decoder on the unidata web site. They
> >>>>>
> >>>>>have
> >>>>>
> >>>>>
> >>>>>>>netcdf
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>decoders for GRIB, METAR, Upperair, Synoptic, Buoy, Ascii, and
> >>>>>>>>>>ldmConnect. What data do you want me to pull from the ldm queue?
> >>>>>
> >>>>>All
> >>>>>
> >>>>>
> >>>>>>>>>>of it?
> >>>>>>>>>>
> >>>>>>>>>>- Charlie
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>>--
> >>>>>>>*---------------------------------------------------------------------
> >>>>>
> >>>>>-*
> >>>>>
> >>>>>
> >>>>>>>| Doug Lindholm, Software Engineer | E-mail: address@hidden
> >>>>>
> >>>>>|
> >>>>>
> >>>>>
> >>>>>>>| Unidata Program Center | Phone: 303-497-8659
> >>>>>
> >>>>>|
> >>>>>
> >>>>>
> >>>>>>>| UCAR Office of Programs |
> >>>>>
> >>>>>|
> >>>>>
> >>>>>
> >>>>>>>| P.O. Box 3000 | There's no place
> >>>>>
> >>>>>|
> >>>>>
> >>>>>
> >>>>>>>| Boulder, Colorado 80307-3000 | like $HOME
> >>>>>
> >>>>>|
> >>>>>
> >>>>>
> >>>>>>>*---------------------------------------------------------------------
> >>>>>
> >>>>>-*
> >>>>>
> >>
> >>--
> >>*----------------------------------------------------------------------*
> >>| Doug Lindholm, Software Engineer | E-mail: address@hidden |
> >>| Unidata Program Center | Phone: 303-497-8659 |
> >>| UCAR Office of Programs | |
> >>| P.O. Box 3000 | There's no place |
> >>| Boulder, Colorado 80307-3000 | like $HOME |
> >>*----------------------------------------------------------------------*
> >>
> >
> >
> > ===============================================================================
> > Robb Kambic Unidata Program Center
> > Software Engineer III Univ. Corp for Atmospheric
> > Research
> > address@hidden WWW: http://www.unidata.ucar.edu/
> > ===============================================================================
>
===============================================================================
Robb Kambic Unidata Program Center
Software Engineer III Univ. Corp for Atmospheric Research
address@hidden WWW: http://www.unidata.ucar.edu/
===============================================================================