Re: 20040220: metar2nc

On Fri, 20 Feb 2004, Unidata Support wrote:

>
> ------- Forwarded Message
>
> >To: address@hidden
> >From: Angel Li <address@hidden>
> >Subject: metar2nc
> >Organization: University of Miami/RSMAS
> >Keywords: 200402202002.i1KK20rV004136
>
> Hi,
>
> I'm using metar2nc to file away METAR data and I'm using another perl
> script to access the data from web pages. My perl script scans each
> netCDF metar file from latest to oldest looking for a station report and
> I've noticed that there is quite a lag between a report coming in and my
> script finding it. metar2nc has a 20 minute timeout and I'm wondering if
> that's too long. I'm guessing the netCDF file is not available for read

Angel,

The metar2nc decoders sync's all open netCDF files everytime a new file is
opened. It doesn't matter if the decoder has the file open, onced synced
it up to the disk s/w to get the data to the disk file. The problem with
the decoder most disk  i/o s/w waits for the buffer to be filled before
writing and the metar2nc writes are relatively small. Thus the latency
that you are noticing.

> while the decoder has it opened for write?? Also, in my pqact.conf I
> invoke the decoder with an action line that looks like "PIPE metar2nc
> ...." I changed it to "PIPE -close metar2nc ..." and after restarting
> ldm the load average went really high with a lot of metar2nc processes
> running. Is that expected?

Yes, by add the -close flag it causes a new pipe to be created for every
surface bulletin in the data, similar to adding about 4000 pipes and file
closes per hour. I suggest you don't do it.


Good news, I have revamped the metar2nc decoder. There is only one record
per station in the file, all duplicate have been eliminated and most
importantly the netCDF is synced everytime the decoder writes to another
netCDF file. The latency should only be disk buffer writes now. I haven't
release the code because it's being tested. I would appreciate if you
would install it and send me your comments. It's in the ~ftp/pub/decoders
directory called metar2nc.new*   metar.cdl.new  One can use the same pqact
entry, just append ".new" to decoder and cdl file.

Another option is the THREDDS MetarServer, it's much faster because the
data is stored in ascii data files as it arrives. The server also has a
web interface that might be interesting to you.  THe decoding is
not performed until requested. It's been operational for about a year now.


http://motherlode.ucar.edu:8080/thredds/cgi-bin/MetarServer.pl?format=html&stn=KDEN&time=latest

For the list of station IDs:

http://motherlode.ucar.edu:8080/thredds/cgi-bin/MetarServer.pl?format=qc

Today we are doing maintenance so the main server is not working but the
backup one is working. To use the backup, replace thredds with thredds.old
in the above urls.

There's more info here about MetarServer and other THREDDS servers.

http://www.unidata.ucar.edu/projects/THREDDS/tech/threddsData.html

Also this is open for comments



Robb...


>
> Thanks,
>
> Angel
>
> --
> NOTE: All email exchanges with Unidata User Support are recorded in the
> Unidata inquiry tracking system and then made publically available
> through the web.  If you do not want to have your interactions made
> available in this way, you must let us know in each email you send to us.
>
> ------- End of Forwarded Message
>

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================