[Support #SBB-325304]: Re: 20110712: CONDUIT request -- fire weather grids

Hi Justin,

Preliminary comments on gribinsert test running on 140.90.33.32:

A comparison of the volume stats for the test CONDUIT datastream
received on gale.unidata.ucar.edu and the "operational" CONDUIT
datastream as seen by our toplevel IDD relay idd.unidata.ucar.edu
shows what might be a problem.

Compare:

CONDUIT volume on gale.unidata.ucar.edu
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?CONDUIT+gale.unidata.ucar.edu

CONDUIT volume on idd.unidata.ucar.edu
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?CONDUIT+idd.unidata.ucar.edu

The volume for the 22+ hours (as I write this note) on gale.unidata.ucar.edu
is significantly less than for the same 22+ hour time period on 
idd.unidata.ucar.edu.
I expected that the volume on gale.unidata.ucar.edu would be measurably 
_greater_ than
on idd.unidata.ucar.edu because you indicated that the fireweather products 
were being
inserted in addition to the products being sent in the "operational" CONDUIT 
datastream.
If this is not the case, my surmise is completely invalid.

If there are no other problems, this may mean that the insertion of the 
fireweather
products into the LDM queue on 140.90.33.32 is interfering with the other 
products
being inserted into the queue.  The first thing that jumps to mind as being a
potential problem is that the LDM queue on 140.90.33.32 is not large enough for
both sets of products to be queue-resident long enough to be delivered to the
downstream requester (gale.unidata.ucar.edu).  Since gale.unidata.ucar.edu is
only ingesting the test CONDUIT datastream, and since it is a high-end 
workstation
(dual quad core CPUs w/24 GB RAM) that is essentially idling, and since the
network connection on gale.unidata.ucar.edu is 1 Gps through Internet2, I am not
hopeful that the problem could be on it.

Please check the age of the oldest product in the LDM queue on 140.90.33.32
to see if it ever becomes exceedingly small.  This check should be routinely
performed (like every minute via a cron job) so that we can see if the addition
of the fireweather products has exceeded the 6 GB queue size.

Comments:

- way back when I was working with Patrick O'Reilly on CONDUIT issues, I 
strongly
  recommended that routine monitoring of things like the age of the oldest
  product in the LDM queue be made.  I don't recall if Patrick ever setup this
  monitoring, so I can't say if you are already doing what needs to be done
  (which is running the 'ldmadmin addmetrics' action from a cron job running
  in the account where the LDM is running on 140.90.33.32).  If you are
  already running the metrics gathering actions from cron, you can plot
  the time history of things like age of the oldest product in the queue
  by running 'ldmadmin plotmetrics' (the ability to plot the metrics is
  based on the availability of gnuplot).

- I will be merging additions from the GRIB tables Becky provided yesterday
  to the GEMPAK tables being used in my gribinsert v1.4 setup here at Unidata
  today.  After the merge (which looks a little messier than I expected), I
  will see if all of the fields available in the firewx.hires products I FTPed
  from the NCEP FTP server can be identified/cracked/harvested for metadata.
  More later on this...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: SBB-325304
Department: Support CONDUIT
Priority: Normal
Status: Closed