[CONDUIT #IJW-881590]: NDFD Data Issues

Hi Nathan,

re:
> While we are only making requests from a single server now, this did not 
> solve the
> problem and we continue to get partial forecast files.

The question in our minds is if you are receiving all of the NDFD products
that NOAA is putting into CONDUIT.  It is clear that NOAA (NCEP) is not putting
all of the possible NDFD products into CONDUIT, and I believe that this was
be design when NDFD was first added to CONDUIT.

re:
> In terms of the manifest files,
> we are indeed requesting them and filing them.

OK.  Are you comparing the NDFD products with the manifest files entries
that show which products were successfully put into the top level
LDM queue where the products are first made available?  If yes, what
do your comparisons show?

re:
> 1. How can we report back real-time statistics?

Real-time statistics are reported back to us when both of the following
are met:

- there is an uncommented EXEC line that runs 'rtstats' in the LDM
  configuration file, ~ldm/etc/ldmd.conf

  This line will look like:

#
# Real time statistics
EXEC   "rtstats -h rtstats.unidata.ucar.edu"

  If this line is commented, 'rtstats' will not be run as part of the LDM
  program group, so no stats will be sent back to us.

- the local/site firewall(s) allow outbound traffic to port 388 on the
  machine rtstats.unidata.ucar.edu

re:
> 2. Since we are only requesting the manifest files, and these seem to be 
> incomplete,
> does this mean there could be an issue with what is being inserted by NOAA?

Question:

- you are "only REQUESTing the manifest files"?

  Surely you are also REQUESTing the products themselves, true?

re:
> 3. When we first wanted NDFD grids, we were pointed to the manifest files -

  I don't remember mentioning the manifest files until our last exchange.
  Did you keep the email in which you were pointed to the manifest files?

  Again, the manifest files that I am talking about are created by the
  process that puts the NDFD products into the LDM queue at NCEP.  The
  file entries will show if a product (grib2 message) is successfully
  inserted into the LDM queue or not.  The insertion would fail if
  a product with the exact same MD5 signature is already in the queue.
  Since this situation does not happen much, if ever, anymore, all of
  the entries in the manifest files should reflect products successfully
  inserted into the LDM queue at NCEP.  If there is no loss in the
  path from NCEP to the receiving LDM (meaning you), then you should
  receive all of the products inserted.  The only way that there could
  be a product not delivered is when the receipt latency is greater
  than the product(s) residency time(s) in any of the LDMs in the
  delivery path.  If we were receiving real-time stats from you, we
  could tell if this situation was happening anywhere along the delivery
  path to you.  Without real-time stats, the way for an end-user
  to tell what latencies they are seeing is by running the 'ldmadmin watch'
  for the datastream in question (e.g., 'ldmadmin watch -f CONDUIT')
  and then to compare the local time shown in the listing with the
  product creation time also shown in the listing.

re:
> Thanks for your help.

No worries.

By the way, I am operating under the (possibly bad?) assumption that
you are, in fact, getting every NDFD product that is being sent in
CONDUIT, but the list of the products being sent is not what you
are expecting/wanting.  If, however, your receipt latencies are high,
it could well be the case that you are actually not receiving all
of the products being sent.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: IJW-881590
Department: Support CONDUIT
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.