[Support #TOL-308327]: statistics

Hi,

re:
> Fist of all I would like to thank you for your help in
> trying to solve my problem. 
No worries.

> I made the changes as suggested, spliting the request as
> by your e-mail. The results was good if you look at the
> statistics (laency bellow 500).

This was our objective.

> However there is now a new
> problem because no file are reveived with the size it
> should have ( comparing with the ncep's computer data file
> size). The size is almost ok since as an exemple, instead
> of the order of 27048228 bytes I am receiving around
> 21873520 --> all files with about the same file size are
> being received (but with no correct file size).

Please remember that each product in the CONDUIT datastream
is a single GRIB/GRIB2 message, and none of these messages will
be larger than a few hundred thousand kilobytes.  The size
you note above represents the sum of all GRIB/GRIB2 messages
produced for a model run.  Also, comparing the total size
of products received with the size published by NCEP can differ
since the LDM injection process will eliminate duplicate products.
We regularly see one or more duplications of a GRIB/GRIB2 message
in the model output file.  The LDM's duplicate product detection
and rejection mechanism will send the first instance of the product,
but not the duplicates.

In order to convince yourself that you are receiving all of the
data that is being sent in the IDD feed, you should compare what
you received with the list of files that were sent.  The list of
files sent is contained in a catalog file that is added at the
end of the stream of GRIB/GRIB2 messages.  You can learn more about
the catalog file at:

http://www.unidata.ucar.edu/data/conduit/ldm_idd/index.html

> Just to remember you, if I make one request instead of
> several requests I was receiving almost 80 % of data with
> correct size; but with high latency.

I would suspect that if you really are not seeing all the data,
your pqact.conf action(s) are not writing all messages in the way
you want.  If they are not, then the size of your output file will
differ from what the model creates.  Also, if the model output file
contains many duplicate fields, the volume of the data you receive
will be less than what was originally available ** but you will have
received everything so this should not be a problem **.

> Today I tested: seting computer clock to UTC but the
> statistics shows negative values. So that I returned back
> to  local time ( UTC + 1 hour ).

I would run your computer on your local time.  The conversion
to UTC will be automatic if your machine's clock is correct.

> Also, I started with -m option to see if makes any
> difference.

I don't think that you are not receiving products that
are available.  However, to make sure, please send us the
following:

- the size of your LDM queue
- your ~ldm/etc/ldmd.conf file
- your ~ldm/etc/pqact.conf file

> Is there any other test I could make for try ?

The statistics indicate that you should be receiving all of the
data.  Since you are now getting the data faster than you were
previously, it is possible that your LDM queue size is not large
enough to hold the data long enough to process it before it gets
overwritten by new data being received.  Letting us know the size
of your LDM queue will go a long way towards identifying the
possible problem you are seeing.  My instinct tells me that the
most likely things are:

- your LDM queue is too small to allow processing of the products
  before getting overwritten by new ones received

- the removal of duplicate products is making it look you are receiving
  less data than you should

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TOL-308327
Department: Support IDD
Priority: Emergency
Status: Closed