[Support #TOL-308327]: statistics

Hi,

re:
> I just started the pp42.fis.ua.pt computer to  request
> IDS|DDPLUS

Thanks.  We see that you are also requesting CONDUIT and
HDS on this same machine.  This is exactly the test setup we
wanted.

> In this computer, which I installed the new version o ldm,
> I had to set the clock as local time and not in UTC ( my
> lag is one hour during summer season and 0 hour during
> winter ). If I set to UTC the statistics produces negative
> values!

All that the LDM cares about is the indicated UTC time be
accurate.

> Regarding the conduit data, the reception is almost
> fine,(as you can see in the atm77.fis.ua.pt statistics)
> except that after about 30 minutes of reception, seems to
> to have something disturbing the transmission during at
> least 15 minuts (It happens on  00, 06, 12 and 18 gfs
> forecasts which are in the ldm conduit aroung 04, 10, 16,
> 22 clock time).

I interpret the CONDUIT latencies you are seeing on both atm77
and pt42 to mean that the volume of CONDUIT data you are requesting
is too much for a single ldmd.conf 'request'.  My view is supported
by the fact that the latencies for both the IDS|DDPLUS and HDS
datastreams have near zero latencies:

IDS|DDPLUS latency on pp42:
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+pp42.fis.ua.pt

HDS latency on pp42:
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?HDS+pp42.fis.ua.pt

NOTE: there are traces of high latencies indicated for both of these feeds.
Those traces are almost entirely due to the redundant feed request to
solon.meteoro.ufrj.br.  This indicates two things:

- the feed from solon to pp42 is very slow

- the LDM product queue on pp42 is not large enough to allow for duplicate
  product detection and rejection.  What is happening is that the data
  being received on pp42 is causing old products to be scoured out of
  the queue _before_ the same product is received from solon.  Since
  duplicate product detection/rejection requires that the first instance
  of the product be in the queue, and since you are receiving enough data
  to force scouring of 'old' products out of your queue, some products
  coming from solon are not being rejected even though they were previously
  received from idd.cise-nsf.gov.  You can convince yourself that this is
  a small number of products by looking at the volume plots for IDS|DDPLUS
  and HDS data:

IDS|DDPLUS volume on pp42:
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?IDS|DDPLUS+pp42.fis.ua.pt

HDS volume on pp42:
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+pp42.fis.ua.pt

Back to CONDUIT ingestion...  Your CONDUIT latencies would decrease 
dramatically if
you took your single ldmd.conf 'request' line:

request CONDUIT "MT.gfs_CY.00*|MT.gfs_CY.12*" idd.cise-nsf.gov PRIMARY

(or whatever it is currently)and split it into 5 request lines:

request CONDUIT "MT.gfs_CY.(00|12).*[05]$" idd.cise-nsf.gov PRIMARY
request CONDUIT "MT.gfs_CY.(00|12).*[16]$" idd.cise-nsf.gov PRIMARY
request CONDUIT "MT.gfs_CY.(00|12).*[27]$" idd.cise-nsf.gov PRIMARY
request CONDUIT "MT.gfs_CY.(00|12).*[38]$" idd.cise-nsf.gov PRIMARY
request CONDUIT "MT.gfs_CY.(00|12).*[49]$" idd.cise-nsf.gov PRIMARY

These patterns select mutually exclusive subsets of the CONDUIT GFS data,
so they effectively split the volume in fifths.

Please consider reworking your CONDUIT data requests using the above
examples for ideas.  I am convinced that the latencies you see
for CONDUIT products could go from upwards of 3000 seconds down to
well under a minute.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TOL-308327
Department: Support IDD
Priority: Emergency
Status: Closed