[Support #TOL-308327]: statistics

Hi Yoshihiro,


re: the size of your LDM queue
> As I undestand is set in the ldmadmin and is : $pq_size =
> "400M"

This is most likely the cause of the problem you are reporting.
The volume of GFS data you are requesting from the CONDUIT stream
routinely approaches or exceeds 1 GB per hour:

http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?CONDUIT+atm77.fis.ua.pt

After splitting the ~ldm/etc/ldmd.conf request lines, you are getting
the data much faster than you were previously, and your machine is not
able to process the data out of the LDM queue before it gets overwritten
by new data being received.  Just so you know, this is a common situation
that is easily solved by increasing the size of your LDM queue.  We
routinely recommend that users getting the CONDUIT datafeed setup
their ~ldm/etc/ldmadmin-pl.conf file for a 2 GB queue.

Here is what I recommend:

<as 'ldm'>
cd ~ldm/etc
-- edit ldmadmin-pl.conf and change the $pq_size variable from
   400M to 2G
cd
ldmadmin stop
ldmadmin delqueue
ldmadmin mkqueue -f
ldmadmin start

Increasing your LDM queue to 2 Gigabytes should allow you to receive
all data within an hour and process it out before any gets overwritten. 
The only way that this would not work would be if the of data processing
you are attempting is taking a very long time.  I do not think that this
is the case since you were not exepiencing the problem when the
data was coming in much slower.

> Last night I included the idd.unidata.ucar.edu computer as
> ALTERNATE and I finally received all files, with correct
> size. However, the latency increased --> anyway , that was
> a good news, although with not so good results.

The increase in latency being beneficial in processing all
of the data out of the queue supports my contention that your
queue is much too small to do the processing you want.

> So, I decided to go further on the problem :
> 
> Looking at my logs I noticed that it was an error in my
> request when I was doing it only with idd.cise-nsf.gov
> (therefore with no ALTERNATE).
> 
> There was a mistype in the ldmd.conf and the message was
> --> Couldn't get IP address of host ide.cise-nsf.gov
> (MISTYPED: "ide" ).

> Actually the error was in one of the five partioned
> request, the -->  Request CONDUIT
> "MT.gfs_CY.(00|06|12|18).*[16]$" ide.cise-nsf.gov
> PRIMARY

Ah Ha!  This would cause one of the request lines to not get
any data.  In the case of the feed split 5 ways, this would
cause 20% of the data to not be requested.

> Other 4 request was correct. I suspect that this error was
> causing the file size reception problem as I already
> described.

Yes, 20% of the data would not be requested/received.  This
does not, however, change my recommendation to increase your
LDM queue size.

> Now, I returned back without including the ALTERNATE
> source to see the results. I hope this solve the size and
> latency problem.

Please increase your LDM queue size.  The performance on your
machine will not be degraded, and you will provide yourself with
a buffer that gives your machine more time to process the incoming
data.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TOL-308327
Department: Support IDD
Priority: Emergency
Status: Closed