[Support #VXV-940104]: Re: [conduit] missing and incomplete GFS grids on CONDUIT feed

Hi David,

re:
> The queues are 12gig on Freshair1, and 47gig on freshair2

Given the relatively small size of the LDM queue on freshair1
(12 GB), I would be concerned that the data being ingested does
not all get processed out of the queue before it is deleted to
make space for newly received products.

Here is a snapshot for the volume of data being received on
freshair1:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_summary_volume?freshair1.atmos.washington.edu

Data Volume Summary for freshair1.atmos.washington.edu

Maximum hourly volume  77304.569 M bytes/hour
Average hourly volume  50940.267 M bytes/hour

Average products per hour     424795 prods/hour

Feed                           Average             Maximum     Products
                     (M byte/hour)            (M byte/hour)   number/hour
DIFAX                 14940.632    [ 29.330%]    19698.170     6558.867
CONDUIT               11356.061    [ 22.293%]    33512.944   104044.800
NEXRAD2                8342.439    [ 16.377%]    10551.661    93665.244
NIMAGE                 6660.848    [ 13.076%]     9208.322     5947.067
FNMOC                  3314.296    [  6.506%]    10420.792     8907.289
NEXRAD3                2579.811    [  5.064%]     3181.140   115404.022
NGRID                  1945.366    [  3.819%]     2671.519     1465.778
HDS                    1335.876    [  2.622%]     1761.119    41595.244
GEM                     174.053    [  0.342%]     1354.518     1014.333
FNEXRAD                 112.879    [  0.222%]      128.202      103.356
UNIWISC                  96.327    [  0.189%]      145.256       50.311
IDS|DDPLUS               69.188    [  0.136%]       84.413    45416.178
EXP                      11.971    [  0.023%]       14.977      563.400
LIGHTNING                 0.521    [  0.001%]        1.642       58.644

The average queue residency time would be:

12/50 * 60 ~= 14.4 minutes

The residency time at volume peak, on the other hand, would be more 
like:

12/77 * 60 ~= 7.8 minutes

The residency time in the queue will have an effect in two different ways:

- products will need to get processed out of the queue before they
  are deleted to make room for newly arriving products

- duplicate products received will not be rejected since the
  duplicate product detection and rejection requires that a
  product still be in the queue when the same product is received
  the 2nd (or 3rd, etc.) time

  It has always been our recommendation to size one's LDM queue to
  be large enough to hold 1 hour of received data.

re:
> The request lines are the same:
> request CONDUIT         "[09]$" idd.unidata.ucar.edu
> request CONDUIT         "[18]$" idd.unidata.ucar.edu
> request CONDUIT         "[27]$" idd.unidata.ucar.edu
> request CONDUIT         "[36]$" idd.unidata.ucar.edu
> request CONDUIT         "[45]$" idd.unidata.ucar.edu

Hmm... this is very weird indeed.

Questions:

- are freshair1 and freshair2 in different machine rooms?

- is the path from freshair1 to idd.unidata.ucar.edu different than
  the path for freshair2 to idd.unidata.ucar.edu?

- is there something else running on freshair2 that might be causing
  the increased latencies

  We have seen instances where an Gbps Etherent interface is actually
  running at 100 Mbps.  This will have a profoundly bad effect on the
  receipt of data via the IDD.

  You can check to see what speed the relevant Ethernet interface
  is running in two different ways, one of which needs to be run
  as 'root':

  dmesg | grep -i eth

  This will say how the Ethernet interface was configured at boot.

  ethtool <interface>

  This needs to be run as 'root', but it will say what the link speed
  is when run.

- what are the loads on both freshair1 and freshair2?

  I.e., is there a possibility that the latency being experienced on
  freshair2 is somehow related to system loading?  (This should not
  be likely, but...)

- presumably, the CONDUIT products that David O was referring to in
  his original post to the address@hidden list were processed
  on freshair2

  Is this, in fact, the case?

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: VXV-940104
Department: Support CONDUIT
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.