[LDM #CSZ-262645]: errant LDM REQUESTs on sirocco.srcc.lsu.edu

Hi Yixin,

I apologize for the silence today.  We have been meeting to discuss the 
LDM problems being experienced in SRCC...  After reviewing things like
feed latencies, and volumes of data being ingested on the SRCC NOAAPort
ingest system (mistral), we were left with the impression that there
were a number of problems.  Here is a breakdown:

1) NOAAPort ingest on mistral

   Comparisons of volumes of data ingested for each feedtype in NOAAPort
   on mistral versus other NOAAPort ingest machines shows that there is
   some problem on mistral:

   NEXRAD3 ingest volume on mistral:
   
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?NEXRAD3+mistral.srcc.lsu.edu

   NEXRAD3 ingest volume on jackie.unidata.ucar.edu:
   
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?NEXRAD3+jackie.unidata.ucar.edu

   Comparisons in these plots shows that mistral is only processing a fraction 
of
   the NEXRAD Level III products that is being sent in NOAAPort (the other 
feedtypes
   exhibit the same discrepancy).  The question is why the ingest on mistral is
   doing so poorly?

2) examination of the latency of IDS|DDPLUS data being ingested on sirocco
   and/or pavan shows that there is something slowing down the delivery of
   data from upstreams (e.g., idd.unidata.ucar.edu and/or idd.ssec.wisc.edu)
   to the downstreams (i.e., sirocco and pavan)

   What is causing this is a mystery to us.  In the past, latencies this
   high have typically been associated with some sort of imposed limitations
   on the network connections for the feeds (e.g., some sort of packet
   shaping).  The comment earlier today that LSU does not packet shaping
   implies that the problem is with something else.  Again, we don't know
   what is.

re:
> We have updated the system of Sirocco.

Very good.  I see that a new kernel was installed:

[ldm@sirocco ~]$ uname -a
Linux sirocco.srcc.lsu.edu 2.6.18-164.11.1.el5 #1 SMP Wed Jan 6 13:26:04 EST 
2010 x86_64 x86_64 x86_64 GNU/Linux

[ldm@sirocco ~]$ ls -alt /boot
total 20114
drwxr-xr-x  4 root root    1024 Jan 29 10:59 .
drwxr-xr-x  2 root root    1024 Jan 29 10:59 grub
-rw-------  1 root root 3543061 Jan 29 10:56 initrd-2.6.18-348.1.1.el5.img
-rw-r--r--  1 root root  118668 Dec 14 04:41 symvers-2.6.18-348.1.1.el5.gz
-rw-r--r--  1 root root   67861 Dec 14 04:41 config-2.6.18-348.1.1.el5
-rw-r--r--  1 root root 1282892 Dec 14 04:41 System.map-2.6.18-348.1.1.el5
-rw-r--r--  1 root root 2126108 Dec 14 04:41 vmlinuz-2.6.18-348.1.1.el5
-rw-r--r--  1 root root     162 Dec 14 04:41 .vmlinuz-2.6.18-348.1.1.el5.hmac
 ...

[ldm@sirocco ~]$ uptime
 14:59:18 up 322 days, 59 min,  6 users,  load average: 0.01, 0.02, 0.00

The update on sirocco will require a reboot to become completely active.
Since rebooting before shutting down the LDM could result in a damaged
LDM queue, we recommend that you first stop the LDM as follows:

<on sirocco as 'ldm'>
ldmadmin stop

<on sirocco as 'root'>
reboot   (or shutdown, whichever you prefer)

After the machine is back up after the reboot, you will need to login
as 'ldm' and start the LDM:

<as 'ldm' on sirocco after a reboot>
ldmadmin start

re:
> I wonder whether you could help us to make it download data soon.

Let's see what happens with a system reboot!

Please let us know as soon as sirocco has been rebooted.  Thanks!

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CSZ-262645
Department: Support LDM
Priority: Normal
Status: Closed