Re: 19991201: ldm losing large files

On Wed, 1 Dec 1999, Unidata Support wrote:

> >To: address@hidden
> >From: Jeff Masters <address@hidden>
> >Subject: ldm losing large files
> >Organization: .
> >Keywords: 199912011922.MAA04768
> 
> 
>   Hi, I am using ldmsend to transfer large files between ldm machines. I
> am having intermittent problems with ldm dropping files that are bigger
> than about 12Mb. 
> 
>   from the source machine named lanina (running Linux slackware 2.0.36 and
> ldm-5.0.8 with a 400 Mb queue) I issue this command:
> 
> ldmsend -h breeze /logs/lanina.log.gz
> 
>   I see the following activity in the ldm.log file on the destination
> machine (breeze), a slackware Linux 2.2.6 box running ldm-5.0.8 with a 600
> Mb queue: 
> 
> Dec 01 19:08:24 breeze lanina[29943]: Connection from
> lanina.wunderground.com 
> Dec 01 19:08:24 breeze lanina[29943]: hiya: 19991201190824.745 TS_ENDT
> {{EXP,  ".*"}} 
> Dec 01 19:08:24 breeze lanina[29943]: Connection reset by peer 
> Dec 01 19:08:24 breeze lanina[29943]: Exiting 
> 
> sometimes the file (16Mb) will show up, sometimes not. an "ldmadmin watch
> -f EXP" does not show the file coming across and entering the queue when
> the file doesn't show up. No error messages on either machine indicate why
> the file didn't make it.
> 
> Any ideas for troubleshooting? 

Jeff,

ldmsend is not as reliable as the newer ldm commands to send files to a
remote machine.  Try using pqinsert or pqsend to perform the tasks because
they use the LDM internal connections, etc verses ldmsend temporary set up
of connections between machine. For example, the possibility to resend the
product if a network connection is broken.  Look at the man pages for
pqinsert and pqsend.



Is there a command I can issue to see how
> mcuh of the queue is currently being taken up? 

The "pqutil stats" command gives the stats about the queue. Again look at
the man page.

I have about 80 Mb/hour of
> data going between the machines, so the 400-600 Mb queue size I am using
> should be enough to handle a 16Mb file.
> 
I would check the log file for entries about "deleting oldest" to insert
a product into the queue.  If you don't see any of these type of messages
than the queue is large enough. A 400-600 Mb queue is large enough for
80Mb / hour.

Robb...


ps. Can you add an "allow UNIDATA" line for the following sites for
failover purposes?

grayskies.atmos.washington.edu
norte.sfsu.edu
typhoon.atmos.ucla.edu
vision.soest.hawaii.edu
ocs.oce.orst.edu
bodie.met.nps.navy.mil 
ldmhost.dri.edu
meteora.ucsd.edu
nimbus.atmo.arizona.edu

Thanks,
Robb...


> Thanks, Jeff
> ------------------------------------------------------------------------------
>  Dr. Jeff Masters (address@hidden)                           (  )  
>  Chief Meteorologist                              /\ Home of the       (    ) 
>  The Weather Underground, Inc.               /\  /  \  /\       /\    (      )
>  P.O. Box 3605                              /  \/    \/  \ /\  /  \    ------
>  Ann Arbor, MI 48106-3605            ______/              /  \/    \_   \\\\\
>  734-994-8824                                   Weather Underground      \`\`\
>                                             http://www.wunderground.com
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================