[McIDAS #MHL-637032]: question about imgremap

Hi Yi,

re:
> The server is started and avaiable for your access. It was
> out of memory.

Thanks for re-allowing SSH access.  

> I stopped the ldm and checked on the tiff file.
> The file was generated, with some error(so it's not viewable).

OK.  If the file was generated, it should mean that the
ADDE server setup for the NEXRCOMP/4KNTP-NAT set is correct.

In order to troubleshoot the LDM-initiated script you are
running, I started the LDM after deleting and remaking the
queue:

<as 'ldm'>
cd ~ldm
ldmadmin clean && ldmadmin delqueue
ldmadmin mkqueue -f && ldmadmin start

I see that the mcrun.log file reached its maximum size of
2 GB sometime yesterday (21st).  This was due to the extremely
long lists of NEXRACOMP/4KNTP-NAT.ALL files that were being produced
by the IMGLIST invocation I added to your ~ldm/mcrunNEXRAD.sh
script.  I modified the script to:

- remove the DATALOC LIST NEXRCOMP entry
- change the IMGLIST to IMGLIST NEXRCOMP/4KNTP.NAT

The IMGLIST listing will now contain the information for the
most recent image only.

> I checked the mcrun.log, there are thousands of files listed
> (all centered at TWX), I suspect they are not NEXRCOMP files
> which I want.

All NEXRAD Level III national composites are labeled as being
centered on TWX since it is the closet NEXRAD station to the
actual center of the image.  So, these are the NEXRCOMP images
you want.

> According to scouring you suggested, because all kinds of
> historical data are also used in our project, we keep them
> on harddisk without deleting any of them. We will think about
> organizing them in an efficient way.

I think you need organize the data as soon as possible since
ADDE access to files in the directory will take longer and longer.
The reason for this is the ADDE server needs to open each file,
determine its date and time and then time sort them.  As things
stand now, there are too many images in the directory to be able
to do an 'ls' on the contents.  I am concerned that the mcrunNEXRAD.sh
invocations will start failing if they havn't already.

I suggest organizing the images into daily directories; that is what
we do on our ADDE/THREDDS servers.

> Thanks a lot for your help,

RECAP:  After restarting your LDM and watching as national composite
total precipitation images came in, I noticed that your machine
was taking an excessively long time to run mcrunNEXRAD.sh.  In
fact, since I restarted the LDM, none of the 6 invocations of
mcrunNEXRAD.sh has progressed to the point of listing information
for the most recently received image.  The load average on the
machine is almost 8, and the top CPU users are the ADDE server
that are trying to get a listing of the national composite total
precipitation images.  My interpretation of this is there are
just too many images in the NEXRCOMP/4KNTP-NAT dataset.  In
fact, the situtation is getting worse by the minute since new
invocations of mcrunNEXRAD.sh are being kicked off before old
ones have finished.  At some point very soon, the CPU load will
go critical, and memory will get exhausted.  

So, the first thing to do is reorganize the files into daily
subdirectories and redefine the ADDE NEXRCOMP/4KNTP-NAT dataset
to point to the current day (easiest way to do this is create a
'current' link to the current day and then define the dataset
using the 'current' in the DIREFILE= regular expression.


Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: MHL-637032
Department: Support McIDAS
Priority: Normal
Status: Open