20020418: ADDE and ldm-mcidas log files rotation in cron

>From: Clint Rowe <address@hidden>
>Organization: UNL
>Keywords: 200204181519.g3IFJaa23128 ldm-mcidas ADDE log

Clint,

>I just noticed a 12M SERVER.LOG and a 25M ldm-mcidas.log in ~ldm/logs.

I thought that I had setup rotation of these log files!  

<after a quick login to papagayo>

I did setup rotation of the logs:

<login to papagayo as 'ldm'>

%crontab -l

 ...

#
# Rotate ADDE access log files just past 12 midnight on first day of month
#
1 0 1 * * bin/newlog logs/SERVER.LOG 3; chmod 666 logs/SERVER.LOG

 ...

Now, the question is what is wrong with the cron entry?  My intention
was to rotate the log file on the first minute of the first day of
every month.  I chose monthly rotation of the log files since papagayo
was not getting as much ADDE traffic as, say, motherlode.

>How do I go about reducing the size of these without reducing any functional
>use you get out of them?

The size of an individual log file will depend on how many ADDE hits
you get before the file gets rotated.  The only way to keep the total
number of bytes down is to keep few of the log files.

>I already removed a really big png.log, but I see
>it's coming back.  Can I institute some process to do this automatically?

The png.log file would need to be rotated as well, but I decided to do 
away with it.  See below.

Since it looks like papagayo is getting more ADDE hits, I changed
the 'ldm' cron entry to do weekly rotations and only keep 2 SERVER.LOG.*
files:

#
# Rotate ADDE access log files just past 12 midnight on Saturdays
#
1 0 * * 6 bin/newlog logs/SERVER.LOG 3; chmod 666 logs/SERVER.LOG

This matches the rotation on motherlode, so no individual SERVER.LOG.*
file will get to be that big.

Now, on to the png.log file.

Last night I updated papagayo with a new distribution of ldm-mcidas
decoders.  One of these decoders is a replacement for the readpng
decoder that has been unpacking your NIMAGE data for some time.  Since
the new decoder, pngg2gini, is an ldm-mcidas one, it makes more sense
for it to log into the ~ldm/logs/ldm-mcidas.log log file.  Also, since
pnga2area, nldn2md, and cdftomd are also ldm-mcidas decoders, I also
changed their logging to logs/ldm-mcidas.log.

Since you will now see logs/ldm-mcidas.log files, I added a cron entry
to rotate those at the same time that the LDM log files are rotated:

#
# Rotate ldm-mcidas decoder log files just past 12 midnight daily
#
0 0 * * * bin/newlog logs/ldm-mcidas.log 2

I also deleted the logs/png.log file since it will no longer be updated.

I will be updating McIDAS on papagayo sometime today.  The mods
will relate to ADDE serving of imagery in GINI format.  This includes
the GOES GINI imagery that you have been getting for some time and,
also, the NEXRAD Level III composite imagery that you were getting
on the NIMAGE feed.  Last night, Chiz and I changed the NEXRAD composite
distribution to FNEXRAD, and I added decoding of those images and others
with the new ldm-mcidas decoder, pngg2gini.  You are now getting:

1 km N0R national composite base reflectivity
2 km N1P national composite 1-hour precipitation
4 km NTP national composite storm total precipitation

While at it, I also setup McIDAS to serve these images.  The only change
that you might notice is in where the data is being filed.  All of the
files are going into subdirectories of /data/pub/raw/gini/RADAR:

/data/pub/raw/gini/RADAR :

1km  2km  4km

The names of the subdirectories under 1km are 'n0r'; 2km is 'n1p', and
4km is 'ntp'.  The names of files in those subdirectories follow:

n0r_CCYYMMDD_HHMM
n1p_CCYYMMDD_HHMM
ntp_CCYYMMDD_HHMM

I see that there is some problem with the N0R filing, so I will check
the LDM ldmd.conf request line to make sure that you are requesting
all of the data...  You are:

request FNEXRAD ".*"    flood.atmos.uiuc.edu

Now I have to find out why you are not receiving the N0R composites.
I will report back on this later.  The N1P and NTP composites are coming
in and being decoded with no problem, so I am at a loss right off forr
what may be going wrong.

Stay tuned...

Tom