20020520: McIDAS-XCD GRIB decoding, and ldm-mcidas 7.8.0 upgrade

>From: Gilbert Sebenste <address@hidden>
>Organization: NIU
>Keywords: 200205201534.g4KFYWa01146 McIDAS-XCD ldm-mcidas

Gilbert,

>Gilbert here. A couple of things.
>
>I turned off the HRS feed for LDM-McIDAS on weather2.admin.niu.edu. I 
>didn't want to, but I guess a 40 GB hard drive just doesn't hack it.

While I will be the first one to admit that the McIDAS GRID files
created by XCD are large,  I must say that if decoding and scouring is
working correctly, then you will use a maximum of 4 GB per day of GRID
data saved.  I'm pretty user that I set up your scrouing to only keep
one day, so a 40 GB hard disk should have plenty of space.

Is something not working?  What: decoding or scouring?

>ANyway, I have no problem with McIDAS upgrades, but I am scared of messing 
>up the LDM-McIDAS upgrade. Is there any chance you could do that, or 
>provide a detailed set of instructions on how to do it? Currently, I do 
>not ingest the FNEXRAD feed, but want to become a relay for the new 
>mosaics as well as make some for my web site.

The ldm-mcidas upgrades are easier than the McIDAS upgrades especially
since there are binary distributions for virtually all of the platforms
that folks are using out there.  Even though I don't mind doing the
upgrade for you, I would rather walk you through the process so you
won't be wary of it in the future.

>As always, thanks for all your help. It is greatly appreciated!!!

No worries.

Here goes with the 1-2-3 for upgrading ldm-mcidas.  All steps are
done by the user running the LDM on your system (typically 'ldm'):

1) login as 'ldm'

2) create the ldm-mcidas directory (it may already exist)

   cd ~ldm
   mkdir ldm-mcidas
   cd ldm-mcidas

3) download the binary appropriate to your system (you would choose the
   binary for linux_2.4-i686) to the ~ldm/ldm-mcidas directory:

   o either use FTP to ftp.unidata.ucar.edu as user 'anonymous' and pass
     'your email address'.  Make sure to download in binary mode

   - or -

   o download the file from the ldm-mcidas webpage:

     Unidata LDM-McIDAS
     http://www.unidata.ucar.edu/packages/mcidas/mcidd/
       Download Software
       
http://www.unidata.ucar.edu/packages/mcidas/mcidd/ldm-mcidas-download.html

4) unpack the ldm-mcidas distribution:

   zcat ldm-mcidas-7.8.0.tar.Z | tar xvf -

5) copy the updated decoders to a directory in the PATH of 'ldm'.  This is
   typically ~ldm/decoders or ~ldm/util:

   cd ldm-mcidas-7.8.0/bin
   cp pnga2area pngg2gini nldn2md proftomd ~ldm/decoders

6) update your LDM pqact.conf file to decode the FNEXRAD imagery.  In
   the ldm-mcidas distribution, in the ldm-mcidas-7.8.0/etc directory,
   you will find a file named ldm-mcidas-pqact.conf.  This file contains
   _example_ pqact.conf entries that you should copy to your pqact.conf
   file ** making sure to keep tabs in appropriate places ** and then
   edit -- if necessary.

   Part of this process will be you deciding where on disk you want to 
   save the decoded images.  This decision will vary greatly from site-to-site
   since each site has different objectives, storage capacities, etc.
   For you, since you are not running GEMPAK, I would recommend using a
   top level directory structure something like:

   ~ldm/data/nexrad/NEXRCOMP
   
   Since there are up to 6 different images in the FNEXRAD stream, you would
   end up with 6 subdirectories of this directory.  I recommend:

   ~ldm/data/nexrad/NEXRCOMP/6KN0R-NAT
   ~ldm/data/nexrad/NEXRCOMP/10KRCM-NAT
   ~ldm/data/nexrad/NEXRCOMP/1KN0R-FLT
   ~ldm/data/nexrad/NEXRCOMP/n0r
   ~ldm/data/nexrad/NEXRCOMP/n1p
   ~ldm/data/nexrad/NEXRCOMP/ntp

   If you decide to use this directory structure, your pqact.conf entries
   would end up looking like:

#########################################################################
#
# FNEXRAD section
#
#########################################################################

#
#### PNG compressed FNEXRAD image (AREA) files
#

# NEXRCOMP 6 km National BREF mosaic
FNEXRAD ^pnga2area Q5 (RL) (.*) (.*) (.*) (.*) (........) (....)
    PIPE    -close
    pnga2area -vl logs/ldm-mcidas.log
    data/nexrad/NEXRCOMP/6KN0R-NAT/\4_\6_\7

# NEXRCOMP 10 km National RCM mosaic
FNEXRAD ^pnga2area Q5 (RN) (.*) (.*) (.*) (.*) (........) (....)
    PIPE    -close
    pnga2area -vl logs/ldm-mcidas.log
    data/nexrad/NEXRCOMP/10KRCM-NAT/\4_\6_\7

# NEXRCOMP 1 km Regional BREF mosaic
FNEXRAD ^pnga2area Q5 (RO) (.*) (.*) (.*) (.*) (........) (....)
    PIPE    -close
    pnga2area -vl logs/ldm-mcidas.log
    data/nexrad/NEXRCOMP/1KN0R-FLT/\4_\6_\7

#
#### PNG compressed FNEXRAD image (GINI) files
# rad/NEXRCOMP/1km/n0r_20020520_1450
#

# NEXRAD Level III radar composites in PNG-compressed GINI format
FNEXRAD ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
    PIPE    -close
    pngg2gini -vl logs/ldm-mcidas.log
    data/nexrad/NEXRCOMP/\2/\2_\3_\4


  Mind the tabs!!!!!!

7) after making any change to pqact.conf, you should always verify its
   integrity:

   ldmadmin pqactcheck

8) if the 'ldmadmin pqactcheck' did not indicate errors in pqact.conf,
   you can send a HUP to pqact telling it to reread pqact.conf:

   ldmadmin pqactHUP

It is possible that the new GINI decoder, pngg2gini, may not be found
by the LDM processes (it should, but I have seen this happen).  If
it happens to you, I would exit the LDM; do a rehash at the Unix
command prompt; verify that you can find the decoder; and restart your
LDM:

ldmadmin stop
<wait for all LDM processes to exit>
rehash
which pngg2gini
ldmadmin start

The next item of business is setting up scouring of the ingested images.
This is a crutial step given how large the images are after decoding
(14 MB per image for each 1 km N0R composite, and they arrive every
6 minutes!).

If you follow the setup above, then you can scour all of the new composites
with one cron entry.  Since the scour technique included with the LDM
scours by number of days you want to keep, it is most likely that you
will want to do something else (240 x 14 MB == lots 'o disk space used :-).

We make available a Unix C-shell script that sours data files by number
instead of by time.  You can grab this script, prune_nexrcomp.csh,
from the pub/ldm5/scour directory of anonymous FTP:

<as 'ldm' on your system>
cd decoders
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> your_full_email_address
  cd pub/ldm5/scour
  binary
  get prune_nexrcomp.csh
  quit
chmod +x prune_nexrcomp.csh

After transferring the file, you need to edit it to setup values
pertinent to your setup.  In particular, you may need to modify the
script's internal settings for:

PATH
KEEP
areadir

For instance, if you put prune_nexrcomp.csh in ~ldm/decoders, and this
directory is actually /usr/local/ldm/decoders, your PATH setting would
become:

setenv PATH /usr/local/ldm/decoders:${PATH}

KEEP is the number of files you want to keep on disk.  This is the number
that will be left each time the scouring routine runs from cron.  It is
defaulted to 12, so you may want to change it.

'areadir' is the directory under which all subdirectories containing
files to be scoured can be found.  In the example we are working on,
this would be /usr/local/ldm/data/nexrad/NEXRCOMP.

After you edit the entries in prune_nexrad.csh, you need to add an entry
to the crontab file for your user 'ldm'.  How often you kick off this
scouring script is up to you, but we would recommend doing it no
less than once per hour.

The final step in bringing these data "to life" on your system is to
setup ADDE access to them.  Why don't we tackle this one after you
have data ingestion, decoding, and scouring setup?

Tom