20020418: changes in LDM feed for composite NEXRAD products (cont.)

>From: Unidata Support <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200204181155.g3IBtAa14008 IDD McIDAS ADDE NEXRAD composites

Tom,

>Why don't you walk me through the changes. I want to try to keep up with
>this as much as I can. 

OK, here goes.

First, some background:

o before yesterday evening, we were sending the 1 km N0R NEXRAD Level III
  national composite base reflectivity product in PNG-compressed GINI
  format in the NIMAGE datastream.  The product header began with 'rad_'

o last night, we changed the datastream that this product is being sent
  in to FNEXRAD.  Furthermore, we changed the header of the products so
  that they now begin 'rad/NEXRCOMP/.../'.  Here the '...' is now the
  resolution of the product.

o in addition to the 1 km N0R composite, we are now also sending a 2 km
  N1P national 1-hour precip. product, and a 4 km NTP national storm
  total precip. product.

o Headers for the N0R, N1P, and NTP products in the FNEXRAD feed now look
  like:

  rad/NEXRCOMP/1km/n0r_20020418_1817
  rad/NEXRCOMP/2km/n1p_20020418_1821
  rad/NEXRCOMP/4km/ntp_20020418_1822


I. Getting the NEXRAD composites over the IDD

Your system, cacimbo, is currently setup to request the N0R composite from
the NIMAGE feed and only products whose headers begin with 'pnga2area'
from the FNEXRAD feed:

request NIMAGE "^rad_" atm.geo.nsf.gov
request FNEXRAD "^pnga2area Q5" atm.geo.nsf.gov

We need to change these request lines to request all of the new composites.
Since these are not being sent in the NIMAGE feed any longer, that request
can be commented out or deleted.  The remaining request line should look
like:

request FNEXRAD "^(rad/|pnga2area)" atm.geo.nsf.gov

Since you have versions of ldmd.conf for failover LDM feeds, you will
need to make this change in all ldmd.conf.* files in ~ldm/etc.  After
making the change, you will need to stop and restart your LDM, but
you might as well wait to do that until after you modify your pqact.conf
file to decode the files.

II. Decoding the NEXRAD composites with ~ldm/etc/pqact.conf actions

I have made a new release of the ldm-mcidas decoder package (not yet
officially announced, but available nonetheless).  This new bundle
contains updated decoders for the PNG-compressed images in AREA file
format in the MCIDAS datastream (pnga2area); a new decoder for
PNG-compressed images in GINI format in the FNEXRAD datastream (pngg2gini),
and new versions of the NLDN lightning and FSL wind profiler decoders,
nldn2md and cdftomd, respectively.

The big changes in the decoders you are currently using are:

o they are linked against McIDAS-X 7.8x
o they are linked against Zlib 1.1.4
o they are linked against PNG 1.2.1

The new decoder, pngg2gini, is a direct replacement for the unsupported
'readpng' decoder that has been decoding the NEXRAD composites from the
NIMAGE feed on cacimbo.

In order to start using the new/newly updated decoders, you will need
to:

1) login to cacimbo as 'ldm'
2) FTP the new ldm-mcidas binary distribution, Version 7.8.0, from
   Unidata:

cd ~ldm
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> your_full_email_address
  cd pub/binary/linux_2.4-i686
  binary
  get ldm-mcidas-7.8.0.tar.Z
  quit

3) unpack the ldm-mcidas distribution:

zcat ldm-mcidas-7.8.0.tar.Z | tar xvf -

4) verify that the new decoders will run on your system:

cd ldm-mcidas-7.8.0/bin
ldd pnga2area
ldd pngg2gini
ldd nldn2md
ldd cdftomd

4) IF all looks OK with the new decoders, copy them to the
   ~ldm/decoders/bin directory

cp pnga2area pngg2gini nldn2md cdftomd ~/decoders/bin

5) Currently, you have an action in ~ldm/etc/pqact.conf that will decode
   N0R NEXRAD composites from the NIMAGE feed:

#
# 1 km National N0R composites in GINI format produced by Unidata
NIMAGE  ^rad_(........)_(....)
        PIPE    -close
        decoders/bin/readpng -n
        /data/nexrad/NEXRCOMP/1km/national/rad_\1_\2

   And, you have three actions that will decode other FNEXRAD composites
   using 'pnga2area':

#
# NEXRCOMP 6 km National BREF mosaic
FNEXRAD ^pnga2area Q5 (RL) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        /usr/local/ldm/ldm-mcidas/bin/pnga2area -v
        /data/nexrad/NEXRCOMP/6km/\4_\6_\7
#
# NEXRCOMP 1 km Regional BREF mosaic
FNEXRAD ^pnga2area Q5 (RM) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        /usr/local/ldm/ldm-mcidas/bin/pnga2area -v
        /data/nexrad/NEXRCOMP/1km/floater/\4_\6_\7

#
# NEXRCOMP 10 km National RCM mosaic
FNEXRAD ^pnga2area Q5 (RN) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        /usr/local/ldm/ldm-mcidas/bin/pnga2area -v
        /data/nexrad/NEXRCOMP/10km/\4_\6_\7

  You will need to change these entries as follows:

  a) change the existing FNEXRAD entries to use the 7.8.0 ldm-mcidas
     version of pnag2area from the ~ldm/decoders directory:

     N.B.: be careful to put/keep tabs where needed!!!!!

#
# NEXRCOMP 6 km National BREF mosaic
FNEXRAD ^pnga2area Q5 (RL) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        decoders/bin/pnga2area -v
        /data/nexrad/NEXRCOMP/6km/\4_\6_\7
#
# NEXRCOMP 1 km Regional BREF mosaic
FNEXRAD ^pnga2area Q5 (RM) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        decoders/bin/pnga2area -v
        /data/nexrad/NEXRCOMP/1km/floater/\4_\6_\7

#
# NEXRCOMP 10 km National RCM mosaic
FNEXRAD ^pnga2area Q5 (RN) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        decoders/bin/pnga2area -v
        /data/nexrad/NEXRCOMP/10km/\4_\6_\7

  b) replace the entry for decoding of NIMAGE products with one that
     uses the new ldm-mcidas decoder for PNG-compressed GINI imagery:

     N.B.: be careful to put/keep tabs where needed!!!!!

#
# 1 km National N0R composites in GINI format produced by Unidata
NIMAGE  ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
        PIPE    -close
        decoders/bin/pngg2gini -v
        /data/nexrad/NEXRCOMP/\1/national/\2_\3_\4

6) whenever you make any changes to ~ldm/pqact.conf, always run ldmadmin
   to see if you made any mistakes:

   ldmadmin pqactcheck

7) if ldmadmin did not admonish you about errors, stop and restart your
   LDM:

   ldmadmin stop
   <wait until ALL LDM processes have exited>
   ldmadmin start

   At this point, your LDM should start getting all of the composites in
   the FNEXRAD datastream, and pqact should decode all of these into
   formats usable by McIDAS and GEMPAK.

8) the new pqact.conf action that decodes the composites will name
   them differently than your current action.  Instead of files whose
   names look like:

   rad_CCYYMMDD_HHMM

   you will now get files named like:

   n0r_CCYYMMDD_HHMM
   n1p_CCYYMMDD_HHMM
   ntp_CCYYMMDD_HHMM


Comments:

You may want to rethink the directory structure you are using for
storing the NEXRAD composites.  This is not absolutely necessary,
but you may consider it nonetheless.  The current structure looks like:

N0R composites:

/data/nexrad/NEXRCOMP/1km/national/rad_20020417_1425

This will start looking like:

/data/nexrad/NEXRCOMP/1km/national/n0r_20020417_1425
/data/nexrad/NEXRCOMP/2km/national/n1p_20020417_1425
/data/nexrad/NEXRCOMP/4km/national/ntp_20020417_1425

It might make more sense to change the storage to look more GEMPAK-like:

/data/nexrad/NEXRCOMP/1km/n0r/n0r_20020417_1425
/data/nexrad/NEXRCOMP/2km/n1p/n1p_20020417_1425
/data/nexrad/NEXRCOMP/4km/ntp/ntp_20020417_1425

If you make this change, you will need to let me know so I can update
the ADDE server configuration for the files, and you may need to make
modifications in the scouring action you have in your 'ldm' cron file
(I can say, because I can't look at the entry).

Anyway, as soon as you make the mods to ldmd.conf and pqact.conf, you
should start seeing the new products being filed.  If I didn't make
any mistakes in the ADDE setup, you should also be able to look at
the new products using the NEXRCOMP dataset of ADDE from cacimbo.

Please let me know if you run into any snags.

Tom