20030113: Ingesting NIMAGE data with the LDM (cont.)

>From: "Paul L. Sirvatka" <address@hidden>
>Organization: COD
>Keywords: 200301132246.h0DMkDx08825 LDM NIMAGE regular expression

Paul,

re: Gilbert's basis for not thinking you can handle all of NIMAGE

>I think that he misspoke. He was referring to the volume of data in ch's 1
>and 2. I am looking just to grab the satellite imagery from GOES east and
>west...

This _is_ what is in chs 1 and 2, and all of that is in the NIMAGE stream.
Volumewise, the biggest portion of the NIMAGE stream is the 1 km VIS
images.  Each of these is ~14 MB compressed, and ~26 MB uncompressed.

>ultimately I would like the 1km VIS, 4KM IR and 8 km WV.

OK.  The other products are East/West composites; some images that cover
a larger geographical area; and images focused in on Puerto Rico, Alaska,
and Hawaii.

re: to request just the GOES-East/West 1 km VIS images, use the following
patterm

>> request      "VIS.*CONUS"    papagayo.unl.edu

re: and test your pattern(s) using notifyme

>> <login as 'ldm'>
>> notifyme -vxl- -f NIMAGE -o 3600 -h papagayo.unl.edu -p "VIS.*CONUS"

>OK ...so I am confused here.
>
>I will set up the ldmd.conf file to request 
>VIS.*CONUS and IR.*CONUS and WV.*CONUS

So, this could be done with:

request "(VIS|IR|WV).*CONUS"    papagayo.unl.edu

>This will send me that data. I think I can handle that. It is not too much
>is it?

There are 4 VIS images per hour from both GOES-East and GOES-West.  After
uncompression, you will have 100 MB of VIS imagery from _each_ satellite.
The other imagery is small in comparison to this.

>THen in the pqact I would do what?

Add actions for the decoding (unPNGing) and filing of the images.

Again, the headers for the VIS images look like:

sat/ch1/GOES-8/VIS/20030113 2215/EAST-CONUS/1km/ TIGE01 KNES 13221
sat/ch2/GOES-10/VIS/20030113 2200/WEST-CONUS/1km/ TIGW01 KNES 132200

An example pqact.conf action for filing these might look like:

NIMAGE  ^sat/ch[0-9]/.*/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
        PIPE    -close
        decoders/pngg2gini -vl logs/ldm-mcidas.log
        data/nport/\8/\9/\1/\1_\2\3\4\5_\6\7


This assumes:

1) you put the ldm-mcidas decoders into the ~ldm/decoders directory

2) you want logging from the ldm-mcidas decoder pngg2gini to be written
   into the ~ldm/logs/ldm-mcidas.log file

3) you want to file the images into the directory ~ldm/data/nport/...
   directory hierarchy.  Directories you will see there will be
   EAST-CONUS, and WEST-CONUS:

   ~ldm/data/nport/EAST-CONUS/1km/VIS
   ~ldm/data/nport/EAST-CONUS/4km/3.9
   ~ldm/data/nport/EAST-CONUS/4km/12.0
   ~ldm/data/nport/EAST-CONUS/4km/IR
   ~ldm/data/nport/EAST-CONUS/8km/WV

   ~ldm/data/nport/WEST-CONUS/1km/VIS
   ~ldm/data/nport/WEST-CONUS/4km/3.9
   ~ldm/data/nport/WEST-CONUS/4km/12.0
   ~ldm/data/nport/WEST-CONUS/4km/IR
   ~ldm/data/nport/WEST-CONUS/8km/WV

4) you want file names to reflect the wavelength:

   VIS  -> VIS_ccyymmdd_hhmm
   3.9  -> 3.9_ccyymmdd_hhmm
   12.0 -> 12.0_ccyymmdd_hhmm
   IR   -> IR_ccyymmdd_hhmm
   WV   -> WV_ccyymmdd_hhmm

One other thing, you will need to setup data scouring to remove images
so your disk won't fill.  We make available scripts that can be
tailored for how many images you want to keep and then run from cron.
The script that you will need to grab, configure, and run from cron
is prune_gini.csh.  This can be found in the pub/ldm5/scour directory
of anonymous FTP on our FTP server, ftp.unidata.ucar.edu.  Grab this
script in binary mode, and put it into the directory where you
put your ldm-mcidas decoders.  Also, make sure that you change it
so that it has execute permission so it will run from cron.  Finally,
edit the script and setup the top level directory you want to start
scouring from and how many images you want to keep.  If, for instance,
you do choose to write the NIMAGE images in ~ldm/data/nport/...
and you want to keep 12 of each image, you would modify prune_gini.csh
to look like:

change:

setenv PATH /usr/local/ldm/util:${PATH}
set KEEP=12
if($#argv == 0) then
   set areadir=/usr/local/ldm/data/gempak/nport
   set START_DATE=`date -u`
else
   set areadir=$1
endif


to:

setenv PATH /usr/local/ldm/decoders:${PATH}
set KEEP=12
if($#argv == 0) then
   set areadir=/usr/local/ldm/data/nport
   set START_DATE=`date -u`
else
   set areadir=$1
endif


This assumes that the HOME directory for your 'ldm' user is /usr/local/ldm.
Adjust this if necessary.

A cron entry for prune_gini.csh might look like:

5,20,35,50 * * * * decoders/prune_gini.csh >/dev/null 2>&1


This entry runs every 15 minutes.

>Do I want more than one file?

I don't understand this question.  Do you mean do you want to keep more
than one file?  I would say yes so that you can do loops from local imagery.

>It is an AREA file is it not?

The images will be in GINI file format, not AREA.  The Unidata McIDAS
GINI ADDE servers know how to handle images in this format.

>What would the naming be and how would i then set up dataloc?

The naming is determined by your pqact.conf action.  In McIDAS, you
will need to steup ADDE datasets to access the imagery.   The file
GINIADDE.BAT is included in the distribution as a template to help you
do this.  You need to make a copy of GINIADDE.BAT and modify it to
match your setup.  Let's get to that after you get data ingestion,
decoding, and scouring working.

>Sorry to be such a bother lately but I want to get these images working
>for good!

OK, sounds good.

Tom