20030814: access to NIMAGE datasetream

>From: =?ISO-8859-1?Q?Christian_Pag=E9?= <address@hidden>
>Organization: UQAM
>Keywords: 200308132035.h7DKZ1Ld004120 IDD NIMAGE

Hi Christian,

>Three questions :
>
>1- Why is there a special scouring script? Why not using the standard  
>ldm scouring?

The LDM 'scour' utility (run by 'ldmadmin scour') works on a day-by-day
basis and in single directories.  We (Steve Chiswell) developed several
scour scripts that can be used to scour more frequently than once per
day and recursively scour directory trees.  We find that for high
volume datastreams like NIMAGE and NNEXRAD scouring needs to be done
either more frequently than once per day, and, in some instances, needs
to be done not by time, but by number of files desired to keep.  An
example of the latter is the scouring script scour_bynumber.csh that is
located in the same anonymous FTP directory as the other scouring
scripts, ftp://ftp.unidata.ucar.edu/pub/ldm/scour.

>2- Are there examples of McIDAS setup to serve this data?

The setup to serve the NOAAPORT GINI data, either compressed or
uncompressed, is contained in the GINIADDE.BAT file that gets
installed in the ~mcidas/data directory.  My recommendation is to make
a local copy of GINIADDE.BAT:

<as 'mcidas'>
cd ~mcidas/data
cp GINIADDE.BAT UQAMGINI.BAT

and then edit the local copy to set the information needed to serve
the various dataset groups GINICOMP, GINIEAST, and GINIWEST.  The
actions set in your local copy of GINIADDE.BAT are made active in
the 'mcidas' account with:

cd ~mcidas/workdata
batch.k UQAMGINI.BAT

The ADDE servers that I wrote and include in Unidata McIDAS (servers
for GINI, NNEXRAD, NIDS, and NOWRAD imagery) support two different
methods of configuration:

- the DIRFILE= keyword of DSSERVE
- the INFO= keyword of DSSERVE

GINIADDE.BAT shows how to use the DIRFILE= keyword approach.  For
example, the "raw" entry:

DSSERVE ADD GINIEAST/GE1KVIS    GINI  TYPE=IMAGE 
DIRFILE=directory_file_regular_expression/EAST-CONUS/1km/VIS/VIS_* "GINI 1 km 
VIS East CONUS

is designed for the user to change 'directory_file_regular_expression'
into something like '/data/ldm/gempak/nport' so that the entry ends up
looking like:

DSSERVE ADD GINIEAST/GE1KVIS    GINI  TYPE=IMAGE 
DIRFILE=/data/ldm/gempak/nport/EAST-CONUS/1km/VIS/VIS_* "GINI 1 km VIS East 
CONUS

This is, of course, for GOES-East VIS images that are stored in the
/data/ldm/gempak/nport/EAST-CONUS/1km/VIS directory and are named something
that matches the regular expression VIS*.  The test to see if you have set
the DIRFILE value correctly is to use it in an 'ls' invocation as the
user 'mcidas':

ls /data/ldm/gempak/nport/EAST-CONUS/1km/VIS/VIS_*

If 'ls' works with the fully qualified pathname regular expression, then
the ADDE server will also.

The same entry for the other approach would look like:

DSSERVE ADD GINIEAST/GE1KVIS    GINI  TYPE=IMAGE INFO=UQAMGINI.CFG "GINI 1 km 
VIS East CONUS

The user (you) would then need to create the file
~mcidas/workdata/UQAMGINI.CFG with appropriate entries that tell the
server which files to look for.  The template for this configuration
file is the file GINI.CFG that gets installed in the ~mcidas/workdata
directory.  The recommendation is to make UQAMGINI.CFG a local copy of
GINI.CFG which has been edited to set directory locations, file name
masks, and also provide access control.  I developed the INFO=xxx.CFG
file approach outside of the standard SSEC context since I wanted to be
able to easily control access to subsets of datasets.  Take a look at
GINI.CFG to get a better idea of what I am trying to explain :-)

>3- Are there specifications for headers so that I can select what I  
>want with pqact.conf?

The headers for the NIMAGE products are pretty much self explanatory.
The easiest way to get familiar with what they look like is to use
the LDM 'notifyme' facility:

<login as 'ldm'>
notifyme -vxl- -f NIMAGE -o 3600 -h atm.geo.nsf.gov

Here is some sample output:

Aug 14 20:52:50 notifyme[24401]: Starting Up: atm.geo.nsf.gov: 
20030814195250.630 TS_ENDT {{NIMAGE,  ".*"}}
Aug 14 20:52:50 notifyme[24401]: Connected to upstream LDM-5
        NOTIFYME(atm.geo.nsf.gov) returns OK
Aug 14 20:52:50 notifyme[24401]: NOTIFYME(atm.geo.nsf.gov): OK
Aug 14 20:52:51 notifyme[24401]: 9ad478cc8a62c3057fbfac4ffe8c99b1   167498 
20030814195603.854  NIMAGE 3492899  satz/ch2/GOES-10/3.9/20030814 
1945/AK-REGIONAL/8km/ TIGA04 KNES 141945
Aug 14 20:52:51 notifyme[24401]: 006f0936b0023b97b22c800357ac6fce    17635 
20030814195604.043  NIMAGE 3492900  satz/ch2/GOES-10/WV/20030814 
1945/AK-REGIONAL/16km/ TIGA05 KNES 141945
Aug 14 20:52:51 notifyme[24401]: eb0c7e76e52d4f60646574e1316c3c55   165646 
20030814195612.127  NIMAGE 3492928  satz/ch2/GOES-10/IR/20030814 
1945/AK-REGIONAL/8km/ TIGA02 KNES 141945
Aug 14 20:52:51 notifyme[24401]: 29d7f9b4e06f409ad7290f0b8c15d294 17694004 
20030814195616.065  NIMAGE 20072  satz/ch2/GOES-10/VIS/20030814 
1945/WEST-CONUS/1km/ TIGW01 KNES 141945

I think that by comparing the example pqact.conf entries I sent in the
previous email (or the one before that) with the headers you see from
the 'notifyme' invocation, you will catch on to the header structure
quickly.

>Thanks,

No worries.

Tom

>From address@hidden Fri Aug 15 14:31:39 2003

Hi,

I am receiving correctly now the GINI images and I am able to view them  
in McIDAS correctly. :-))

Thanks!

Christian Pagé
UQAM