20040406: LDM ingest, ldm-mcidas decoding, McIDAS access to imagery (cont.)

>From: "Luis A. Lopez" <address@hidden>
>Organization: UPRM
>Keywords: 200404051637.i35GbNCT003268

Hi Luis,

>Thank you for your explanation on how to use mcidas/ldm. programs.

No worries.

>I'm
>not sure what data the researchers need right now but certainly that the
>Puerto Rico area must be included.  How can I modify the file pqact.conf
>so ldm/mcidas can download files for Puerto Rico area??  I'll work
>around with your instructions later.

The images in the UNIWISC datastream contain coverage for North America
and, when available, GOES-East coverage of South America.

Images specific to Puerto Rico are included in the NIMAGE datasetream.
Do you want to use the LDM to ingest the NIMAGE images for just Puerto
Rico, or do you think that there is enough bandwidth to ingest images
for Puerto Rico and the Continental US?

If you want to first get just the images for Puerto Rico, I recommend
adding the following 'request' line to your ~ldm/etc/ldmd.conf file.

request NIMAGE  "PR"    atm.geo.nsf.gov

It is best to keep the request lines in the same area of your ldmd.conf
file.  This way you will be able to see at a glance all of the feeds
you are trying to get.

After making any change to ldmd.conf, you will need to stop and
restart your LDM for the changes to take effect:

ldmadmin stop
ldmadmin start

However, I recommend you do this after you add the necessary decoding
entries in ~ldm/etc/pqact.conf.

Here is what to add in ~ldm/etc/pqact.conf:

# Zlib compressed NOAAPORT GOES-East/West GINI images -- FILE
NIMAGE  ^satz/ch[0-9]/.*/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
        PIPE    -close
        util/ldmfile.sh data/gempak/images/sat/\8/\9/\1/\1_\2\3\4\5_\6\7

You must be careful about spaces versus tabs here:

- white space between NIMAGE and ^satz is a tab
- white spaces before and after PIPE are tabs
- white space before util is a tab

The action may look a little strange since it includes 'gempak'. The
reason for this is that a lot of Unidata sites use both GEMPAK and
McIDAS.  Give this the template I send out in McIDAS to setup ADDE
datasets of NIMAGE data use the above formalism.

Next, you need to grab the ldmfile.sh script from the our FTP server,
ftp.unidata.ucar.edu:

<as 'ldm'>
cd ~ldm/util
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> address@hidden
  cd pub/ldm-mcidas/util
  binary
  get ldmfile.sh
  quit
chmod +x ldmfile.sh

At this point, you can stop and restart your LDM (see above).  If the
LDM does not start, it means that you have a typo either in ldmd.conf
or pqact.conf.

On to setting up a McIDAS ADDE dataset for the NIMAGE data:

<as 'mcidas'>
cd ~mcidas/data
cp GINIADDE.BAT LOCGINI.BAT
- edit LOCGINI.BAT and change every occurrance of:

  directory_file_regular_expression

  to the fully qualified directory name under which the images will be
  filed.  In your case, using the example in the previous email, this
  would be:

  /data/ldm/gempak/images/sat

cd ~mcidas/workdata
batch.k LOCGINI.BAT

Something new:

The NIMAGE images get filed in a directory hierarchy, and new files get
added to an output directory each time a new image comes in.  This will
grow without bound unless you take steps to scour the image
directories.  You do this as follows:

<as 'ldm'>
cd ~ldm/util
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> address@hidden
  cd put/ldm/scour
  binary
  get prune_gini.csh
  quit
chmod +x prune_gini.csh

- edit prune_gini.csh and modify if needed the setting for PATH, KEEP,
  and areadir:

  PATH - modify so that ~ldm/util is included
  KEEP - modify to keep more or fewer of each kind of image you ingest/file
  areadir - modify to match the top level directory under which the
            images are being filed.  In your case, this would be:

change:

   set areadir=/usr/local/ldm/data/gempak/nport

to:

   set areadir=$HOME/data/gempak/images/sat

   -- or --

   set areadir=/data/ldm/gempak/images/sat

Finally, run the prune_gini.csh script out of cron.  Here is an example
invocation we use:

5,20,35,50 * * * * util/prune_gini.csh >/dev/null 2>&1

One last comment:  McIDAS users do not need the data to be local to
display/analyze it.  All of the imagery in the NIMAGE IDD stream is
available on more than one ADDE server that is accessible to users
running McIDAS as long as they have Internet access and have ports 500,
503, and 112 open in their firewalls.  This means that a McIDAS user
does not need to have the LDM running at all!  Instead, they can
"point" their McIDAS-X session at one of the ADDE servers that has the
data they want, and begin using it straight off.  If you want to test
this, do the following:

<login as 'mcidas'>
cd workdata
- start a McIDAS-X session using:

mcidas -config

Make sure to select the radiobutton labeled MCGUI so that the Unidata
MCGUI interface will be started, and then click on the Start button.

Once you have a McIDAS-X MCGUI session, click the button button that
has two computers on it, one over the other. This is just to the right
of the button with the red Z on it.  This brings up a GUI that allows
you to set the ADDE server(s) you will use for a list of datasets.
Click on the tab labeled IMAGE, and then click on the button to the
right of the dataset that says GINIEAST.  Select atm.geo.nsf.gov, and
then click on the button labeled Update and Exit.

Next, click on the Display dropdown and select Imagery and then
GINIEAST.  A GUI will open that allows you to select the type of image
you want to display from the dataset GINIEAST.  Choose GPN8KIR (GINI
Puerto Rico 8 km IR) and click on the button labled Display.  You
should get a display of the most recent Puerto Rico National 8 km IR
(10.7 um) image from the ADDE server on atm.geo.nsf.gov.

Please let me know if you were able to do the above and what you
think.

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.