19990618: July 1st Datastream changes (cont.)

>From: "Jennie L. Moody" <address@hidden>
>Organization: UVa
>Keywords: 199906162038.OAA01382 McIDAS LDM cron ROUTE PP

Jennie,

>> Just in the nick of time ;-)
>Anyway, that remains to be seen.

re: look into doing something based on use of pqact since that is the
best indication of when data comes in.
>Okay.

>I was hoping there might be some examples to work from in
>an updated pqact.conf from Unidata, but I pulled the
>current file from the unidata ftp site, and no such luck :-)

No, something like this is too specific to put into an example
pqact.conf file.

>So, before I spend too much effort on this (though it seems
>inevitable that this is going to be an intense learning
>experience), let me just run past you what I think I
>have to do.  Then you can let me know if this is really 
>wrong-headed thinking! 

How about asking the members of the McIDAS community if they have done
something similar to the work you need (address@hidden)?

>o current batch files fire when the mcidas routing table
>  gets updated, for example, when the early domestic product
>  (the NGM) arrives

Right.  The question is whether or not you are using the data in the UW
NGM GRID product?  I would guess not.  If not, then all you really need
to know is when the model run you are interested in is most likely to
have been received.

>o we need to mimic the detection of product arrival by 
>  watching for headers in the pqact.  

That is certainly a way of improving your odds that the data you need
has arrived.

>problem- there will be lots of headers for a given grid
>         (like the NGM early domestic product, as an
>         example), so it will be necessary to use a very
>         specific but representative regular expression
>         looking for some key component of a grid.
>e.g.
>HRS    ^YTQA50 KWB. (..)(..).*/mNGM

Right.

>(at least I think, after reviewing header info this afternoon, that
>this would be the 500mb T analysis on the NGM #211 CONUS 80km grid,
>the point is, I need a pattern that would only be matched _once_
>per new-incoming-grid)

I assume that you say this so that you don't kick off a script multiple
times; true?  If so, then you could get around coming up with a totally
specific header (next to impossible, by the way) by creating a file
that the script checks for.  If the file exists, then the script has
already been fired off; if it doesn't already exist, create it and
continue in the PP script.  If the script is a McIDAS script, then you
could use the string or system key tables to store a flag that the
script can interrogate to see if it should continue.

>o This pattern match could then trigger an action, e.g.
>
>       EXEC ngm.scr
>
>o The shell script ngm.scr would contain either just a
>command to sleep (long enough for the GRID to finish coming in,
>a time I don't have a handle on yet), 

Or you could reschedule the script using 'at'.

>or something with logic that would sleep/look_at_incoming/data/xcd
>until it found a current GRID in the right number range (specified 
>by the xcd decoders), of approximately the right size with a current 
>time stamp, 

This would work as well.

>o then it would invoke something like batch.k, 
>that is, the current ldma-run shell script which
>builds the right mcidas-environment, and sends the correct 
>McIDAS arguments to the real executable command: batch.k

Sounds feasible.

>problem- I am wondering this: 
>       Is there a way that the xcd-decoders
>       can write to the SYSKEY.TAB so that arguments 
>       similar to those used now in PP BATCH *.BAT files get
>       defined? 

The surface, upper air, synoptic, ship/buoy, and pirep/airep already do
this.  They do this because we added this capability to them.  I guess
that you are requesting that the GRID decoder be modified to do the
same based on the receipt of a particular kind of data?

>What I mean is this, sticking with the NGM example,
>right now the arrival of the NGM grid in the Unidata-McIDAS 
>datastream results in an updated SYSKEY table, so that 
>SYSVAL 2031=GRID# of NGM; 2032=HOUR; 2033=DAY associated 
>w/ this grid.  These are arguments passed in the
>current batch invocation (for a batch like ADDGRID.BAT).

Right.

>Either way, with the impending change in the datastream, this
>information will obviously be gone.

Yes, but I was under the impression that you were not using the NGM
data in the UW stream.

>So is there a way that similar
>kinds of information gets (can get?) created by the decoders (e.g.
>ingebin)?  

Unfortunately, no.  You could run GRDLIST to list out what grids are
available and parse the ouput to see if the kind you are really
interested in exists.  This is not really hard, but it is a little
tedious.

re: ROUTE PP BATCH stuff for images will continue to work (for awhile)
>Good news!

Well, my messages can't be all bad news.  People would stop asking
questions :-).

>Thanks for your help.

Sorry I don't have the time to dive into this problem right now.  It is
something that I feel like I have to address at some point, but other
things are higher up on the list of things to do.  The three things
that I really want to work on are:

o finishing my GUI
o getting an ADDE GINI server running
o getting an ADDE TeraScan (tm) server running

With the last two facilities sites would be able to access all channels
of the fabulous high resolution data from both GOES platforms.  Chiz
has been working on stuff that Glenn was in the middle of that was
related to developing a card/software for ingesting all four of the
NOAAPORT channels on a single PC.  He has been grabbing the rapid scan
(7.5 minute) GINI images off of the feed and looking at them in
GEMPAK/GARP.  SSEC wrote a GINI to AREA converter some time ago, but
they havn't given it to me yet.  As soon as they do, I will roll the
code into an AGET and ADIR server so that users can browse/load/copy
GINI imagery.  The rapid scan loops that we have looked at recently are
spectacular.

Following that, a server for imagery in TeraScan format (tdf) will
allow users to browse/load/copy imagery in native satellite projection
(GINI is remapped into Lambert Conformal).  The TeraScan development is
primarily for JOSS, but I have been promised that the the data will be
made available to the outside world (Unidata sites, at least).  This
would mean that you could browse any of the GOES East/West imagery on
windfall and copy over what you want/need.  In addition, JOSS will be
archiving the data, so historic data sets will then be available (after
they build up an archive, that is).  Pretty cool, huh?

Tom