[CONDUIT #MAD-323242]: regular expression for global GFS model output in CONDUIT

Hi Mark,

re: email you sent to ldm-users a week ago regarding getting a portion of
the CONDUIT feed

I apologize for taking so long to get back to you on this.  We were waiting
to see if someone in the LDM-using community would jump in with guidance
for you...

re:
> I've been trying to get one model from the CONDUIT data feed for our ldm
> but I'm not sure how to set up the request in my ldmd.conf.  I'm only
> interested in the global GFS model output and do not want to request the
> entire CONDUIT feed.

OK.  This would require fashioning a regular expression that would limit
the data requested in CONDUIT.  More below...

> I believe this is available from our NOAAport.

The CONDUIT IDD datafeed contains no data from NOAAPort.  CONDUIT is populated
with high resolution model output that is available from the NWS/NCEP FTP
servers.  There is, however, global GFS data in NOAAPort; it is the lower
resolution (e.g., 1.25 degree thinned) data.  CONDUIT contains global 1.0 and
0.5 degree GFS output.

Question:

- which global GFS output are you really after?

> I cannot seem to find exactly what the header would be for this product so
> I thought I'd ask the list to see if anyone else might know.

The LDM has a utility named 'notifyme' that is most useful in situations
like yours as it can:

- list out the data your LDM is receiving
- list out the data the upstream LDM you are feeding from is receiving
- do either/both of the above on a per-datastream basis
- do any of the above specifying a regular expression for the products
  you would like to get

The last feature is the capability that you should use to determine which
data you want to request.  The regular expression pattern is specified
using the '-p' flag to 'notifyme'; the regular expression might need to be
surrounded with shifted quotes so that your Unix/Linux shell does not
try to interpret them itself.

Here are some examples you can try as the user 'ldm':

1) list out the contents of the NOAAPort-populated HDS IDD datastream:

notifyme -vl- -f HDS -o 10000

   This asks to list all of the HDS products your local LDM has
   received in the past 10000 seconds (if the upstream still has
   10000 seconds of HDS data in its LDM queue) and on into the future
   (meaning that if you leave the command running, it will list new
   products as they are received).

2) You can interrogate an upstream LDM that you are ALLOWed to request
   data from in the same manner.  For instance, you could interrogate
   the toplevel IDD relay node idd.cise-nsf.gov:

notifyme -vl- -f HDS -o 10000 -h idd.cise-nsf.gov

3) Here is a refinement of the same invocation in 2) that limits the
   request to products that contain GFS in their product IDs:

notifyme -vl- -f HRS -o 10000-h idd.cise-nsf.gov -p GFS

   Here is some example output from this command run at approx. 16Z this
   morning, Saturday, March 14:

 ...

Mar 14 16:15:53 notifyme[18944] INFO:     5420 20090314160625.972     HDS 
79075417  ZRQV65 KWBC 141200 /mGFS !grib/ncep/GFS/#211/200903141200/F102/RH/650 
mb/ 
Mar 14 16:15:53 notifyme[18944] INFO:     9954 20090314160625.972     HDS 
79075418  ZVQV52 KWBC 141200 /mGFS 
!grib/ncep/GFS/#211/200903141200/F102/VGRD/525 mb/ 
Mar 14 16:15:53 notifyme[18944] INFO:     9954 20090314160625.972     HDS 
79075419  ZVQV72 KWBC 141200 /mGFS 
!grib/ncep/GFS/#211/200903141200/F102/VGRD/725 mb/ 

 ...

Notice that the product IDs (the last pieces of each line listed) contain
the grid that the field is on (#211 -> grid 211 in all three lines above).

So, if you are looking to use the global 1.25 degree thinned GFS model data in
the HRS datastream, then you will need to look for fields on the following
grids:

NH=#037, #038, #039, #040
SH=#041, #042, #043, #044

  NB: a good reference for the grid numbers for the various grids used in
  NCEP models is the UCAR/COMET MetEd site:

  http://www.meted.ucar.edu/nwp/pcu2/SBNgrids/sbn2.htm

  You have to register as a MetEd user (free) to view this page, however.

The approach taken by most, if not all, display/analysis packages is to
combined the 8 octets of thinned grids into a global grid.

Here is a 'notifyme' invocation that will list out the fields on any
of the GFS thinned grid octets broadcast in NOAAPort and available in
the IDD HDS datastream:

notifyme -vl- -f HDS -o 1000000 -p "GFS/#0(37|38|39|40|41|42|43|44)" -h 
idd.cise-nsf.gov

The regular expression pattern in this example, 
"GFS/#0(37|38|39|40|41|42|43|44)", can
be written more compactly as: "GFS/#0(3[789]|4[01234])".  The resultant 
'notifyme'
invocation would then be:

notifyme -vl- -f HDS -o 1000000 -p "GFS/#0(3[789]|4[01234])" -h idd.cise-nsf.gov

So, if this is the data you want, you would use this regular expression to
request it from your in-house NOAAPort reception system or from an upstream
IDD relay node:

Example ~ldm/etc/ldmd.conf REQUEST entry using the above regular expression
for idd.cise-nsf.gov (replace the machine with an appropriate upstream
feed site):

request HDS "GFS/#0(3[789]|4[01234])" idd.cise-nsf.gov

Remember:  whenever you make a change in the LDM configuration file,
~ldm/etc/ldmd.conf, you will need to stop and restart your LDM to
make the change active.

> At this point I'd just like to file the data to raw grib files on the
> disk. This is a sample of what I was going to put in my pqact.conf.
> I'm open to suggestions and corrections on this too.
> 
> CONDUIT prod/gfs.*pgrb2
>     FILE    data/grib/gfs/(\1:yyyy)(\1:mm)\1\2_gfs230.grib2

Some comments/questions on your example pattern-action file entry:

- your output file expression makes use of the '\1', and '\2'
  patterns, but your regular expression 'prod/gfs.*pgrb2' does
  not identify these fields (meaning that you do not have any
  pieces of your regular expression that are surrounded with
  parentheses).  This action will, therefore, fail.

- the product IDs for the model data in the NOAAPort-populated
  HDS and NGRID IDD datastreams and the NCEP model-populated
  CONDUIT datastreams contain date information.  You do not,
  therefore, have to use the \1:yyyy and \1:mm constructs to
  synthesize the year and month for the data; you can simply
  extract the year, month, etc. from the product IDs themselves.

- are you sure that you want a grid from the CONDUIT feed

We strongly believe that learning how to use the LDM 'notifyme' utility
will greatly ease your job of determining what data you are getting; what
data you would like to get; what data your upstream feed host(s) have;
fashioning ldmd.conf REQUEST entries; and fashioning pattern-action
file entries.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: MAD-323242
Department: Support CONDUIT
Priority: Normal
Status: Closed