[IDD #WFA-619667]: Request for CONDUIT feed

Hi Martha,

re:
> Thanks,Tom, for your patience with us...

No worries.

> I am attaching the latest versions of our pqact.conf for decoding/filing
> with XCD, pqact.thredds.tasc for creating the thredds data structure (which
> is working in most cases), and ldmd.conf.

OK, thanks.

> I remembered you writing something about adding CONDUIT to our pqact.conf
> file to handle the binary CONDUIT data, so did that but with no luck even
> though we are definitely receiving the data as shown by notifyme, but we
> are still not capturing the GEFS data.

When I looked at the pqact.thredds.tasc.txt file you sent I found several
instances of spaces being used where tabs are required.  I have attached
my edited file to this reply, but I am not sure that you will be able to
use the edited version directly as I edited it using 'vi' in a Cygwin shell
on my Windows XP box here at home.  (I am unsure about whether or not the
file will have *nix style newlines or Windows style newlines.)

The changes I made were:

- use a tab (and _no_ spaces) as the first character of each continuation
  line in an action.  This includes a tab before each FILE and a tab
  before those lines where the data file specification was included on
  a line following the FILE action.

  Here is one example:

CONDUIT    /data/nccf/com/gens/prod/gefs\.(........)/(..)/pgrb2a
    FILE   
/data/pub/native/grib/NCEP/GFS/Global_1p0deg_Ensemble/GFS_Global_1p0deg_Ensemble_\2_\100.grib2

  Tabs (no spaces) are _required_:

  - between CONDUIT and /data/nccf/com/...
  - before FILE
  - before /data/pub/native/grib/...

  NB:

  - if the file name expression is on the same line as FILE, then the whitespce
    separating FILE from the file name expression must be a tab

  - if the file name expression is on the next line after FILE, the line
    must begin with a tab

Since some of the errors I saw were before your CONDUIT action, the CONDUIT 
action
would never be used -- processing stops in the pqact.conf.xxx file at the first
error.
 
> Also, we continue to attempt to get SREF  to concatenate into one file,
> and pqact.thredds.tasc contains our attempts at doing that as well as doing
> the same for the GEFS from CONDUIT.

The action I see to process SREF is after the action for CONDUIT.  Again, 
neither
of these actions would have been used because of errors earlier in the file.

> Note that the ldmd.log doesn't show any syntax errors.

The 'ldmadmin pqactcheck' action (which is run automatically for you when
starting the LDM in LDM-6.7+) is a _gross_ syntax checker.  Evidently, it
was not picking up the errors I saw in your pqact.thredds.tasc.txt file.

Question:

- what editor are you using to edit your pattern-action files?

Comment:

- I always use 'vi' since it is so easy to find occurrances of any character
  including tabs.  If you are using some other editor, it may or may not be
  difficult to spot where spaces are inadvertently being used instead of tabs.

I also see that your 'exec' line that specifies the use of pqact.thredds.tasc
explicitly specifies to _NOT_ process CONDUIT data:

exec    "pqact -f ANY-CONDUIT-NNEXRAD-CRAFT 
/usr/local/ldm/etc/pqact.thredds.tasc

The "ANY-CONDUIT-NNEXRAD-CRAFT" statement says to send data from ANY feed
_except_ CONDUIT, NNEXRAD, and CRAFT to the pattern-action file
/usr/local/ldm/etc/pqact.thredds.tasc.

If the reason that the pqact.thredds.tasc.txt file you sent had syntax errors
was that whatever turned pqact.thredds.tasc into pqact.thredds.tasc.txt changed
some tabs into spaces, then the cause of your problem may be the incorrect
exclusion of CONDUIT in the list of datafeeds to process in the ldmd.conf exec
line. I have attached an edited version of ldmd.conf.txt for your review.

> Hope your presentation goes well, and thanks as always for your help.

The presentation went very well, thanks.  I did not present; I was there
to answer questions regarding use of Unidata technologies like THREDDS,
RAMADDA, and the IDV if they came up.


Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: WFA-619667
Department: Support IDD
Priority: Normal
Status: Closed

# GFS/AVN grid output
# Grid #201 N. Hemisphere 381km: ^Y.A... KWBC.*/m(AVN|GFS|SSIAVN|SSIGFS)
# Grid #202 CONUS         191km: ^Y.I... KWBC.*/m(AVN|GFS|SSIAVN|SSIGFS)
# Grid #203 Alaska        191km: ^Y.J... KWBC.*/m(AVN|GFS|SSIAVN|SSIGFS)
# Grid #211 CONUS          80km: ^Y.Q... KWBC.*/m(AVN|GFS|SSIAVN|SSIGFS)
# Grid #213 CONUS          95km: ^Y.H... KWBC.*/m(AVN|GFS|SSIAVN|SSIGFS)
# Select any/all grids desired from [QAIJH]
#
HRS     ^[YZ].(A)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(AVN|GFS|SSIAVN|SSIGFS)
        FILE
        
/data/pub/native/grid/NCEP/GFS/N_Hemisphere_381km/GFS_N_Hemisphere_381km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(I)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(AVN|GFS|SSIAVN|SSIGFS)
        FILE
        
/data/pub/native/grid/NCEP/GFS/CONUS_191km/GFS_CONUS_191km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(Q)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(AVN|GFS|SSIAVN|SSIGFS)
        FILE
        
/data/pub/native/grid/NCEP/GFS/CONUS_80km/GFS_CONUS_80km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(H)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(AVN|GFS|SSIAVN|SSIGFS)
        FILE
        
/data/pub/native/grid/NCEP/GFS/CONUS_95km/GFS_CONUS_95km_(\2:yyyy)(\2:mm)\2_\300.grib1
#
# RADAR grid products
#
HRS     ^HAXA00 K... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
/data/pub/native/radar/10km_mosaic/RADAR_10km_mosaic_(\1:yyyy)(\1:mm)\1_\2\3.grib1
#        
/data/pub/native/radar/10km_mosaic/(\1:yyyy)(\1:mm)\1/RADAR_10km_mosaic_(\1:yyyy)(\1:mm)\1_\2\3.grib1
#
#NPORT
NPORT   ^[LM].E... KWBD ([0-3][0-9])([0-2][0-9])([0-6][0-9])
        FILE    
/data/pub/native/grid/NCEP/DGEX/CONUS_12km/DGEX_CONUS_12km_(\1:yyyy)(\1:mm)\1_\200.grib2
NPORT   ^[LM].B... KWBE ([0-3][0-9])([0-2][0-9])([0-6][0-9])
        FILE    
/data/pub/native/grid/NCEP/NAM/CONUS_12km/NAM_CONUS_12km_(\1:yyyy)(\1:mm)\1_\200.grib2
#
#ETA/NAM Grids
HRS     ^[YZ].(U)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(ETA|NAM)
        FILE
        
/data/pub/native/grid/NCEP/NAM/CONUS_20km/noaaport/NAM_CONUS_20km_noaaport_(\2:yyyy)(\2:mm)\2_\300.grib1
#
HRS     ^[YZ].(Q)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(ETA|NAM)
        FILE
        
/data/pub/native/grid/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_(\2:yyyy)(\2:mm)\2_\300.grib1
#
# RUC2 grid on Lambert conformal CONUS grid
HRS     ^Y.W... KWB. (..)(..).*/mRUC2
        FILE
        
/data/pub/native/grid/NCEP/RUC2/CONUS_40km/RUC2_CONUS_40km_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# RUC grid output
#
HRS     ^[YZ].(Q)... KWB. ([0-3][0-9])([0-2][0-9]).*/mRUC
        FILE
        
/data/pub/native/grid/NCEP/RUC/CONUS_80km/RUC_CONUS_80km_(\2:yyyy)(\2:mm)\2_\300.grib1
#Ensemble members
HRS     ^H.[I-P]... KWBK ([0-3][0-9])([0-2][0-9])
        FILE
        
/data/pub/native/grid/NCEP/GFS/Ensemble_1p25deg/GFS_Ensemble_1p25deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#  GFS grid - Extended range grids E and F
#
HRS     ^H.[EF][A-Z][0-9][0-9] KWB. ([0-3][0-9])([0-2][0-9])..
        FILE
        
/data/pub/native/grid/NCEP/GFS/Extended_Global_5p0deg/GFS_Extended_Global_5p0deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# ECMWF grid on global 2.5 x 2.5 degree grid
HRS     ^H.[A-L]... ECMF (..)(..)
        FILE
        
/data/pub/native/grid/ECMWF/Global_2p5/ECMWF_Global_2p5_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# Global UKMet thinned grids
HRS     ^H.[I-P]... EGRR ([0-3][0-9])([0-2][0-9])
        FILE
        /data/pub/native/grid/UKMET/thin/UKMET_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# CMC GEM grid data
CMC     
CMC_GEM_reg_.*_.*_([12][0-9][0-9][0-9])([0-1][0-9])([0-3][0-9])([0-2][0-9])_(.*).grib
        FILE    /data/pub/native/grid/CMC/GEM/GEM_\1\2\3_\400.grib1
#
# FNMOC data available via IDD (NOGAPS)
# -------------------------
FNMOC   ^US058....-.......(0058)_...._.........(....)(..)(..)(..)_(.*)
        FILE    /data/pub/native/grid/FNMOC/NOGAPS/NOGAPS_\2\3\4_\5_\1_\6
#
# CONDUIT Global_1p0deg_Ensemble
CONDUIT /data/nccf/com/gens/prod/gefs\.(........)/(..)/pgrb2a
        FILE    
/data/pub/native/grid/NCEP/GFS/Global_1p0deg_Ensemble/GFS_Global_1p0deg_Ensemble_\2_\100.grib2
#
#SREF 
NGRID   ^[LM].R... KWBL ([0-3][0-9])([0-2][0-9])..
        FILE    /data/pub/native/grid/NCEP/SREF/sref_YYYYMMDDHH_us.grib2

#####
# $Id: ldmd.conf,v 1.18 2003/02/27 17:12:57 steve Exp $
# Sample ldmd.conf for ldm5
####
#
# This is the main configuration file for the LDM server. All lines that start
# with a "#" sign are comments.
#
# To debug an LDM that hangs on start up, run the following from LDM home:
# % bin/rpc.ldmd -vl - -q data/ldm.pq etc/ldmd.conf
#
# If the LDM still hangs, comment out all lines in this file except noted
# allow line entry below, try again.
#
###############################################################################
# WARNING: Pathological Product-ID Patterns
###############################################################################
#
# The REQUEST and ACCEPT entries both have product-ID patterns that are regular
# expressions (RE-s).  It is EXTREMELY IMPORTANT that these patterns not have a
# ".*" prefix because: 1) such a prefix accomplishes nothing (i.e., the set of
# possible matches is unaltered by the prefix); and 2) such a prefix causes the 
# RE to take MUCH, MUCH LONGER TO EXECUTE (by 3 orders of magnitude in many
# cases).  Therefore, DO NOT USE RE-s like
#
#     ".*UFPBA"
#
# Instead, simply use
#
#     "UFPBA"
#
# The regex(1) utility of this package can be used to examine the efficiency
# of an RE.
#
###############################################################################
# Exec Entries
###############################################################################
#
# Programs that can share a queue with rpc.ldmd server, these programs
# are started by the "exec" command and are in the same process group.
#
# Regarding pqbinstats, while pqbinstats will generate .stats files in
# your logs directory, it is necessary to also run 'ldmadmin dostats'
# once per hour via cron in order to email the stats to the UPC and
# remove the .stats files that are no longer needed.  See
# http://www.unidata.ucar.edu/packages/ldm/ldmConfigInstallList.html#s7
# for more information.
exec    "pqact"
exec    "xcd_run MONITOR"
#exec   "pqbinstats"
exec    "pqact -f ANY-NNEXRAD-CRAFT /usr/local/ldm/etc/pqact.thredds.tasc"
#
# rtstats uses the LDM to send product latency statistics to the UPC.
# We are currently evolving from pqbinstats to rtstats, but in the
# mean time we hope sites will run both.  For more info on rtstats
# please see the man page.
#exec   "rtstats -h rtstats.unidata.ucar.edu"
#exec   "pqsurf"
# 
###############################################################################
# Request Entries
###############################################################################
#
# LDM5 servers request data from Data Sources
#
#       request <feedset> <pattern> <hostname pattern>
#
#request WMO ".*" uni0.unidata.ucar.edu
# Added Per MUG instructions 01.27.05 rlg
#request         PPS|DDS|HDS|IDS|NNEXRAD         ".*"    noaapnew.west.tasc.com
#request        PPS|DDS|HDS|IDS|NNEXRAD ".*"    noaapnew.west.tasc.com  PRIMARY
#request        PPS|DDS|HDS|IDS|NNEXRAD ".*"    10.2.15.86      PRIMARY
#request        PPS|DDS|HDS|IDS|NNEXRAD|FNEXRAD ".*"    noaapnew.it-protect.com 
PRIMARY
request ANY     ".*"    noaapnew.it-protect.com PRIMARY
request CONDUIT "(gefs)"        idd.cise-nsf.gov                
#
###############################################################################
# Allow Entries
###############################################################################
#
# Giving permission for a Data Sink to perform a request to your LDM
#
# allow <feedset> <hostname pattern>
#
# Giving permission to your own machine and Unidata
#
# Under no circumstances comment out the next allow entry to localhost
# The LDM will NOT start if the lines are commented out.
allow   ANY
    ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$))
#
###############################################################################
# Accept Entries
###############################################################################
# ACCEPT: Who can feed us, currently this action is ONLY needed for WSI data
#
# accept <feedset> <pattern> <hostname pattern>
#
# accept anything from yourself
#
#accept ANY ".*" ^((localhost|loopback)|(127\.0\.0\.1\.?$))
#
# accept from your upstream site
#
# WSI is using ldm4 protocol so the accept is still required
#accept WSI
#    .*
#    ^[a-z].*\.uni\.wsicorp\.com$
#
###############################################################################
# End 
###############################################################################