[LDM #JCB-153899]: idd.cise-nsf.gov to *.fis.ua.pt failure

Hi Yoshihiro,

re:
> Although the statistics seems indicating that both, the
> atm77.fis.ua.pt and atm78.fis.ua.pt, computers are
> receiving data we are actually do not have anything in our
> files.

OK.

> Could you please check the tar file (in anex), with
> ldmd.conf and pqact.conf, to see if there is something
> wrong ?

Ah Ha!  We see problem:  it looks like you did not see the announcement
of a change to the CONDUIT datasetream products and naming convention
back in August.  Here is a recent post to the 'conduit' email list
that includes the August announcement:

  Date:    Wed, 18 Oct 2006 11:33:39 MDT
  To:      address@hidden
  From:    Steve Chiswell <address@hidden>
  Subject: 20061018: Changes to CONDUIT data stream

  CONDUIT recipients,

  At 18Z, the remaining CONDUIT files (NAM, RUC, and GFS) still being
  inserted using the NWSTG naming convention will be transitioned to the
  NCEP file names:

  http://www.unidata.ucar.edu/data/conduit/ldm_idd/gfs_files.html
  http://www.unidata.ucar.edu/data/conduit/ldm_idd/ruc2_files.html
  http://www.unidata.ucar.edu/data/conduit/ldm_idd/nam_files.html

  As always, please contact address@hidden should
  you have any problems with your receipt of these products,

  --
  Steve Chiswell <address@hidden>
  Unidata


  -----------------------------------------------------------------------
  CONDUIT Users,

  On August 9, additional data sets were added to the CONDUIT data stream
  originating from the new top level LDM server which accesses data sets
  using the ftpprd.ncep.noaa.gov file naming. On September 27 and October
  17, the remaining data sets on CONDUIT will be transitioned to the new LDM
  server as described below. The migration of these data sets to the new server
  provides additional redundancy in the hardware used to insert the products,
  as well as relieving dependencies on the AFS file system.

  On September 27, the ENSEMBLE data sets being delivered by CONDUIT will
  transition from the tgftp.nws.noaa.gov files to the GRIB2 files found
  under:

  ftpprd.ncep.noaa.gov/data/nccf/com/gens/prod/gefs.YYYYMMDD/HH/pgrb2a/

  The global ensembles currently comprise 10 members. In the near future,
  NCEP will increase the number of members to 14, as well as provide additional
  parameters. Sample LDM patterns for ensemble data can be found at:
  http://www.unidata.ucar.edu/data/conduit/ldm_idd/ensemble_files.html#GRIB2

  On October 17, the remaining NAM, RUC and GFS files already distributed by
  CONDUIT will change origination to the NCEP file names.

  Please send any questions you may have to
  address@hidden.

So, a number of your ~ldm/etc/ldmd.conf request lines are incorrect, and the
single ~ldm/etc/pqact.conf line you have for processing the data is incorrect:

your current ldmd.conf request lines are:

# Requests para o GFS de 0.5 grau - novos nomes
#
REQUEST CONDUIT "prod/gfs.*pgrb2.*[05]$" idd.cise-nsf.gov    PRIMARY
REQUEST CONDUIT "prod/gfs.*pgrb2.*[16]$" idd.cise-nsf.gov    PRIMARY
REQUEST CONDUIT "prod/gfs.*pgrb2.*[27]$" idd.cise-nsf.gov    PRIMARY
REQUEST CONDUIT "prod/gfs.*pgrb2.*[38]$" idd.cise-nsf.gov    PRIMARY
REQUEST CONDUIT "prod/gfs.*pgrb2.*[49]$" idd.cise-nsf.gov    PRIMARY
#
REQUEST CONDUIT "prod/gfs.*pgrb2.*[05]$" idd.unidata.ucar.edu    PRIMARY
REQUEST CONDUIT "prod/gfs.*pgrb2.*[16]$" idd.unidata.ucar.edu    PRIMARY
REQUEST CONDUIT "prod/gfs.*pgrb2.*[27]$" idd.unidata.ucar.edu    PRIMARY
REQUEST CONDUIT "prod/gfs.*pgrb2.*[38]$" idd.unidata.ucar.edu    PRIMARY
REQUEST CONDUIT "prod/gfs.*pgrb2.*[49]$" idd.unidata.ucar.edu    PRIMARY

The above requests will get the Grid #004, 0.5x0.5 degree lat/lon grids.

#GRIB 1
REQUEST CONDUIT "(MT.gfs_CY.(00|06|12|18).*[05]$)" idd.cise-nsf.gov     PRIMARY
REQUEST CONDUIT "(MT.gfs_CY.(00|06|12|18).*[16]$)" idd.cise-nsf.gov     PRIMARY
REQUEST CONDUIT "(MT.gfs_CY.(00|06|12|18).*[27]$)" idd.cise-nsf.gov     PRIMARY
REQUEST CONDUIT "(MT.gfs_CY.(00|06|12|18).*[38]$)" idd.cise-nsf.gov     PRIMARY
REQUEST CONDUIT "(MT.gfs_CY.(00|06|12|18).*[49]$)" idd.cise-nsf.gov     PRIMARY

REQUEST CONDUIT "(MT.gfs_CY.(00|06|12|18).*[05]$)" idd.unidata.ucar.edu PRIMARY
REQUEST CONDUIT "(MT.gfs_CY.(00|06|12|18).*[16]$)" idd.unidata.ucar.edu PRIMARY
REQUEST CONDUIT "(MT.gfs_CY.(00|06|12|18).*[27]$)" idd.unidata.ucar.edu PRIMARY
REQUEST CONDUIT "(MT.gfs_CY.(00|06|12|18).*[38]$)" idd.unidata.ucar.edu PRIMARY
REQUEST CONDUIT "(MT.gfs_CY.(00|06|12|18).*[49]$)" idd.unidata.ucar.edu PRIMARY

The above requests were for the Grid #003, 1x1 degree lat/lon grids and
the Grid #002, 2.5x2.5 degree lat/lon grids.  How these were renamed is detailed
in the GFS webpage that Chiz referenced above:

http://www.unidata.ucar.edu/data/conduit/ldm_idd/gfs_files.html

Your current pqact.conf line for processing the data is:

#---------------email--------11oct2004
CONDUIT (MT.gfs*) !(.*)!
        FILE    /data/conduit/\1

Since the GRIB 1 grids were renamed, this action will do nothing.

You should change your GRIB 1 request lines to look more like the ones you
have for GRIB 2 data:

#GRIB 1
REQUEST CONDUIT "(prod/gfs.*pgrb[^2].(00|06|12|18).*[05]$)" idd.cise-nsf.gov    
 PRIMARY
REQUEST CONDUIT "(prod/gfs.*pgrb[^2].(00|06|12|18).*[16]$)" idd.cise-nsf.gov    
 PRIMARY
REQUEST CONDUIT "(prod/gfs.*pgrb[^2].(00|06|12|18).*[27]$)" idd.cise-nsf.gov    
 PRIMARY
REQUEST CONDUIT "(prod/gfs.*pgrb[^2].(00|06|12|18).*[38]$)" idd.cise-nsf.gov    
 PRIMARY
REQUEST CONDUIT "(prod/gfs.*pgrb[^2].(00|06|12|18).*[49]$)" idd.cise-nsf.gov    
 PRIMARY

REQUEST CONDUIT "(prod/gfs.*pgrb[^2].(00|06|12|18).*[05]$)" 
idd.unidata.ucar.edu PRIMARY
REQUEST CONDUIT "(prod/gfs.*pgrb[^2].(00|06|12|18).*[16]$)" 
idd.unidata.ucar.edu PRIMARY
REQUEST CONDUIT "(prod/gfs.*pgrb[^2].(00|06|12|18).*[27]$)" 
idd.unidata.ucar.edu PRIMARY
REQUEST CONDUIT "(prod/gfs.*pgrb[^2].(00|06|12|18).*[38]$)" 
idd.unidata.ucar.edu PRIMARY
REQUEST CONDUIT "(prod/gfs.*pgrb[^2].(00|06|12|18).*[49]$)" 
idd.unidata.ucar.edu PRIMARY

You must also change your pqact.conf processing line to reflect the GRIB 1 name 
change:

CONDUIT (prod/gfs.*pgrb[^2]) !(.*)!
        FILE    /data/conduit/\1

Finally, the best way to test out regular expressions used for ldmd.conf 
requests
and pqact.conf processing is to use the LDM 'notifyme' facility.  For instance,
on atm77.fis.ua.pt, you can run 'notifyme' to tell you what CONDUIT data your 
upstream
feed host is receiving:

<as 'ldm'>
notifyme -vxl- -f CONDUIT -o 10000 -h idd.cise-nsf.gov
 -- and/or --
notifyme -vxl- -f CONDUIT -o 10000 -h idd.unidata.ucar.edu

If the data you are interested in is being received at the time you run this
'notifyme' invocation, you will see a listing of the products being received.
This listing includes the product headers that your regular expressions are
being matched against:

notifyme -vxl- -f CONDUIT -o 10000 -h idd.unidata.ucar.edu
Oct 24 22:30:17 notifyme[16325] NOTE: Starting Up: idd.unidata.ucar.edu: 
20061024194337.082 TS_ENDT {{CONDUIT,  ".*"}}
Oct 24 22:30:17 notifyme[16325] NOTE: LDM-5 desired product-class: 
20061024194337.082 TS_ENDT {{CONDUIT,  ".*"}}
Oct 24 22:30:17 notifyme[16325] INFO: Resolving idd.unidata.ucar.edu to 
128.117.140.3 took 0.000479 seconds
Oct 24 22:30:17 DEBUG: NOTIFYME(idd.unidata.ucar.edu) returns OK
Oct 24 22:30:17 notifyme[16325] NOTE: NOTIFYME(idd.unidata.ucar.edu): OK
Oct 24 22:30:18 notifyme[16325] INFO: e4e179387665f9316a55bc17b1f10ceb    85461 
20061024204503.220 CONDUIT 002  
data/nccf/com/nam/prod/nam.20061024/nam.t18z.awip1242.tm00.grib2 
!grib2/ncep/NAM_84/#000/200610241800F042/VSBY/0 - NONE! 000002
Oct 24 22:30:18 notifyme[16325] INFO: b8807b696f9b850a002ab946feacf37e    82543 
20061024204503.230 CONDUIT 002  
data/nccf/com/nam/prod/nam.20061024/nam.t18z.awip1239.tm00.grib2 
!grib2/ncep/NAM_84/#000/200610241800F039/VSBY/0 - NONE! 000002
Oct 24 22:30:18 notifyme[16325] INFO: 9fcce8ea7beb231c13593967ce642a7e    20976 
20061024204503.258 CONDUIT 002  
data/nccf/com/nam/prod/nam.20061024/nam.t18z.awip3d39.tm00 
!grib/ncep/NAM_84/#212/200610241800/F039/VIS/sfc! 000002
Oct 24 22:30:18 notifyme[16325] INFO: 39c3a29cd4580b53c7fb607b4927204b    78527 
20061024204503.261 CONDUIT 007  
data/nccf/com/nam/prod/nam.20061024/nam.t18z.awip1242.tm00.grib2 
!grib2/ncep/NAM_84/#000/200610241800F042/AVOR/1000 Pa PRES! 000007
Oct 24 22:30:18 notifyme[16325] INFO: a697357637a9ad3768607c3aafe39b63    78828 
20061024204503.264 CONDUIT 007  
data/nccf/com/nam/prod/nam.20061024/nam.t18z.awip1239.tm00.grib2 
!grib2/ncep/NAM_84/#000/200610241800F039/AVOR/1000 Pa PRES! 000007
Oct 24 22:30:18 notifyme[16325] INFO: c3b55ad6781d99541eecc2b1cd0ef973    17992 
20061024204503.264 CONDUIT 007  
data/nccf/com/nam/prod/nam.20061024/nam.t18z.awip3d39.tm00 
!grib/ncep/NAM_84/#212/200610241800/F039/ABSV/1000_mb! 000007
 ...

Look through the product headers and come up with a pattern you want to test:

notifyme -vxl- -f CONDUIT -o 10000 -h idd.unidata.ucar.edu -p 
"prod/gfs.*pgrb[^2]"
Oct 24 22:31:51 notifyme[16345] NOTE: Starting Up: idd.unidata.ucar.edu: 
20061024194511.528 TS_ENDT {{CONDUIT,  "prod/gfs.*pgrb[^2]"}}
Oct 24 22:31:51 notifyme[16345] NOTE: LDM-5 desired product-class: 
20061024194511.528 TS_ENDT {{CONDUIT,  "prod/gfs.*pgrb[^2]"}}
Oct 24 22:31:51 notifyme[16345] INFO: Resolving idd.unidata.ucar.edu to 
128.117.140.3 took 0.000495 seconds
Oct 24 22:31:51 DEBUG: NOTIFYME(idd.unidata.ucar.edu) returns OK
Oct 24 22:31:51 notifyme[16345] NOTE: NOTIFYME(idd.unidata.ucar.edu): OK
Oct 24 22:31:53 notifyme[16345] INFO: 4160003c9c9c7be697f58f9a1105abd8   114114 
20061024212703.340 CONDUIT 000  
data/nccf/com/gfs/prod/gfs.2006102418/gfs.t18z.pgrbanl 
!grib/ncep/SSIGFS/#003/200610241800/F000/HGT/1000_mb! 000000
Oct 24 22:31:53 notifyme[16345] INFO: 20b52e62f2f98c4735041a193d26ddc8   114114 
20061024212703.492 CONDUIT 009  
data/nccf/com/gfs/prod/gfs.2006102418/gfs.t18z.pgrbanl 
!grib/ncep/SSIGFS/#003/200610241800/F000/HGT/650_mb! 000009
Oct 24 22:31:54 notifyme[16345] INFO: 882d7fb54c657aa5ddb4ae10ec745f58   114114 
20061024212703.495 CONDUIT 010  
data/nccf/com/gfs/prod/gfs.2006102418/gfs.t18z.pgrbanl 
!grib/ncep/SSIGFS/#003/200610241800/F000/HGT/600_mb! 000010
Oct 24 22:31:54 notifyme[16345] INFO: 66f063bcfbb5ed311133a2be79e89e14   122258 
20061024212703.574 CONDUIT 019  
data/nccf/com/gfs/prod/gfs.2006102418/gfs.t18z.pgrbanl 
!grib/ncep/SSIGFS/#003/200610241800/F000/HGT/150_mb! 000019
Oct 24 22:31:54 notifyme[16345] INFO: b98e1d80fa6e826a5d80e56f153cf21a   114114 
20061024212703.406 CONDUIT 004  
data/nccf/com/gfs/prod/gfs.2006102418/gfs.t18z.pgrbanl 
!grib/ncep/SSIGFS/#003/200610241800/F000/HGT/900_mb! 000004
Oct 24 22:31:54 notifyme[16345] INFO: 0f5e256a1ceb9a02a5aa1fe846f2acd1   122258 
20061024212703.577 CONDUIT 020  
data/nccf/com/gfs/prod/gfs.2006102418/gfs.t18z.pgrbanl 
!grib/ncep/SSIGFS/#003/200610241800/F000/HGT/100_mb! 000020
Oct 24 22:31:54 notifyme[16345] INFO: 7847355c1894bf59023164c6f3819bba    81534 
20061024212703.689 CONDUIT 029  
data/nccf/com/gfs/prod/gfs.2006102418/gfs.t18z.pgrbanl 
!grib/ncep/SSIGFS/#003/200610241800/F000/TMP/925_mb! 000029
Oct 24 22:31:54 notifyme[16345] INFO: c15ad7edfbdab5c275865a9f883d14b9   114114 
20061024212703.409 CONDUIT 005  
data/nccf/com/gfs/prod/gfs.2006102418/gfs.t18z.pgrbanl
 ...

Play around with the regular expression to list exactly the set of products you 
are
interested in and then use that regular expression in your ldmd.conf request 
lines.

You can use the same 'notifyme' procedure to test regular expressions you want 
to use
in your pqact.conf action(s).

Please let us know if the above was clear enough, and that you are once again 
getting
and processing the data you want.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: JCB-153899
Department: Support LDM
Priority: Normal
Status: Closed