[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20010516: NEXRAD data (cont.)

>From: "Wayne Bresky" <address@hidden>
>Organization: ?
>Keywords: 200105161402.f4GE2Jp07535

Wayne,

I should have asked this in the last email.  What site are you from?
Your email address doesn't give me a clue.

>site is not getting any new NEXRAD data and yet the ldmd.conf
>file appears to be set up for it:
>
>request NNEXRAD         ".*"    snow.cit.cornell.edu

I hope that you realize that this action requests ALL NEXRAD products from
ALL NEXRAD sites.  This is a LOT of data.  Can your network handle the
load?  Can you store thousands of images?  Do you have scouring setup
to keep your disk from drowning?

>At one point it looks as though we may have been getting the data
>from WSI because I see another line commented out:
>
>#request        WSI             ".*"    vorticity.cit.cornell.edu

The WSI feed should have always been coming from WSI directly.  The IDD
topology for WSI products was always point-to-point (WSI-to-site).  The
NOAAPORT NEXRAD images are distributed in a fanout manner since they
are freely available.

>I just wasn't sure where I should be getting the data from, WSI, IDD
>or UCAR (via McIDAS ADDE).

To get data from WSI you would have to have a contract with them and be
paying them for the imagery/feed.  You should be able to get the NNEXRAD
feed from one of several top level sites that handles the feed.  The
alternatives to all of the images in NNEXRAD are:

o a ldmd.conf request line that does not ask for everything
o use of the NEXRAD floater feed, FNEXRAD

Your request for NNEXRAD should work if:

o the site you are requesting the feed from has the data
o the site you are requesting the feed from allows you to request the
  data

There are two quick ways of finding out if your upstream site is allowing
you to request the NNEXRAD feed:

o look through your LDM log file (~ldm/logs/ldmd.log) and see if you are
  getting 'reclass' messages indicating that your request is being
  modified

o use notifyme to the upstream site to see if you can see them getting
  the images in the feed

The second option would be run from your 'ldm' account as follows:

notifyme -vxl- -f NNEXRAD -o 3600 -h snow.cit.cornell.edu

Since all Unidata machines are enabled in site ldmd.conf files, I can
run this command and give you an idea of what the output should look like:

/local/ldm% notifyme -vxl- -f NNEXRAD -o 3600 -h snow.cit.cornell.edu
May 16 16:44:35 notifyme[13486]: Starting Up: snow.cit.cornell.edu: 
20010516154435.560 TS_ENDT {{NNEXRAD,  ".*"}}
        NOTIFYME(snow.cit.cornell.edu) returns OK
May 16 16:44:36 notifyme[13486]: NOTIFYME(snow.cit.cornell.edu): OK
May 16 16:44:37 notifyme[13486]: 0e236c9ba7002d2c589671b0faba0e25     8680 
20010516154438.593 NNEXRAD 207  SDUS53 KDTX 161543 /pN0SDTX
May 16 16:44:37 notifyme[13486]: f6a948bce8d800a6ba8f6b24a4ebae27    10641 
20010516154438.610 NNEXRAD 212  SDUS53 KUNR 161538 /pNCRUDX
May 16 16:44:37 notifyme[13486]: d07a095735de30491be9850eb7128f35     1013 
20010516154438.618 NNEXRAD 215  SDUS53 KIWX 161538 /pNVLIWX
May 16 16:44:37 notifyme[13486]: 7028a093ad43d67cd1136dd69111a012      423 
20010516154438.619 NNEXRAD 216  SDUS58 PAFC 161538 /pNVLAKC
May 16 16:44:38 notifyme[13486]: 9fbe6be5b81d5bee68c49a8c447ae207    18381 
20010516154438.631 NNEXRAD 217  SDUS54 KEWX 161542 /pN0RDFX
^CMay 16 16:44:39 notifyme[13486]: Interrupt
May 16 16:44:39 notifyme[13486]: exiting

Note a couple of things in the above:

o the NOTIFYME(snow.cit.cornell.edu) line says OK.  This tells me that I
  am 'allow'ed in snow's ldmd.conf file

o I can see snow receiving NNEXRAD products

You should run notifyme on your machine and see what happens.
If you can see snow receiving NNEXRAD data, you need to see if your machine
is receiving the data.  Again, you would use the notifyme command:

notifyme -vxl- -f NNEXRAD -o 3600

Here, the host is assumed to be local host (your machine).

>We don't have room for all of the
>data, but I would like to keep at least the data from stations in NY
>locally.

OK.  You will need to setup a tailored request line in your ldmd.conf file.
The NEXRAD sites in the state of New York are (listing from McIDAS STNLIST
comand):

STNLIST TYPE=N ST=NY
 IDN  ID   Station Name       Data Types      ST CO  LAT        LON        ELE
 ---  TYX  Fort Drum                   N      NY US   43:45:21   75:40:48   563
 ---  BGM  BINGHAMTON                  N      NY US   42:11:59   75:59:05   617
 ---  BUF  BUFFALO                     N      NY US   42:56:56   78:44:12   266
 ---  ENX  ALBANY                      N      NY US   42:35:11   74:03:50   702
 ---  OKX  NEW YORK CITY               N      NY US   40:51:56   72:51:50    32
Number of stations listed: 5
STNLIST: Done

request NNEXRAD  "/p...(BGM|BUF|ENX|OKX|TYX)"    snow.cit.cornell.edu

This will request ALL products from the stations listed (the station ID
is part of the AFOS pil that is being selected with the '/p...'
regular expression).

Of course, you can add to or remove stations from this list as needed.

>If the above looks ok,

The request is OK per se.  I am betting that you are not allowed in the
Cornell ldmd.conf file.  If you are not, you should contact the
administrator at Cornell and ask to be allowed.  You can lookup
the LDM site administrator's name online:

Unidata HomePage
http://www.unidata.ucar.edu
  Software
  http://www.unidata.ucar.edu/Software.html
    IDD
    http://www.unidata.ucar.edu/projects/idd/index.html
      Current Operational Status 
      http://www.unidata.ucar.edu/projects/idd/iddgeneral.html#status
        IDD site contact list 
        http://www.unidata.ucar.edu/projects/idd/sitelist.html

The last page is a table of IDD site contact information.

>I guess I need to look at the action file.  Would you agree?

The action for filing the data is the next thing to look at after
determining whether or not you are actually getting the data.

Your pqact.conf action for filing the products might look like:

# All products from selected sites
NNEXRAD ^SDUS5. .... ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
        FILE    -close  /data/ldm/nexrad/RADAR/\5/\4/\4_(\1:yyyy)(\1:mm)\1_\2\3

This is only one of a myriad number of possibilities.

The step after that is setting up data scouring.

There is a script that you can use for scouring in the pub/ldm
directory of anonymous FTP on our FTP server, ftp.unidata.ucar.edu:

directory: pub/ldm5
file: prune_nexrad.csh

'prune_nexrad.csh' should be FTPed to a directory in the 'ldm' account
that is in the PATH of the user 'ldm'.  We put this in ~ldm/decoders.
'prune_nexrad.csh' needs to be edited to set a couple of variables at
the top:

setenv PATH /usr/local/ldm/util:${PATH}
set KEEP=12
set LOCALKEEP=60
set LOCALSITE="FTG"


Set the PATH to match where you put prune_nexrad.csh.  The default in
'prune_nexrad.csh' is /usr/local/ldm/util.  If you put it in 
/usr/local/ldm/decoders, the PATH statement would look like:

setenv PATH /usr/local/ldm/decoders:${PATH}

KEEP needs to be set to the number of files you want to keep for each
product for each site.

LOCALKEEP needs to be set to the number of products you want to keep for
the LOCALSITE.

LOCALSITE needs to be set to a NEXRAD station ID for which you want to keep
more files than KEEP.

Finally, you need to add a crontab action in the 'ldm' account to scour
the files.  Here is an example:

#
# prune nexrad tree
#
15 * * * * decoders/prune_nexrad.csh /data/ldm/nexrad/RADAR >/dev/null 2>&1

Note how the first argument to 'prune_nexrad.csh' in the crontab entry
matches the top level directory under which you will be storing the
NEXRAD images.

After ingestion and scouring are setup, you need to setup ADDE access
to the images if you are using McIDAS.  More on that later if you like.

Tom