20040202: Western Michigan Univaersity - Request for NEXRAD level II Data

>From: Karthik Samudram Jayaraman <address@hidden>
>Organization: Western Michigan University
>Keywords: 200402021728.i12HSip1008370 IDD CRAFT

Karthik,

>This is Karthik From Western Michigan University. We need the NEXRAD 
>Level II data to be ingested to our server S400.wood.wmich.edu, 
>which currently is not been ingested.
>
>Could you please guide me in how to proceed with the same?

There are three things involved in getting any data through the IDD:

1) getting access to the appropriate datastream and then setting up
   the request line in the LDM ldmd.conf file (located in
   /cutrim1/ldm/etc/ldmd.conf on s400)

2) decoding the data into a directory/directories where there is enough
   room

3) scouring the data so that your disk does not fill up

Here is the information you need for the NEXRAD Level II data:

1) you can request the Level II feed from thelma.ucar.edu:

   LDM upsteam server:  thelma.ucar.edu
   IDD feedname:        CRAFT

   You should decide which radars you want the Level II data from, because
   there is a LOT of data in the CRAFT feed.

   I setup the /cutrim1/ldm/etc/ldmd.conf request line for the radars
   in Michigan (GRR, APX, DTX) and those immediately to the south
   (IWX and LOT) and west (MKX, GRB, MQT).

2) the Level II data comes in chunks of 100 radials each of which are
   compressed using bzip2.  The simpliest way to decode these is to use
   the GEMPAK decoder, dcnexr2.  Since decoding these data puts a
   high demand on a systems CPU, we found that it is best to run this
   decoder from its own ldmd.conf action.  For s400, the action would
   look like:

exec    "pqact -f CRAFT /cutrim1/ldm/etc/pqact.conf_craft"

   I setup this action in the /cutrim1/ldm/etc/ldmd.conf file.

   The contents of pqact.conf_craft would look like:

# CRAFT
CRAFT   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9][0-9])
        PIPE    decoders/dcnexr2
        -s \1 -d /dev/null /cutrim1/ldm/data/craft/\1/\1_\2_\3

   WARNING:  white space in pqact.conf actions may be tabs, not spaces.
   In this example, tabs are used:

   - between CRAFT and ^L2-BZIP2
   - before PIPE
   - between PIPE and decoders/dcnexr2
   - before -s \1...

   IMPORTANT: No line in pqact.conf* should begin with a space.

   I setup /cutrim1/ldm/etc/pqact.conf_craft for you.

3) we scour Level II NEXRAD data using the LDM 'scour' utility run from
   cron.  To do this, 'scour' should be run at least once per day. 
   Here is an example that runs 'scour' multiple times per day:

#
# Scour data directory
0 1,4,7,10,13,16,19,22 * * * bin/ldmadmin scour > /dev/null 2>&1

   Scouring was already setup on s400 exactly as in this example.

   To configure 'scour' (which is run by 'bin/ldmadmin scour'), one
   must setup entries in the etc/scour.conf file (located in
   /cutrim1/ldm/etc/scour.conf on s400).  Here is the appropriate
   entry for scouring Level II data being saved in the /cutrim1/ldm/data/craft
   directory hierarchy:

/cutrim1/ldm/data/craft                 1

   I setup this entry so that the Level II data should get scoured assuming
   that the cron jobs contiue to run.

>Thanks

You must keep an eye on the ingest/decode/scour setup until you are
certain that it is working correctly.  Otherwise, you could easily
fill your disk which would result in you no longer ingesting and
decoding data.

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.