20010302: viewing ncdc radar data with McIDAS (cont.)

>From: alan anderson <address@hidden>
>Organization: St. Cloud State
>Keywords: 200102202133.f1KLXZL10511 McIDAS NEXRAD display

Alan,

>A few more questions on viewing the radar data.
>
>I am currently building ver. 7.7 on waldo, using a copy of 
>McIDAS that I downloaded Feb. 28.  mcinstall showed no errors 
>and now build seem to be going fine.  I think I understand the
>comments you sent me a day or 2 back about this upgrade, so 
>do not have any questions at this point on upgrade.

OK.  I have a response to your second email at the end of this.

>I want to look ahead a little, and ask about the radar files.
>For the moment, I am more concerned about looking at the stuff
>we are getting from NCDC than current radar.

OK, fair enough.

>The original file(s) as ftp'd from NCDC come with a title such as
>
>    KMPX_200004190000_200004200000.tar.gz
>
>After unpacking, it leaves a sequence of files, looks like several 
>per hour with names such as 
>
>    KMPX20000416_1034_00.580
>    KMPX20000415_2205_00.580               KMPX20000416_1052_00.580
>    KMPX20000415_2222_00.580               KMPX20000416_1121_00.580
>    KMPX20000415_2240_00.580               KMPX20000416_1138_00.580

Are these files all for the same station?  Are all the same kind of
product (e.g., Base Reflectivity at Tilt 1, Vertically Integrated
Liquid water, etc.)?

>I am assuming I will put these in a subdirectory on waldo, something
>like  /NCDC/RADAR,  and add them as an ADDE dataset which other 
>terminals can access, but would like some guidance.  I would probably 
>think of the tutorial data listed as BLIZZARD  as an analogy.

The radar data is a little different that the data in the BLIZZARD
dataset in that the radar data is not stored in a McIDAS specific
format (e.g., AREA), nor are they named following McIDAS standard
naming conventions (e.g., AREAnnnn, etc.).

>Can you give me a boost, and comment on whether the above file names
>seem usable?

Sure.  What I would do is use the example contained in the 7.704 version
of DSSERVE.BAT.  In particular, I would extract the section related to
the dataset RTNEXRAD and put it into a separate BATCH file.  The section
I am referring to is:

REM
REM NOAAPORT NEXRAD Level III products
REM

DSSERVE ADD RTNEXRAD/N0R          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Base Reflectivity Tilt 1
DSSERVE ADD RTNEXRAD/N1R          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Base Reflectivity Tilt 2
DSSERVE ADD RTNEXRAD/N2R          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Base Reflectivity Tilt 3
DSSERVE ADD RTNEXRAD/N3R          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Base Reflectivity Tilt 4
DSSERVE ADD RTNEXRAD/N0V          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Radial Velocity Tilt 1
DSSERVE ADD RTNEXRAD/N1V          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Radial Velocity Tilt 2
DSSERVE ADD RTNEXRAD/NCR          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Composite Reflectivity
DSSERVE ADD RTNEXRAD/N0S          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Storm-Rel Mean Vel Tilt 1
DSSERVE ADD RTNEXRAD/N1S          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Storm-Rel Mean Vel Tilt 2
DSSERVE ADD RTNEXRAD/N2S          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Storm-Rel Mean Vel Tilt 3
DSSERVE ADD RTNEXRAD/NVL          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Vertical Liquid H2O
DSSERVE ADD RTNEXRAD/NTP          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Storm Total Rainfall
DSSERVE ADD RTNEXRAD/N1P          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" 1-hour Surface Rainfall
DSSERVE ADD RTNEXRAD/N0Z          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" 248 nm Base Reflectifity
DSSERVE ADD RTNEXRAD/NET          NEXR    1 9999 TYPE=IMAGE INFO=NNEXRAD.CFG   
" Echo Tops

You can see from this snippit that a couple of things have to be decided
before plunging in:

o the group name of the dataset
o the names of the various descriptors of the dataset
o the name of the configuration file for the dataset

The group name of the dataset in the list above is RTNEXRAD.  The set
of descriptors in the example above are N0R, N1R, N2R, ..., NET.
The name of the configuration file in the example above is NNEXRAD.CFG.

For the same of argument, let's choose the group name of the dataset
you want to create to be:  NCDCNEXR.  There is _nothing_ magical about
this name other than it MUST be less than or equal to 8 characters in
length.

The next question is what kind of products your student will be FTPing
from NCDC.  Again, for the sake of argument, let's assume that they
will be any/all of the descriptors listed above.  If there are products
that s/he will be FTPing that are not in this list, you need to create
descriptors for them.  NOTE: there may be one or more types of products
that can be FTPed from NCDC that are not supported by the current
McIDAS NEXRAD ADDE server.  If you find this to be the case, _PLEASE_
let me know about it(them) so I can add support for those products!

Next, let's choose the name of the configuration file to be NCDCNEXR.CFG.
There is nothing magical about this name EXCEPT that it should NOT
be NNEXRAD.CFG.  The reason for this is that a file called NNEXRAD.CFG
is sent out with each Unidata McIDAS-X release.  If you use that file
and make editing changes, then you will lose your changes the next
time you do an upgrade.

Since I don't know how many different products are going to be looked at,
and since I don't know how many sites are going to be looked at, I will
suggest making your directory setup for the data as structured as 
possible.  To me, this means expanding on your original idea of
creating a directory like:

/NCDC/RADAR

and expanding it to:

/NCDC/RADAR/station_id/product_type/products

The idea here is to organize the files by NEXRAD station ID and by
product type.  For instance, if one was to grab base reflectivity
at tilt 1 products for Chanhassen, MN (station ID MPX), then the
directory structure would look like:

/NCDC/RADAR/MPX/N0R
       "        N1R
       "        N2R
       ...

If you don't organize your data files like this, the ADDE server has
to open every file to see:

o if it is for the station you are interested in
o if it is the type of product you are interested in

If you only have a hand full of files, this is no big deal.  If you
continue to FTP files and throw them into the same directory, the
overhead of opening every file every time you go to do an IMGLIST or
IMGDISP (or IMGCOPY, IMGFILT, etc.) becomes greater and greater.

So, continuing on with our example, let's assume that the files that
you have so far are all base reflectivity at tilt 1 products from MPX.
I would put all of the files extracted from the gzipped (*.gz) file
in:

/NCDC/RADAR/MPX/N0R

If you adopt the structure that I suggest, then the contents of the
configuration file can be very simple.  Here is what to do AFTER
you have installed McIDAS-X/XCD 7.7x:

<login as 'mcidas'>
cd workdata
cp NNEXRAD.CFG NCDCNEXR.CFG

<edit NCDCNEXR.CFG and make the last three lines look like:

DIRMASK=/NCDC/RADAR/\ID/\TYPE
FILEMASK=*\ID*.*
IPMASK=*

This assumes:

o the directory structure looks like:

  /NCDC/RADAR/MPX/N0R
               ^   ^__  dataset descriptor
               |______  NEXRAD ID

o the file names look like what you provided:

  KMPX20000416_1034_00.580
   \^/\              /|\ /
    |  \            / | |
    |   \          /  | +--> ???  use '*' to match
    |    \        /   |_____ '.'
    |     \      /
    |      +----+
    |         |_____________ looks like date and time, use '*' to match
    |
    |_______________________ K id, use K\ID to match
    

The next thing you need to do is to define the datasets.  For this, you
would create a BATCH file using the RTNEXRAD entries in DSSERVE.BAT
as a starting point.  The new BATCH file could be called NCDCNEXR.BAT
and look like:

REM
REM NOAAPORT NEXRAD Level III products FTPed from NCDC
REM

DSSERVE ADD NCDCNEXR/N0R          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Base Reflectivity Tilt 1
DSSERVE ADD NCDCNEXR/N1R          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Base Reflectivity Tilt 2
DSSERVE ADD NCDCNEXR/N2R          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Base Reflectivity Tilt 3
DSSERVE ADD NCDCNEXR/N3R          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Base Reflectivity Tilt 4
DSSERVE ADD NCDCNEXR/N0V          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Radial Velocity Tilt 1
DSSERVE ADD NCDCNEXR/N1V          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Radial Velocity Tilt 2
DSSERVE ADD NCDCNEXR/NCR          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Composite Reflectivity
DSSERVE ADD NCDCNEXR/N0S          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Storm-Rel Mean Vel Tilt 1
DSSERVE ADD NCDCNEXR/N1S          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Storm-Rel Mean Vel Tilt 2
DSSERVE ADD NCDCNEXR/N2S          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Storm-Rel Mean Vel Tilt 3
DSSERVE ADD NCDCNEXR/NVL          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Vertical Liquid H2O
DSSERVE ADD NCDCNEXR/NTP          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Storm Total Rainfall
DSSERVE ADD NCDCNEXR/N1P          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" 1-hour Surface Rainfall
DSSERVE ADD NCDCNEXR/N0Z          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" 248 nm Base Reflectifity
DSSERVE ADD NCDCNEXR/NET          NEXR    1 9999 TYPE=IMAGE INFO=NCDCNEXR.CFG   
" Echo Tops

I would put this BATCH file in the ~mcidas/workdata directory.  To
create the NCDCNEXR dataset, you would then run BATCH from a McIDAS-X
session as the user McIDAS:

BATCH NCDCNEXR.BAT

At this point, you should be able to use ADDE to access the dataset.
Each user that wanted to access the data set would run:

DATALOC ADD NCDCNEXR waldo.stcloudstate.edu

Alternatively, you could login to each machine as 'mcidas' and IF
you have MCTABLE_READ and MCTABLE_WRITE properly defined in your session:

MCTABLE_READ="/home/mcidas/workdata/MCTABLE.TXT;/home/mcidas/data/ADDESITE.TXT"
MCTABLE_WRITE="/home/mcidas/data/ADDESITE.TXT"

you could run the same DATALOC command:

DATALOC ADD NCDCNEXR waldo.stcloudstate.edu

and each user would automatically be able to access the NCDCNEXR dataset
from waldo.  This will work if each user's Unix session has MCTABLE_READ
and MCTABLE_WRITE properly defined:

MCTABLE_READ="/home/user/mcidas/data/MCTABLE.TXT;/home/mcidas/data/ADDESITE.TXT"
MCTABLE_WRITE="/home/user/mcidas/data/MCTABLE.TXT"

>Thanks

Let me know if this doesn't make sense.  Working through the setup one
step at a time will help galvanize the concepts.

>>From address@hidden Fri Mar  2 12:46:42 2001
>
>HI
>
>I should have known better, besides it is Friday.
>
>I looked at the build progress on waldo, and see it
>has stopped with a fatal error compiling uacross.

What happened is that your use of gcc/f2c uncovered a problem in the
uacross.pgm code that I developed a fix for and was going to release
in the next addendum for 7.7x.  I made the modifications to your
copy of uacross.pgm and restarted the build.  It ran to completion
so you should be set to do the installation when you are ready.

>Lots of stuff went fine up to this point, although there
>are some warnings. 
>
>I know this is one of the new commands, but am not sure
>why the build should fail.

Bad code exposed by use of gcc/f2c.

>makelog also does not tell
>me much, so I will ask you.  If you are aware of a likely
>cause, let me know;  or you already know how to get on 
>waldo.  ldm is still running, but you indicated earlier
>that it could be left on untill the install.

Right.  The LDM running is no problem.

The 7.704 build is finished, so you can do the installation whenever
you like.  Alternatively, I could do the installation for you this
weekend (I will be working from home).  Please let me know.

Tom

>From address@hidden Sat Mar  3 06:48:03 2001
>Subject: thanks

Tom

I snooped onto waldo yesterday afternoon and saw that you had 
already been there and solved my problem.

I have your radar message and will study that for a while.  Have 
to be gone for part of today, but may try and finish the 
McIDAS upgrade before Monday.  I appreciate your offer, but will 
continue, as you  gave me an excellent map.

Thanks so much for everything.  The support you provide, (and 
UNIDATA)  is just superb, and it is crucial for departments such
as mine.  

Alan