20020301: Archived NEXRAD data from NCDC and NEXRAD reflectivity composites

>From: Louis Nguyen <address@hidden>
>Organization: NASA/Langley
>Keywords: 200203011521.g21FLFx21046 McIDAS NEXRAD ADDE

Louis,

re: national NEXRAD Level III reflectivity composites
>Sound like it's pretty neat stuff.  I'll definitely check it out. 

This reply contains the email you sent as a follow-up to this one.

>Thank you for the information on the format issues.  We are thinking
>about ordering a few days (july 2001) of nexrad level III data from
>NCDC.  I was afraid I wont be able to display it in mcidas.  

You should be able to use McIDAS to look at that data, but there
may be one little snag:

o The NEXRAD servers that I wrote originally worked with the NIDS data
  that one could purchase from WSI Corporation.  WSI added on
  additional record to the files that contained the NEXRAD station ID.
  I wrote my server to use this data if it existed.

o The current NEXRAD Level III products in NOAAPORT contain a header
  that also contains the NEXRAD station ID.  My server code can also
  detect and use this information.

o it is possible (actually likely) that the historic NEXRAD Level III
  products that you will FTP from NCDC may not have the current
  header that contains the station ID, and it definitely will not
  contain the station ID in the additional record supplied by
  WSI.

The third item will cause the snag that I am referring to, but there is
a way around this.  The server will still be able to read the data
so you can display it, but it will not be able to differentiate between
like products from different NEXRADs.  The navigation will be correct,
but you will not be able to select the particular NEXRAD through
the ID= keyword supported by commands like IMGDISP, IMGMAG, IMGOPER, etc.

There are two solutions to this problem:

o I need to find some time and update the server code to use the station
  ID number that is in all of the versions of the Level III products.
  This will be done, but maybe not in the time frame that you need
  for your studies (I will see if I can get to it sooner than later)

o you can organize the NEXRAD products you FTP by station;  put
  all products from a single station in a single directory, or,
  better yet, use a hierarchical structure for your data, and
  create individual datasets based on the NEXRAD ID.  The reason
  for this is that the server will not be able to determine/assign
  a station ID based on the information in the data files.  This
  means that you will have to use the '*' wild card character
  as the value of the ID= keyword on the various IMG* commands:

  IMGDISP FTGNEX/N0R ID=* STA=FTG EU=BREF ...
  IMGDISP TWXNEX/N0R ID=* STA=FTG EU=BREF ...

  I recommend saving the files you FTP in a hierarchical structure like:

  .../\ID/\PROD/file

  Where:

  .../  -> refers to some top level directory like:
             /data/casestudies/nexrad
  \ID   -> NEXRAD station ID (e.g., FTG, TLH, TBW, etc.)
  \PROD -> NEXRAD product type (e.g., N0R, N1R, NVL, NCR, etc.)


  It is then up to you to create the appropriate directory structure
  and correctly populate the various directories.  After doing this,
  you would setup your ADDE dataset using the DIRFILE= keyword
  of DSSERVE something like:

  DSSERVE ADD FTGNEXR/N0R NEXR  1 9999 TYPE=IMAGE 
DIRFILE=/data/casestudies/nexrad/FTG/N0R/* "Denver CO NEXRAD Base 
Reflectivities at Tilt 1

  DSSERVE ADD TWXNEXR/N0R NEXR  1 9999 TYPE=IMAGE 
DIRFILE=/data/casestudies/nexrad/TWX/N0R/* "Topeka KS NEXRAD Base 
Reflectivities at Tilt 1

  etc.

I _really_ need to upgrade the server so that people do not have to jump
through this kind of hoop to look at historic NEXRAD products.

>BTW, do UNIDATA archive any of the nexrad products (JPG or raw data)?

We don't archive the files, but we keep two weeks of all products for
all stations online.

>More related questions:  In mcidas, Do you know how to overlay data
>from a nexrad radar station on top of a satellite image?

In standard (i.e., non-Unidata) McIDAS there is no way to overlay
one image on top of another.  What you have to do is merge the
images together and create a new enhancement that colors the satellite
image portion differently than the radar echos.

>Do you have
>a way to burn it into the graphic frame instead of the image?

In Unidata McIDAS, I include a test routine from the Australian Bureau
of Meteorology that can display image values in the graphics plane.
This routine is _not_ ready for prime time; in fact, it needs work
to smooth out the rough edges and make it more useful.  It does,
however, have the basic technique for doing the overlay, so someone
with a little time and energy could take what is there and come up
with a good routine that should then be contributed back to the
McIDAS Developers Forum (MDF).  Interested?

>What
>about making composites from several stations?

Rick Kohrs of SSEC came up with a McBASI script that can be used to
composite NEXRAD data.  We came up with a different technique using
GEMPAK.  The result of this effort is the composite that I told
you how to access in a previous email.

>We are planning to
>develop these tools but wondering if there were already progress in
>this area.

A lot of the work has been done, but shaping up the ABoM routine to
do image overlays and contributing it back to the McIDAS community
would be a _very_ good thing!

>Maybe we can enhance/modify to suit our needs. It would be
>very useful for us and other users to have such tools.

Excellent!  Let me know if you want to pursue work on the ABoM code.

>From address@hidden Fri Mar  1 13:14:07 2002

re: NEXRCOMP composite NEXRAD base reflectivity products

>That's some neat stuff!  The national composites are great.

Thanks.  I will pass along the kudos to Steve Chiswell, our GEMPAK
guru that developed the compositing code.

>It's much
>easier to use then figuring out the station ids.

Yes, but you have to remember that composites have a lot of assumptions
built in.  For instance, the collection of base relectivity products
that go into a composite come from a finite time period.  In the case
or the ones that we are producting, the window for use of a product is
one half hour.  The compositing routine looks for the latest N0R
product from each NEXRAD for inclusing, but the latest from any one
NEXRAD may be up to a half hour old.  Also, the technique is creating a
grid of the falues over a specific region (CONUS).  The value assigned
to each grid point is the maximum reflectifity seen from all NEXRAD N0R
products that cover that grid point.  This also has to be remembered
when using the product.  So, we will be issuing words of caution to our
users when we announce the availability of the composite products.
When in doubt, a researcher should always use the original, single
station NEXRAD Level III product.  Also, we will be encouraging all
users to do their own comparisons of the composites and the individual
products they are being created from and make their own determination
as to the representativeness of the composite.  I call the collection
of all of these caveats: The Big Boy Warranty.  We are all big boys and
girls, so we should think about the data products we use and not assume
that they are perfect representations of reality!  We will not guarantee
the accuracy of the composites, so they will be use at your own risk.

>I don't think there's
>much to improve.  Everything looks great.

They certainly are pretty :-).

>We would certainly like to
>access your adde servers if you don't mind.

We won't mind as long as there is some tangible benefit back to the
Unidata community; a "tit for tat"/"quid pro quo".  We will need
to talk about this in more detail.

>Also, we would be interested in setting one up here at Langley. 

This will be one of the things that we will have to talk more about.
Until then, please fell free to access any of the 4 cooperating
Unidata community ADDE servers I listed in a previous email.  By
accessing those servers and using those products, you are implicitly
accepting responsibility for their use.  Again, we make no claims
about the validity of the products.  We consider them experimental
products that need scientific review.

>--------------------------------------------------------------
>| Louis Nguyen               Atmospheric Sciences Competency |
>| MS420                                  Tel +1 757 864-9066 |
>| NASA Langley Research Center           Fax +1 757 864-7996 |
>| Hampton, VA 23681-2199              address@hidden |
>--------------------------------------------------------------

Tom