20010204: Gempak, NCDC/FSL CDROM

Diana,

Sorry to take so long in packing this stuff up. Meetings keep
piling up and keeping us from getting other work done!

I have created a tarfile of a mininal GEMPAK environment along
with the software I used to extract data from the NCDC/FSL 
radiosonde data of North America CDROM set.

You can download this from our ftp server:
host: ftp.unidata.ucar.edu
login gbuddy
password: XXXXXX

Once logged in, you will find the tarfile under:
nawips-5.6/binary/cal/gempak.tar.Z


Donwload this file to a clean directory on your computer where you want the 
software to reside. Unpack the tarfile with:
zcat gempak.tar.Z | tar xvf -

Edit the Gemenviron file in the top level directory. Edit the NAWIPS 
environmental
variable definition to the be the path to the directory where you unpacked
the distribution.

The Gemenviron file is a csh script. It is assumed you will run GEMPAK from
a csh (or tcsh) session.

Once you have edited the Gemenviron file, you can source the Gemenviron file 
with
the command "source Gemenviron". This sets environmental variables in your
current working environment for your shell. You will do this each time you want
to run GEMPAK, so it is usually a line added to your .cshrc file in you home
directory.


To convert the bufr format data from the CDROMs that you have into GEMPAK 
format,
you will do the following:

1) Extract the data set from the CDROM to a fsl text format file on your disk
   using the getraob program that NCDC provides (I have included this in the
   tarfile)

2) Convert the fsl format file to a GEMPAK format file using the fsltogem 
program
   I wrote. I eliminated the need for snedit by writing the GEMPAK file directly
   since you mentioned that disk space was a problem.

I provided a script $NAWIPS/bin/scripts/getraob_cdrom.csh
This script runs the getraob program from NCDC. You should edit the
script and change the "/CDROM/raobs" directory location to be wherever
your cdrom mount is located (on my computer, the cdrom mounts to
the directory /CDROM, the /CDROM/raobs directory is the location of
the radiosonde data on the cd.

From the information you provided me, I'll assume that you are going to
extract data from a few stations for a long time period. Change your
working directory to the $GEMDATA directory (this location isn't important
as far as the software goes, so you can change this if needed).

To extract the data, run the script "getraob_cdrom.csh". The script
and GEMPAK executable directories will be added to your path when you source 
the 
Gemenviron file, so all you have to do is type getraob_cdrom.csh at you prompt.
The NCDC/FSL program will ask you several questions to download the data.
I'll step through these as an example using the first cd of the set I have
which is 1946-1965. My input is shown on lines marked >>>>. Things to note
are that my conversion program will expect that you use the "new format 99999"
option, as well as ask for winds in "10ths of m/s".

********************************************************************

Welcome to the NOAA/FSL Rawinsonde Database 
                                Access System - Version 2.3

<snip>

        Enter HELP for general information about this database
        Enter <return> to access the rawinsonde database
>>>>    Your choice:  [RETURN]

    NOTE: Raob data from the directory: /CDROM/raobs/
            is available for the years:  1946 thru 1965


>>>> What is the beginning date? YYYYMMDDHH [1946010100]
>>>> What is the ending date? YYMMDDHH [1965123123]

 What GMT sounding hour(s) do you want?
      0  -  All times
      1  -  0z and 12z only
      2  -  0z only
      3  -  12z only
>>>>  Your Choice? [0]

 How do you wish to access the data set?
      0  -  by station number
      1  -  by geographic location
      2  -  all available observations
>>>>  Your Choice? [0]


                 Station Number Access

>>>> What type of station numbers are you using? [WMO=0, WBAN=1] [0]
>>>> Do you have a file containing this list of stations? [Y/N] [N]
>>>> How many stations for the time period would you like? [1 - 100] [1]
>>>> WMO station number? 72469
     What type of output format do you want?
      0  -  FSL format
      1  -  FAA604 format <USED BY SHARP SOFTWARE>
      2  -  NETCDF format
>>>> Your Choice? [0]

  **************************************************
  We have begun including data above 100mb pressure.  In order to support this
  data a new modified output format contains the following changes:
        1. All pressure levels are in tenths of millibars
        2. The missing indicator changed from 32767 to 99999
        3. Data above 100mb (if available)

  **************************************************

>>>> Do you wish to output data in the new format? [Y/N] Y
 What type of data levels do you want?
      0  -  all levels
      1  -  mandatory levels only
      2  -  mandatory and significant levels only
>>>> Your Choice? [0]
 What type of wind speed units do you want?
      0  -  knots
      1  -  tenths of meters/second
>>>> Your Choice? [1]

 How do you want to organize the output data?
      0  -  Synoptic Sort (time series)
      1  -  Climatological Sort (station series)
    Note:  For large data requests, type help for performance information.

>>>> Your Choice? [1]

 Where do you want to direct the output?
      0  -  to the screen
      1  -  to output file(s)
>>>> Your Choice? [1]

 Where do you want to write the output files?
>>>>  Enter the disk and directory only: ./       

                Station Series Sort 

 How do you want structure your output file(s)?
      0  -  all data in one file      <YYMMDDHH.rao>
      1  -  one station per file      <YYMMDDHH.INI>
      2  -  one sounding per file     <MMDDYYHH.INI>
>>>> Your Choice? [0]

 STATUS:  Writing output file: ./1948101603.rao
 STATUS:  number of soundings written: 18545


>>>> Do you wish to continue accessing this data? [Y/N] [N]
>>>> Would you like to request more data? [Y/N] [N]

*************************************************************************


Ok, that is step one. I included in the distribution tar file the sample data 
set for Denver I obtained above in the $GEMDATA directory. That is 25MB so you
can remove it as necessary- but I wanted you to have a test data set to work 
with.

Now, you can convert the FSL text format file created above onto a GEMPAK format
file using the "fsltogem" program. Using the output file created above 
1948101603.rao,
run fsltogem as:

fsltogem 1948101603.rao


The program will run for a while creating yearly GEMPAK files from the above 
data.
When complete, you will have:
% ls 19??.gem
1948.gem  1950.gem  1952.gem  1954.gem  1956.gem  1958.gem  1960.gem  1962.gem  
1964.gem
1949.gem  1951.gem  1953.gem  1955.gem  1957.gem  1959.gem  1961.gem  1963.gem  
1965.gem


Each yearly file is about 2MB and can store up to 2000 times, for 200 stations.
Assuming 2 soundings per day is 765 soundings in a yearly file. Since sounding
times have changes over the years, you shouldn't assume that all soundings
are at 0Z and 12Z (eg, the choice above for "all sounding times" in the cdrom
extraction program). In fact, for the above data, most soundings are at 3Z and 
15Z 
with many other times. You can run the getraob program for each CD in the set 
you
have and use fsltogem to convert each to GEMPAK format files. You can remove the
FSL format ascii file when you are done converting to GEMPAK format.

To use SNLIST to look at the data for 1950:

% snlist
 SNFILE   = 1950.gem
 AREA     = @72469
 DATTIM   = all
 SNPARM   = dset
 STNDEX   = lift
 LEVELS   = all
 VCOORD   = pres
 OUTPUT   = t
 MRGDAT   = y
GEMPAK-SNLIST> r


To plot a time serires of LIFT for 1950:

% sntser
 SNFILE   = 1950.gem
 DATTIM   = all
 TAXIS    = 500101/0000-510101/0000
 LEVELS   = sfc
 VCOORD   = pres
 SNPARM   =  
 STNDEX   = lift
 AREA     = @72469
 PTYPE    = lin
 YAXIS    = ///1;1;1
 BORDER   = 3
 LINE     = 5
 MARKER   = 0 
 TITLE    = 1/-1
 CLEAR    = y
 PANEL    = 0
 TEXT     = 1/21/1/hw
 DEVICE   = xw
 GEMPAK-SNTSER>r
Creating process: gplt for queue 52
Creating process: xw for queue 53
 SNTSER PARAMETERS

 Sounding file:      1950.gem
 Parameter:          LIFT
 Area:               @72469
 Number of times:    1126
 Date range:         500101/03-1231/21                               
 Data range:           -6.445       24.26    
 Panel location:     0                                                          
                                                                     
Enter <cr> to accept parameters or type EXIT:




Some notes about STNDEX parameters (you can use the "phelp stndex" help command 
in the
gempak programs since I did include the online help files).

The default for computing LIFT using the average temperature over the lowest 
100 mb
(this helps remove some problems when the surface layer is strongly influenced
by surface radiation). You can tell GEMPAK how deep if a layer to use using the 
!
character. For example, to use only the surface temperature and no layer:
STNDEX = lift!0


Also, in the above I am using the xw device (so I have set the environmental 
variable
DISPLAY to my X server). I have also included the ps and gf (postscript and gif)
drivers as well (I created a denver_lift_1950.gif file in $GEMDATA from the
example above).



The Solaris binaries were compiled using the Sun fortran compiler.
If you don't have the shareable libraries installed on your system
(you can test this by running: 
ldd $GEMEXE/sntser
        libF77.so.4 =>   /usr/local/SUNWspro/lib/libF77.so.4
        libM77.so.2 =>   /usr/local/SUNWspro/lib/libM77.so.2
        libsunmath.so.1 =>       /usr/local/SUNWspro/lib/libsunmath.so.1
        libm.so.1 =>     /usr/local/SUNWspro/lib/libm.so.1
        libc.so.1 =>     /usr/lib/libc.so.1
        libdl.so.1 =>    /usr/lib/libdl.so.1
        /usr/platform/SUNW,Ultra-2/lib/libc_psr.so.1


If you are missing the fortran/sunmath libraries on your system, you can
download them from the gbuddy ftp account under the directory:
nawips-5.6/binary/solaris_sparc

If you have to download these libraries, create a $NAWIPS/lib/sol directory and
place them in there. The add the directory to your LD_LIBRARY_PATH environmental
variable.

You will find documentation about the FSL format and the cdrom software in
the $NAWIPS/etc directory.

I tried to assemble things based on your email messages. Iff you have
questions, let me know.

Steve Chiswell
Unidata Uset Support












>From: Diana DeRubertis <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200102042124.f14LO8X09718

>Hi Steve,
>       I have to meet a deadline and need to get started on this research
>asap.
>You probably have a million things to do...just wondering if you could get
>the file to me sometime this week?  If not, I can load the program and data
>into two different accounts, to avoid memory problems.
>
>Thanks again,
>Diana
>
>
>
>>Hi Diana.
>>I have the latest update to the NCDC cdrom (V 2.3 of getraob)
>>which I will pack up with snlist and a script with program to
>>import the data into GEMPAK.
>>
>>One question I wanted to ask you for tailoring the script is
>>if you will primarily be looking at a time series of values
>>for a few stations. If you will only be extracting data for a few stations,
>>then I'll set the script up differently than if you are planning to
>>get all stations for all times since the getraob program that FSL/NCDC
>>provide
>>for the cdroms will create either files by date or files by station.
>>
>>Steve Chiswell
>>Unidata User Support
>>
>>
>>>From: Diana DeRubertis <address@hidden>
>>>Organization: UCAR/Unidata
>>>Keywords: 200101221739.f0MHd9e18479
>>
>>>Hi Steve,
>>>     I'm back to work after travelling east.  I was wondering if you
>>>could still send me that tarfile of the GEMPAK programs I would need to
>>>calculate stability indices in SNLIST, and possibly SNEDIT, SNTSER, SNFILE.
>>>I'll also need to loop files and run SNLIST for about 6 months of the year,
>>>for the 50 years in the record.  I'll be using a Sun operating system and
>>>the upper air data from the NCDC.
>>>
>>>Let me know if this is too difficult, and I'll try to download the whole
>>>program...but i'm worried about memory problems.
>>>
>>>Thanks,
>>>Diana
>
>******************************************************************************
>
>Diana DeRubertis
>Dept of Geography
>University of California, Berkeley
>Office:  199 McCone Hall
>Email:  address@hidden
>
>