20020909: McIDAS data archiving options - other universities

>From: William C Klein <address@hidden>
>Organization: Valparaiso
>Keywords: 200209092213.g89MDp127059 McIDAS archive

Bill,

>We are still trying to come up with archiving options for McIDAS (and
>soon, GEMPAK) datasets.  I've been told that tar, archive and zip are my
>options.

These are the typical things used on Unix systems, yes.

>I'm sure that there has to be some other utilities that would
>allow our Met Faculty to save datasets for research and keep them
>available for viewing in McIDAS.

Here is one idea:

Since the objective is to save off datasets for later use in McIDAS, I
would first advise you (or someone else at Valparaiso) to become
familiar with the concepts behind making McIDAS ADDE datasets.  This
will require the user to understand McIDAS file naming conventions, the
McIDAS file REDIRECTion facility, and the concept of MCPATH and how it
is used to tell McIDAS how to find data files.  If I were that person,
I would start by identifying a set of data that is to be saved off and
reused at a later time, and move/copy the files to a separate
directory/directory hierarchy and then create an ADDE dataset that is
composed of the various types of data being saved (e.g., IMAGE data
(imagery in AREA files or radar data in native formats (as delivered by
the Unidata IDD)), POINT data (observational data in MDXX files), and
GRID data (objective analysis data/model output in GRID files).

In the dataset creation process, the user will need to understand that
the MDXX and GRID files will need to be renamed so that access to the
copied files will not intefere with access to realtime data files being
created by the McIDAS-XCD decoders.  The MDXX "cylinders" that should
not be used when saving off data into archives are 0001 - 0200.  The
current use of MDXX file "cylinders" ("cylinder" is a reference to the
4 digit number that is the second part of the MDXX file name:  MDXX0001
-> "0001" is a cylinder number, etc.) for realtime (current) data is as
follows:

MDXX0001 - MDXX0010   -  SAO/METAR observations
MDXX0011 - MDXX0020   -  Mandatory level upper air observations
MDXX0021 - MDXX0030   -  Significant level upper air observations
MDXX0031 - MDXX0040   -  SHIP/BUOY data
MDXX0041 - MDXX0050   -  NGM MOS
MDXX0051 - MDXX0060   -  SYNOPTIC data
MDXX0061 - MDXX0070   -  PIREP/AIREP data
MDXX0071 - MDXX0080   -  NLDN data
MDXX0081 - MDXX0090   -  Hourly summary FSL Wind Profiler data
MDXX0091 - MDXX0100   -  6-minute FSL Wind Profiler data

When one creates a dataset, s/he must be careful to not use any of the
above file names for POINT data as they will conflict with the files
being used to store realtime data.  This is important!  In general, I
would adopt the approach of adding 1000 to the cylinder number for a
particular data type:

original file         copyied file
MDXX0001              MDXX1001 or MDXX2001 or ... or MDXX8001, etc.
MDXX0011              MDXX1011 or MDXX2011 or ... or MDXX8011, etc.

Also, I would copy all of the files to a single directory whose name is
representative of the dataset being created.  For instance, you might
be saving a set of data for a tornado incident.  I would call the
directory in which the files are copied .../tornado/...  Even better
would be to incorporate the date of the event into the directory name,
something like: .../tornado_20020910/..., etc.

The same sort of renaming must be done for GRID files.  GRID files
created by the XCD decoders occupy cylinder numbers that range from
5001 to 6400 (e.g., GRID5001 - GRID6400, inclusive).  You must rename
the GRID files you want to save so that their cylinder numbers do not
match those being used for realtime data storage.

Once the files have been saved into a directory/directory hierarchy, it
is time for the person to create an ADDE dataset that allows him/her to
access those files.  To do this, I would review the contents of the
~mcidas/data/DSSERVE.BAT file contained in the Unidata McIDAS
distribution; the online and manual helps for the command DSSERVE; and
the McIDAS Learning Guide:

http://www.unidata.ucar.edu/packages/mcidas/2002/mcx/mclearn/index.html

sections on creating ADDE datasets.

Creation of the dataset will most likely involve the user setting up a
series of file REDIRECTions (review the online help for the McIDAS
command REDIRECT) so the data files can be found.  The REDIRECT
commands used to define the REDIRECTions should be saved into a McIDAS
BATCH file and saved with the data (for later use).

After one can successfully create a dataset for data saved into an
arbitrary directory/directory hierarchy, it is time to save the data to
some non-volatile medium like tape, CDROM, etc.  Either tape or CDROM
will allow the set of data to be returned to the hard disk when the
case is to be used.  Saving to CDROM has the advantage of one simply
mounting the CDROM and being able to access the dataset without going
through a step of copying data back to disk.

>What are other schools using to accomplish this.

Most schools save off data files to tape using tar.  For more
information from users, you could post your question to the mcidas-x
email list that we (Unidata) maintain.  Before one can post to a
Unidata-maintained email list, s/he must subscribe and be accepted to
that list.  I just did a check of the mcidas-x list members and see
that you are the only person from Valparaiso that is subscribed.  If
you are the person that will be responsible for creating the datasets
and saving the data, this would be sufficient.  If others will be
involved, I would recommend that they subscribe to the list.  As a
reminder, one can subscribe to Unidata-maintained email lists online
at:

http://www.unidata.ucar.edu/mailinglist/mailing-list-form.html

Tom