[McIDAS #YAF-607526]: Schema format

Hi Kwan,

re:
> In my last email, I told you that the surface data could be listed by
> PTLIST.  As I tried to display the data, I encountered a few
> problems.  The first problem was that McIDAS did not display any data
> at all.  This baffled me for more than a hour before I recalled that
> latitude in the western hemisphere in McIDAS is defined as positive
> instead of negative.

The western-hemisphere positive convention in McIDAS is hard to remember
for those who are used to working in "the real world" :-)

> The second problem I got was that when I tried using HMS, McIDAS could
> not find it in the MD file.  So I added a HMS column in the data, and
> that worked.

Yup, only keys defined in the MD file can be accessed.  So, asking for HMS
requires that HMS be in the file.

> The third problem was the following.
> 
> PTDISP PTSRC/METAR.1 1 PAR=T[F] TD[F] PSL[MB] GUS[KT] MUL=X X 10 X
> LSI=6 6 6 6 COL=8 8 8 8 OFF=-8 -13 3 -13 -8 4 3 4 FORMAT=I3 I3 K3 I3
> SEL='DAY 2007106 2007106; HMS 11:40:00 12:00:00' DEC=YES LIN=1 MAX
> ELE=1 MAX TITLE=' '
> PTDISP: WARNING: Too many digits for FORMAT on PARAMETER = GUS
> PTDISP - done

I just added the schema defined in the KSFC.txt file you sent to my SCHEMA
file and created an output MD file on my home Fedora Core 6 development
system:

SCHE KSFC.txt
KSFC.txt     will be treated as a text file
" KSFC (JSP) 07330 SCHEMA -- GENERAL SURFACE SCHEMA
"        NAME VSN DATE  ID "TEXTID
"        ---- --- ----- -- -------
  SCHEMA KSFC  1 2007330 0 "Archived METAR data
*--SCHEMA * KSFC *, VERSION * 1 * ADDED TO SCHEMA FILE *  KEYS=15

 --END OF SCHEMA REGISTRATION PROGRAM

LSCHE KSFC

NAME: KSFC  VERSION:  1  DATE: 2007330  TEXTID: "Archived METAR data
----        -------
   DEFAULT NUMBER OF ROWS:   24                        INTEGER ID: 0
                     COLS:   5000                   MISSING DATA VALUE:  
-2139062144
   REPEAT GROUP:  NUMBER OF REPETITIONS:   1
                      STARTING POSITION:   3
                                   SIZE:  13
   NUMBER OF KEYS IN ROW HEADER:     2
                     COL HEADER:     0   STARTING AT POSITION   3
                    DATA RECORD:    13   STARTING AT POSITION   3
                                  ----
                                    15 TOTAL

        KEY  SCALE UNIT     KEY  SCALE UNIT     KEY  SCALE UNIT
        ---- ----- ----     ---- ----- ----     ---- ----- ----
        DAY    0   CYD      HMS    0   HMS      ID     0   CHAR
        TIME   0   0        LAT    4   DEG      LON    4   DEG
        T      2   F        TD     2   F        PRE    2   MB
        PSL    2   MB       DIR    0   DEG      SPD    1   KTS
        GUS    1   KTS      WX1    0   0        ZS     0   0
LSCHE: DONE

REDIRECT ADD metar-mc.txt "/data/casestudy
REDIRECT: Done

TXT2MD metar-mc.txt 1000 KSFC MISS=-9999.00
 TXT2MD -- BEGIN
NEW MD file: 1000
 TXT2MD -- DONE:  1000


I then listed out all of the data using:

PTLIST MYDATA/PTSRCS.1000 PAR=ROW COL DAY HMS ID TIME LAT LON T TD PRE PSL DIR 
SPD GUS WX1 ZS NUM=100

and got the listing I expected/wanted.  I have attached this listing (KSFC.out) 
for
your review.

> I checked the graphics output.  It displayed *** for some stations
> where the gusts were -9999.00, which represented missing. (I had the
> surface data in a GEMPAK file, I then listed it and changed it to
> McIDAS text format.)  So it seemed to me that when I ran

As you will see from the attached PTLIST listing, my conversion to MD
did not fail to correctly mark missing data values as truely missing (i.e.,
I have no -9999.0 values in my MD file).

> txt2md.k metar-mc.txt 100 KSFC MISS=-9999.00

This is the same thing that I did.

> it did not really save -9999.00 as a missing flag but treated it as an
> actual value.

OK, very weird.

Question:

- what OS are running McIDAS on

- what version of McIDAS are you using (SSEC or Unidata and
  the version number of the distribution)

- does your distribution contain the Unidata version of TXT2MD, or
  does it have the SSEC version.  You can tell which version
  you have by getting a HELP listing for TXT2MD.  The Unidata
  version will look like:

HELP TXT2MD
 TXT2MD - Creates an MD file from an ASCII text data file
    TXT2MD sfile dfile schema <keywords>
 Parameters:
    sfile  | source ASCII text file name and extension
    dfile  | destination MD file number
    schema | MD file schema name, e.g. IRAB, ISFC
 Keywords:
    MISS=numeric character
           | numeric   - value representing missing numeric data
                         in sfile (def=-9876)
           | character - value representing missing character data
                         in sfile (def=MISS)

 Remarks:
    TXT2MD is used to convert files containing lists of data to
    MD files.

 Rules:
    1) The source file (sfile) must be in ASCII text format, i.e.,
       created with an editor that stores ASCII text with a new line
       specifier at the end of each line.

    2) The format of sfile must be columnar:
         o the first line containing the names of the schema keys;
           schema key names must be no more than 4 characters in
           length
         o <optional> the second line can contain a list of units
           for the data values in the file; if these units do not
           match those specified in the MD file schema, the data
           will be converted to match the schema units;  for units
           that do match the schema units, an X may be used as a
           placeholder; units must be no more than 4 characters in
           length; in addition, data values may be:
           o multiplied by a scaling factor
               specify the new unit as a '*' followed immediately
               by a number (no spaces!)
           o divided by a scaling factor
               specify the new unit as a '/' followed immediately
               by a number (no spaces!)
           o have a positive offset added
               specify the new unit as a '+' followed immediately
               by a number (no spaces!)
           o have a negative offset subtracted
               specify the new unit as a '-' followed immediately
               by a number (no spaces!)
           NOTE: the line containing units MUST begin with 'UNIT='
         o subsequent lines must contain data values corresponding to
           the schema keys; the first datum in the list must NEVER be
           a missing data value (see 7) below)!
         o extra spaces between the columns are ignored

           Example 1, the first 3 lines might look like (T is
                      in K; SPD is in KT):

           TIME  ID   LAT    LON    T       PSL      DIR  SPD
            12   MSN  43.13  89.33  280.42  1020.95  270  12
            12   MKE  42.95  87.90  278.04  1020.3   290  14

           Example 2, the first 4 lines might look like (here we
                      specify that T is in C and SPD is in MPS):

                TIME ID   LAT    LON     T       PSL   DIR  SPD
           UNIT=X    X    X      X       C       X     X    MPS
            12   MSN  43.13  89.33  7.26  1020.95  270  6.17
            12   MKE  42.95  87.90  4.88  1020.3   290  7.20

           Example 3, the first 4 lines might look like (here we
                      specify that T is in C, SPD is in MPS,
                      and LON should be multiplied by -1):

                TIME ID   LAT    LON     T       PSL   DIR  SPD
           UNIT=X    X    X      *-1     C       X     X    MPS
            12   MSN  43.13  89.33  7.26  1020.95  270  6.17
            12   MKE  42.95  87.90  4.88  1020.3   290  7.20

    3) all lines in sfile must be less than or equal to 1024
       characters in length

    4) the field length for each datum must not exceed 12 characters

    5) Each data line in sfile is placed into one record in the
       destination MD file (dfile).  Each data line must also include
       any data to be stored in the record's repeat groups (if repeat
       groups are defined in the schema).

    6) the first ROW header key in the file schema must have a valid
       value (i.e. not a MISSing data value)

    7) the first DATA key in the file schema must have a valid value
       (i.e. not a MISSing data value)

    8) dfile will be created if it does not exist.  If dfile already
       exists, the data will be appended to the first available row.
       Used, any columns in the dfile row that match the column headers
       from sfile will be replaced.  If the column headers in sfile do
       not match any column headers in dfile, the data will be appended
       to new columns at the end of the row.

    9) The values for the row header keys (e.g., DAY, TIME) in dfile
       are set to the values found in the first data line of sfile.
       Values of the row header keys in subsequent data lines in sfile
       are ignored.

   10) The dfile must not exceed 6 column header keys or 10000 columns.
       Each record in dfile, including any repeat groups in that record,
       may not contain more than 64 keys.

----------

If you are using the SSEC version, then either you are using a Unidata
McIDAS distribution that did not have the Unidata version of TXT2MD in it
(this did happened right around the time of the training workshop this past
August; I updated the Unidata distribution with the correct version just after
the workshop), or you are using an SSEC distribution.  If you do not have
the Unidata version of TXT2MD, then I recommend that you upgrade to the Unidata
version.  I have attached the source code for the Unidata version (txt2md.pgm)
so that you can easily upgrade.

To build a new TXT2MD executable, do the following:

<as 'mcidas'>
cd ~mcidas/mcidas2007/src
mv txt2md.pgm txt2md.pgm.ssec
-- copy the Unidata version of txt2md.pgm here
make txt2md.k txt2md.hlp
rm ~/bin/txt2md.k && ln txt2md.k ~/bin
rm ~/help/txt2md.hlp && ln txt2md.hlp ~/help

> The next problem is this:
> 
> SFCCON PMSL OLAY 12 2007106 LSIZE=12 COL=2 LLI=.75 .75 CIN=4
> DAT=PTSRC/METAR.1 PTSRC/SHIP.1 PTSRC/SYN.1 LAB=X 2 15 X X
> Accessing Dataset Name = PTSRC/METAR.1
> Accessing Dataset Name = PTSRC/SHIP.1
> Accessing Dataset Name = PTSRC/SYN.1
> PTCON: No data found matching search conditions
> PTCON:  Done
> SFCCON: PTCON command failed 1
> SFCCON - Done
> SFCCON failed, RC=1
> So, it seemed that it could not understand PMSL.

The problem here is that the schema for your PTSRC/METAR is not what is
being specified in the definition of PMSL that is used by SFCCON. 
The configuration file entry in question is:

PMSL   UNIT=MB \
       ISFC=PRE SVCA=na SYN=PSL ISHP=PSL

This is in the file SFCCON.CORE which should be located in your ~mcidas/data
directory. You will see that this entry specifies the key to use based on schema
name:

ISFC -> PRE
SVCA -> not available
SYN  -> PSL
ISHP -> PSL

What you should do is create a file named SFCCON.USER in your $MCDATA
directory.  The contents of this file should be a replacement definition
for PMSL or a definition for a new name.  For example:

replacement that overrides the definition in SFCCON.CORE:

PMSL   UNIT=MB \
       KSFC=PRE SVCA=na SYN=PSL ISHP=PSL

new definition:

MYPMSL UNIT=MB \
       KSFC=PRE SVCA=na SYN=PSL ISHP=PSL

I suggest the new definition approach since it will not interfere with the
definition used for "standard" MD files.

> I am attaching a subset of the data (northeastern US), the schema, and
> a gif file of the graphic output I got.

Thanks.  As you can see from my comments above and the PTLIST listing that
is attached, I used the data and got the results that I wanted.

> Thanks,

No worries.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: YAF-607526
Department: Support McIDAS
Priority: Normal
Status: Closed

Attachment: KSFC.out
Description: Binary data

Attachment: txt2img.pgm
Description: Portable graymap