20030616: importing and contouring data on GOES imagery (cont.)

>From: Owen Cooper <address@hidden>
>Organization: Aeronomy Laboratory/NOAA
>Keywords: 200306112200.h5BM0fLd024027 McIDAS TXT2MD PTCON

Hi Owen,

>Thank you for the advice on reading ASCII text into MCIDAS for 
>contouring onto GOES imagery.

No worries.

>I tried registering a new SCHEMA file but ran into a problem.
>Please see the attached text file which should explain all.
>Please let me know if you have any ideas as to what the problem may be, 
>and let me know if you need any more info.

As I said on the phone yesterday, the missing piece is knowing what
a schema definition file should look like.  A description of an
existing schema will probably help:

-- DCNLDN --
"       NAME    VSN     DATE    ID      TEXTID
"       ----    ---     ----    --      ------
SCHEMA  NLDN    2       98337   0       "NLDN DATA OBS UNIDATA FORMAT

"KEY    SCALE   UNITS   DESCRIPTION
"---    -----   -----   -----------

ROWS    1000            "1000 ROWS EACH WITH 1000 COLS

DAY             CYD     "YEAR AND JULIAN DAY
TIME            HMS     "TIME OF FIRST OBSERVATION IN ROW
CMAX                    "LAST COLUMN CONTAINING DATA

COLUMNS 1000            "DEFAULT # OF COLS
DATA                    "START OF DATA SECTION

HMS             HMS     "ACTUAL TIME OF OBSERVATION
LAT     4       DEG     "LATITUDE (DECIMAL DEG)
LON     4       DEG     "LONGITUDE (DECIMAL DEG)
SGNL    1               "SIGNAL STRENGTH (150 measures ~ 30 kamps)
MULT                    "MULTIPLICITY

ENDSCHEMA

Each schema description:

- starts with a line that begins with SCHEMA
- the SCHEMA line defines the 4-character name of the schema; the version
  of the schema (schema definitions can evolve); the date the version
  was created; an ID for the schema (not quite sure what this is!);
  and a textual description as a quote field
- definitions of #ROWS, #COLUMNS, and DATA keys (e.g. DAY, TIME, ..., MULT
  in the listing above).  MD file keys are each 4 characters or less in
  length
- specification of how many significant digits of a key value is to
  be kept in the MD file (all MD file values are scaled integers)
- units of the keys (recognized units are defined in the file FLDWDTH)
- comments about the keys (these comments are not used for anything,
  but are useful for the reader)

Your first attempt of defining a schema was an attempt to replicate
the output format that you would see when listing a schema that
is registered (i.e., has been added to SCHEMA using SCHE) using LSCHE.
This was a good guess, but will not work:

############################################################################

NAME: FLEXP  VERSION:  1  DATE: 2003167  TEXTID: "FLEXPART CO CONCENTRATIONS  
----        -------                                                           
   DEFAULT NUMBER OF ROWS:   33000                     INTEGER ID: 0          
                     COLS:     3                   MISSING DATA VALUE:  -999  
   REPEAT GROUP:  NUMBER OF REPETITIONS:  0                                   
                      STARTING POSITION:  1                                   
                                   SIZE:  3                                   
   NUMBER OF KEYS IN ROW HEADER:     3                                        
                     COL HEADER:     0   STARTING AT POSITION   1             
                    DATA RECORD:     3   STARTING AT POSITION   1             
                                  ----                                        
                                     6  TOTAL                                 

        KEY  SCALE UNIT                                                     
        ---- ----- ----     
        LON    6   DEG      
        LAT    5   DEG      
        CO     7   CON

################################################################################

>I'm not sure what the following parameters are for:
>
>NUMBER OF REPETITIONS (seeing as I only want to read each column once, I
>assume the # of repititions is 0)

The number of repetitions is used for schema like for soundings where the
same set of data are available for multiple levels.

>STARTING POSITION  ??
>SIZE  ???
>INTEGER ID ????
>COL HEADER:    STARTING AT POSITION   ???                                     
>DATA RECORD:   STARTING AT POSITION   ???
>Can you take a look at this SCHEMA and let me know where I'm going wrong?

These are artifacts of the LSCHE output.  The schema example above (DCNLDN)
should go a long way towards explaining what needs to be done.  The
schema I came up with below should galvanize some of the concepts.

I played around with the data file you gave me access to yesterday, and
came up with a schema that, while pretty simple minded, works for what
you want:


"       NAME    VSN     DATE    ID      TEXTID
"       ----    ---     ----    --      ------
SCHEMA  FLXP    1       01168   0       "FLEXPART CO CONCENTRATIONS

"KEY    SCALE   UNITS   DESCRIPTION
"---    -----   -----   -----------

ROWS    90              "90 ROWS

LAT     4       DEG     "LATITUDE (DECIMAL DEG)

COLUMNS 360             "DEFAULT # OF COLS

LON     4       DEG     "LONGITUDE (DECIMAL DEG)

DATA                    "START OF DATA SECTION

CO      2       CON     "COD CONCENTRATION

ENDSCHEMA

This schema defines an MD file that is 90 rows long; each row has
360 columns; and each column has one data point, the CO concentration.

Notice that I chose the LAT as a ROW header; the LON as a COLumn header,
and CO as the data key.  The reason for this was mostly so that the
TXT2MD conversion utility would work without complaint.

What you end up with is a MD file where each ROW represents all data at
a single Latitude, each COLumn represents all data at a single a single
Longitude, and there is exactly one datum for each ROW,COL cell.

I then used this schema to create an output MD file and contour its values
on top of a world map:

- create DCFLXP with the contents above in the ~mcidas/workdata directory.
  If you are doing this as a user other than 'mcidas', then you would
  create this file in your ~/mcidas/data directory

- since I didn't want to change the SCHEMA file that I send out in the
  distribution, I made a copy of it into my McIDAS working directory:

  cd ~mcidas/workdata
  cp ~mcidas/data/SCHEMA .
  redirect.k ADD SCHEMA \".

  If you are doing this as a user other than 'mcidas', do the same steps
  in your McIDAS working directory:

  cd ~/mcidas/data
  cp ~mcidas/data/SCHEMA .
  redirect.k ADD SCHEMA \".

- next, I registered the schema in the copy of SCHEMA that I can modify:

  sche.k DCFLXP

  I verified the schema registry using LSCHE:

  lsche.k FLXP

NAME: FLXP  VERSION:  1  DATE: 2001168  TEXTID: "FLEXPART CO CONCENTRATIONS
----        -------
   DEFAULT NUMBER OF ROWS:   90                        INTEGER ID: 0
                     COLS:    360                   MISSING DATA VALUE:  
-2139062144
   REPEAT GROUP:  NUMBER OF REPETITIONS:   1
                      STARTING POSITION:   3
                                   SIZE:   1
   NUMBER OF KEYS IN ROW HEADER:     1
                     COL HEADER:     1   STARTING AT POSITION   2
                    DATA RECORD:     1   STARTING AT POSITION   3
                                  ----
                                     3 TOTAL

        KEY  SCALE UNIT     KEY  SCALE UNIT     KEY  SCALE UNIT
        ---- ----- ----     ---- ----- ----     ---- ----- ----
        LAT    4   DEG      LON    4   DEG      CO     2   CON
LSCHE: DONE

- the file you gave me FTP access to looked like one long line in Unix:
  there were carriage returns at the end of each logical line, not
  new lines.  TXT2MD expects the textual data file to be in ASCII
  format with each line ending in a new line.  Given this, I translated
  your file into a format I could use:

  tr '\015' '\012' < flexpart_20030604030000 > flexprt.txt

  The 'tr' invocation converts CR characters (octal 015) into NL characters
  (octal 012).

  I wrote the output into a file whose name follows the old DOS 8.3
  name format since that is what TXT2MD demands (this should be changed,
  but...).

- after converting your file, I edited it to add one additional line:

change:

LON    LAT     CO
 -178.5    0.5    1.63
...

to:

LON    LAT     CO
UNIT=*-1 X X
 -178.5    0.5    1.63
...

  The reason for this addition is that I _assumed_ that your LON values
  were using the West negative convention, while McIDAS uses a West
  positive convention.  The *-1 syntax for the first positional value
  for UNIT tells TXT2MD to multiply each value by -1 thus converting
  its sign.  BTW the ability to convert units in the UNIT= line was an
  addition I made to TXT2MD a number of years ago; it does not exist in
  the SSEC version (just in case you find yourself using an SSEC
  distribution of McIDAS and wonder why you can't do unit conversions).

- finally, I was able to run TXT2MD and convert your file:

  <I was still in ~mcidas/workdata, you may well be in ~/mcidas/data>

  mdu.k DEL 1000                <- just to make sure that the output MD file
                                   doesn't exist yet
  txt2md.k flexprt.txt 1000 FLXP

All data values were read in from your text file and stored in MDXX1000.

The next step was to verify that I could contour the data on top of
a navigated display.  In order to do this, I first made sure that
I had the MYDATA datasets defined:

batch.k MYDATA.BAT

Then I contoured the data and saved the output into a GIF (tm) file for you
to take a look at:

mcenv

erase.k
map.k WORLD
ptcon.k MYDATA/PTSRCS.1000 PAR=CO CINT=15 NAV=C LSIZE=5
frmsave.k 1 flexprt.gif

exit

Have a look at the result in:

http://www.unidata.ucar.edu/staff/tom/gifs/owen/flexprt.gif

I guess it is now an exercise for the reader (you ;-) to verify that
the display of the data is correct.

>Thanks!

Please let me know if there is anything that I did that does not make
sense.

>Thanks!

No worries.

Tom

>From address@hidden Wed Jun 18 12:14:43 2003

Hi Tom,
This looks awesome, and I will give it  a try very soon.
Next week I will post near real-time GOES-east and Meteosat IR images on 
the web and will overlay the carbon monoxide data from the FLEXPART 
model.  As far as I know this will be the first realtime display of 
pollution plumes travelling across the Atlatntic and superimposed onto 
geostationary satellite data.  Once everything is working I'll send you 
the URL so you can see the animations.
Thanks!
Owen

Owen R. Cooper
Cooperative Institute for Research in Environmental Sciences (CIRES)
University of Colorado/NOAA Aeronomy Laboratory

NOAA Aeronomy Laboratory, R/AL4
325 Broadway
Boulder CO 80305
ph:  (303) 497-3599
fax: (303) 497-5686