20011218: setup of ldm-mcidas, LDM, XCD, and McIDAS at COFC (cont.)

>From: "James R. Frysinger" <address@hidden>
>Organization: College of Charleston
>Keywords: 200111061842.fA6Igt112242 LDM binary install

Jim,

re: changing pointing to ADDE servers on the fly in MCGUI

>        Oh, that is sooooo nice!

And the list of servers is growing.

re: getting help
>I had used merely '?' on the mcidas command line. I see that I get the
>same results from '^?'.

I don't know what I was thinking about.  The thing to enter _is_ the
question mark.

>I tried kill on both the mcidas command line
>and from the terminal window, using the latter only as a last resort.

As long as the GUI Console mode entry is not wedged, running kill
from the GUI Console is equivalent to running kill from the Unix
command line.  The only difference is that you can not run a 'kill -9'
from McIDAS.

>Perhaps in my efforts I had wedged the program a couple of times. I
>must say that mcidas seems to be one of the most stable programs I have
>encountered yet, especially considering its size!

McIDAS _is_ big, but you havn't seen GEMPAK yet (or have you).  It is
bigger.

>New topic:
>
>        I've made some use of your 'Help' button to speed up the command
>shopping I've been doing. What I have been doing is scouting around to
>get a feel for the kinds of things that mcidas can do and to lay out
>plans to (1) instruct myself for and (2) design a protocol for a study
>I am interested in.

At this point, let me recommend that you work your way through the
McIDAS Learning Guide:

Unidata McIDAS-X 7.80 HomePage
http://www.unidata.ucar.edu/packages/mcidas/780/index.html
  Documentation and Training
  http://www.unidata.ucar.edu/packages/mcidas/780/document.html
    McIDAS-X Learning Guide 
    http://www.unidata.ucar.edu/packages/mcidas/780/mclearn/index.html

This should help fill in the gaps in your understanding of how McIDAS
works.

>Let me tell you where my thoughts are going so that
>you can tell me if I seem to be on the right track. Also, you might be
>able to provide some insights on accomplishing the particular steps I
>think I need to take and point me towards the commands/programs to bone
>up on.

Ready...

>        Here's my arena of interest. I am working on sea breezes in the
>coastal South Carolina area, centered on Charleston, which happens to
>be blessed with a fairly straight, neighboring coast line, thus
>minimizing the horizontal divergence and convergence effects of harbors
>and points of land. It also has a nice flat, gently sloping topograpy
>that will minimize orographic effects. The geographic area of interest
>runs from nearly our border with Georgia (32:29N/80:20W) to Bulls Bay
>(33:12N/79:10W) and from perhaps 30 km offshore to a line in
>midstate that is parallel to the coastline.
>
>        Within this geographic box, I want to archive for later study all the
>available weather information I can lay my hands on. GOES 8 images will
>help with the cloud cover and identification of frontogenisis events.
>The observational data will provide numerical horizontal and vertical
>parameter values that can be used in the study. I have already
>identified some of the statistically correlated causal parameters (and
>wrote a thesis on that, as well as a paper that is being considered)
>but there is more there to be found.

OK, I am with you so far...

>        Since much of this will be a matter of inductive study, an archive of
>data is essential so that various combinations of parameters can be
>studied and the results compared. But I don't want to archive all the
>world's meteorological data. I want to extract the data for the area of
>interest and archive it locally. Still this will be quite extensive,
>but I can migrate hard disk files to CD-Rs periodically to save hard
>drive space.
>
>        The data coming in via LDM and LDM-MCIDAS can probably be archived via
>a batch file running ahead of the scour routine. So that's one thing I
>need to figure out how to do. That seems relatively simple. I can
>figure out the stations in and near my box of interest and archive
>their data, ignoring the rest. Obviously I'll need to include stations
>outside my box of interest to some extent to let contouring routines
>work properly.

Right.

>        Satellite data is harder. Since this data is not stored locally, I
>would need a routine that somehow pulls each GINIEAST product in and
>runs it through MCIDAS to build the image section of interest and then
>store it back in the appropriate form for later display and analysis on
>McIDAS and also by other programs.

If you are really going to use imagery in your study, I would not use
the GINI sectors as they have already been remapped (so the data is
one step away from being original).  For the IR imagery, I would use
the images coming in the Unidata-Wisconsin datastream (LDM MCIDAS feedtype)
since these have not been remapped.  If you need higher than 4 km VIS
images, I would use the GINI data as it is remapped 1 km.  Very soon
now (really) we will be installing a server at SSEC that will give folks
access to full resolution, 10-bit, VIS imagery from both GOES platforms.
This data will be accessible through ADDE as are the other datasets
you see in the ADDE Client Routing Table interface in MCGUI.  That
data should be much better to do analysis with since it has not been
remapped.

>I see that a display centered on
>32:48N/80:00W at display level 1 provides a comfortable margin around
>my area of interest, providing enough perspective to see the synoptic
>cloud patterns. Perhaps a McIDAS command such as AXFORM can be used to
>do this.

You can use the IMGCOPY command in McIDAS to copy over portions
(i.e., sectorize) of larger images into local files.  To do this,
you would need to setup a local dataset and do the copy.

>Hopefully, by cutting out a smaller area, we can afford the
>space to archive each satellite image, in all available bands.

Yes, and if your machine(s) has(have) the capability of buring CD ROMS,
you can cheaply save the data off of your hard disk.  For instance, I
bought an Iomega CD-RW unit ($50) and 100 CD-Rs ($25) that I can now
use to create case studies, archive code, and the like.  This has
turned out to be a very cheap way of getting into the "archive" business.

>(The use
>of information in all bands, coupled with RAOB data might prove useful
>in studying sea-breeze cells; alas, we have no provilers in our area.)

I agree.

>        Our sea-breeze season will be starting in a couple of months so I
>would like to have my batch files and routines in place and to collect
>a year's worth of data (into early November). If you can comment on my
>overall plan of approach and perhaps point me toward some specific
>mcidas commands of interest, that would be very helpful.

To get your feet wet, I suggest playing with a couple of things already
setup on your machine(s): IMGCOPY and the MYDATA/IMAGES dataset.
For example, try the following:

IMGDISP GINIEAST/GE1KVIS STATION=KCHS PLACE=C MAG=-2 -2 EU=IMAGE SU=X ALL=11 
SF=YES REFRESH='EG GRA=(GRA);MAP FILE=OUTLUSAM MCOL=1 WID=1 GRA=(GRA);BAR (GRA) 
SU=X X ORIENT=VER;TE ?LASTFRM (GRA)'

(i.e., use MCGUI to display a GINIEAST VIS image centered over Charleston, SC
(KCHS) blown down by a factor of two and draw a map on it.)

Play with the blow-up/blow-down till you get the regional coverage your
are interested in.  While doing so, remember that the areal extent of the
data that you will be looking at is governed by your blow-up/blow-down
settings AND the size of your image frame.  To reinforce this notion,
try exiting McIDAS and bringing up a session with differently sized image
windows.

Once you figure out the areal extent and magnification (blow-up/blow-down)
you want, you can use IMGCOPY to copy from the remote dataset into your local
one.  Here is an example of a local save of the above image:

IMGCOPY GINIEAST/GE1KVIS MYDATA/IMAGES.3000 STA=KCHS PLACE=C MAG=-2 -2
Beginning Image Data transfer, bytes= 307968
Transferring AREA data outbound, bytes= 308128
IMGCOPY: GINIEAST/GE1KVIS.49 copied to MYDATA/IMAGES.3000
IMGCOPY: done

Now, the sector you are interested in is on your local machine in the
MYDATA/IMAGES dataset in position 3000.  Since MYDATA/IMAGES was setup
so that the dataset position number is a one-to-one match with AREA
file number, you now have a file named AREA3000.  This file will be
located in the directory either pointed to by a McIDAS file REDIRECTion
(if one fits for it) or in the first writable directory in your MCPATH.

A couple of things to note here:

1) if you are doing the IMGCOPY to a system that has the realtime image
   ingest, you must be very careful to NOT write to a position number
   for an image being created by the LDM ingest!  For instance, you
   would _NOT_ do the following:

IMGCOPY GINIEAST/GE1KVIS MYDATA/IMAGES.123 STA=KCHS PLACE=C MAG=-2 -2

   The reason is that AREA0123 will contain realtime GOES-West VIS
   images.  This is by virtue of the DSSERVE command (run from LSSERVE.BAT
   on ingesting machines).  A 'DSSERVE LIST RTIMAGES' invocation will
   show you this setup:

RTIMAGES/GW-VIS          IMAGE AREA 120-129          GOES-West Western US VIS

   Given the setup with real time ingest, you will want to stay away from
   position numbers 1-1999 and 9000-9999 in MYDATA/IMAGES as these
   correspond to images either ingested in realtime (1-1999) or reserved
   for Unidata use (9000 - 9999).  This leaves you with 8000 positions to
   work with, so you have some breathing room for awhile.

2) you can contol the size of the IMGCOPYied image using the SIZE=
   keyword on the IMGCOPY command line.

One other quick idea:  if you get into the business of IMGCOPYing
imagery down to your machine, I would recommend that your script
does the additional work of renaming the IMGCOPYied file to something
that is outside of the default McIDAS image namespace (AREA0001 - AREA9999).
For instance, if the IMGCOPY above was for 19:30 Z, GOES-East data,
you may want to rename AREA3000 to something like 200112181930_goese.vis.

Now, you may want to be able to look at the images you are writing into
your archive (duh :-), but they are no longer accessible through the
MYDATA/IMAGES dataset (which only concerns images in McIDAS AREA format
using McIDAS AREA file naming conventions).  So, I would create another
dataset that can access those files.  For purposes of illustration,
let's call this new dataset BREEZE, and let's group the GOES-East VIS
images in the dataset by the descriptor GE-VIS (you can, of course,
choose other names).  Finally, again for the purpose of illustration,
let's locate the images in the BREEZE dataset hierarchically in
directories under /export/home/mcdata:

GE-VIS  -> /export/home/mcdata/breeze01/GOES-East/1km/VIS
GE-IR   -> /export/home/mcdata/breeze01/GOES-East/4km/IR
GE-39   -> /export/home/mcdata/breeze01/GOES-East/4km/39
GE-12   -> /export/home/mcdata/breeze01/GOES-East/4km/12
 ...

Here are the DSSERVE commands that will setup the dataset access in
McIDAS:

DSSERVE ADD BREEZE/GE-VIS AREA 
DIRFILE=/export/home/mcdata/breeze01/GOES-East/1km/VIS/*.vis TYPE=IMAGE 
"GOES-East 1km VIS images for sea breeze study
DSSERVE ADD BREEZE/GE-IR  AREA 
DIRFILE=/export/home/mcdata/breeze01/GOES-East/4km/IR/*.ir TYPE=IMAGE 
"GOES-East 4km 10.7 um IR images for sea breeze study
DSSERVE ADD BREEZE/GE-39  AREA 
DIRFILE=/export/home/mcdata/breeze01/GOES-East/4km/39/*.39 TYPE=IMAGE 
"GOES-East 4km 3.9 um images for sea breeze study
DSSERVE ADD BREEZE/GE-12  AREA 
DIRFILE=/export/home/mcdata/breeze01/GOES-East/4km/12/*.12 TYPE=IMAGE 
"GOES-East 4km 12.0 um images for sea breeze study

After setting up the dataset definitions through DSSERVE command(s), you
will be able to access the data using ADDE:

IMGDISP BREEZE/GE-VIS STA=KCHS EU=IMAGE ...

A quick comment: You can not copy directly into the BREEZE dataset (the
one using the non-standard McIDAS naming convention) since output for
McIDAS copy commands (IMAGE, POINT, GRID) can only be directed into
datasets using standard McIDAS file naming (AREAnnnn, MDXXnnnn, GRIDnnnn).
So, your script has to do a little more work up front.

>        I would greatly appreciate your thoughts and suggestions on this.

What do you think?  Got the idea?  Give it a try and let me know how
it goes.
   
Tom