20020226: McIDAS dataflow (cont.)

>From: James Frysinger <address@hidden>
>Organization: College of Charleston
>Keywords: 200202210415.g1L4Fbx02565 McIDAS dataflow

Jim,

re: 1 km national N0R composites

>Yeahhhhhhhhh, that is superb!

Yes, we think those composites are pretty nice alright.

>In fact, I think that this may be an excellent 
>data resource to include in my archives for sea breeze studies. The local 
>data come from KCHS's radar in Jedburg and it's location is such that the 
>lower slices do a great job of mapping at about the right height. In fact,
>IMGDISP NEXRCOMP/1KN0R-NAT #?LL PLACE=C EU=BREF REFRESH='EG;MAP H;BAR SU=BREF'
>where ?LL is the college's lat/lon makes an excellent graphic on my 600x800 
>window.

One quick of warning.  What is wrong with the display you made is the
identification of the dBz values in the BAR on the right.  The 1 km
composite contains reflectivity values that range from about -28 to
+75.  The BREF.ST stretch used in the BAR SU=BREF is for radar images
that have reflectivities that range from 0 to 75.

I have just put together a new enhancement that displays echos
correctly, and have been working on the AUX block that the server
generates so that BAR can produce the correct mapping between color and
reflectivity.  The mods to the GINI ADDE server and the new enhancement
will be rolled into the addendum that I will release for McIDAS when I
get finished.  The mods to the GINI server will not be important to you
unless you get hold of the original composite images and then want to
serve them locally.  This is possible, by the way, since compressed
versions the images are being sent in the NIMAGE steam of the IDD.
Compressed, the 1 km national composites drop down to about 800 KB in
size.  Uncompressed, and they balloon up to 14 MB!  Radar images
compress so well since they are devoid of information everwhere except
where there are echos.

Also, the 6 km composites are being sent out in the FNEXRAD feed.
We turned all of this on at the very end of last week, and we have
been tweaking the images ever since.  As soon as we stop adjusting
the knobs (:-), we will announce the images to the community and
let people know that they can access them through ADDE.

>I'll  have to play with mag factors to get the right size IMGCOPY 
>download. I'm guessing at a blow up/down of 0.

Actually, a blowup of 1 and a blow down of -1 are equivalent: no
magnification or shrinking.

>I can play with that to get the right mag factors.

Right, it always takes a little bit of tweaking to get what you want/need.
I know about this, 'cause it always seems like I am tweaking something
or other to get just what I want.

re: raw gini

>OK, I'm confused. I thought that the GINIEAST data I was getting were 
>'remapped and packaged' by your servers, not the raw data.

The GINI images in NOAAPORT are remapped versions of what the GOES
satellites see.  We (Unidata) are doing no remapping of these images.
All of the sectorization and remapping is done back at NESDIS
(SATEPS).  What is being served from the ADDE servers is exactly what
is in the GINI images that come from NOAAPORT.

>In your 18 Dec 2001 13:27:22 -0700 message you said
>"If you are really going to use imagery in your study, I would not use
>the GINI sectors as they have already been remapped (so the data is
>one step away from being original).

Right.  For satellite research, I would use the original GOES scans,
not the GINI remapped sectors.

>>For the IR imagery, I would use
>>the images coming in the Unidata-Wisconsin datastream (LDM MCIDAS feedtype)
>>since these have not been remapped.

This is my point:  the Unidata-Wisconsin images have been sectorized, but
not remapped.

>>If you need higher than 4 km VIS
>>images, I would use the GINI data as it is remapped 1 km.

The reason for this comment was that currently, the average Unidata user
does not have access to high resolution GOES imagery, but they do
have access to those GINI images that are sent in NOAAPORT through
ADDE.

>>Very soon
>>now (really) we will be installing a server at SSEC that will give folks
>>access to full resolution, 10-bit, VIS imagery from both GOES platforms.
>>This data will be accessible through ADDE as are the other datasets
>>you see in the ADDE Client Routing Table interface in MCGUI.  That
>>data should be much better to do analysis with since it has not been
>>remapped."

The machine has been delivered to SSEC and is being installed in their
machine room.  As soon as it is up on their network with the blessing
of their system administrators security-wise, I will be setting up serving
of unremapped GOES data through ADDE.

>Could you elaborate a bit here? Remember, I'm an A-B-C sorta guy.

GINI images are remapped sectors created for the NOAAPORT broadcast.
Since they are remapped, the data values in them are not exactly the
same as those that came from the original GOES scans.  To the casual
observer, the differences may well seem to be insignificant. To the
satellite researcher, the differences could be enough to affect
their usefulness.

>Is it that I should stick with 'MAG=0 0' and use line/element declarations
>to pick my area of interest?

First, a MAG=0 0 gets translated internally to MAG=1 1, but that is
a minor point.  The idea is that if you want the original data
(original to whatever dataset is being served), then do not blow down
or blow up when doing IMGCOPYs.  So, yes, specify MAG=1 1.

re: a number of the GINI images are BIG

>Yeah, I realized that! I could send the price of CD-Rs up if I'm not 
>careful. Seriously, some mental calcs tell me that I can probably get a 
>season's worth of images and MD files on just a few CDs. Not bad!

I agree. Mo data, mo data, mo data...

>Then I (and 
>our students) can spend the winter playing with the data to see why we had 
>those cooling sea breezes on any given summer day.

Exactly.  You would essentially create case studies as collections of
data that you can go back and review anytime you like.

re: running invocations from cron vs running from the McIDAS scheduler

>After I sent this message, I rediscovered the mcenv stuff in the Users 
>Guide. I can see that doing this stuff by script file from cron is the way to 
>go. Also found the section that tells how to give mcidas commands from a 
>shell script (the .k bit).

OK, good.  Check out the mcbatch.sh and mcrun.sh shell scripts that I
include in the distribution.  Again, these are skeletons that you can
copy and modify to get things going.

>You saved me a lot of work, Tom! Looks like, after I tailor the paths to my 
>workstation's setup, I can stick my commands in where indicated in mcrun.sh 
>and do what I need to do for the downloads.

Please make a copy of mcrun.sh into a file of a different name and then
use this copy.  Remember that each time you do a McIDAS install, the
mcrun.sh and mcbatch.sh files included in the distribution will
overwrite the ones from the last distribution.  If you simply use the
ones from the last distribution and don't rename or relocate them, you
will lose your work when you upgrade.

>I'm guessing I can even take care 
>of any renaming/relocation there, too, but if not, I can daisy chain that 
>script to this in my crontab, putting the scripts on the same line separated 
>by a semicolon. Do I have the big flick?

If I understand you, then yes, you have the picture.

>After my big crash this winter, I reinstalled mcidas but never got around to 
>reinstalling LDM (since that had been more of an academic exercise to check 
>linux out). I had recovered the skeleton of the LDM user's directory from the 
>old /home files but it didn't contain area2png or pnga2area. So I sftp'd to 
>'weather' and did 'get' to pull them into my home machine. They don't work 
>here (with either a null argument list or with -h) so I guess I would need to 
>reinstall LDM here first.

Hold on!  The versions of area2png and pnga2area you grabbed from
weather were Sun Solaris executables.  The versions you need for your
home machine are Linux executables.  So, you should grab the
appropriate Linux ldm-mcidas distribution and extract area2png and
pnga2area from it.  For reference, binary distributions of the
ldm-mcidas package can be found in the pub/binary/... directories of
anonymous FTP on our FTP server, ftp.unidata.ucar.edu.

>But I did do those on 'weather' and transferred the 
>'help' results to here. They look like good candidates. Haven't thought about 
>it much but I'm guessing I can do something like
>   ./area2png -l - -f /...../AREA3000 %Y%m%d%H%M\SAT_\RES|\BAND_
>to build a temporary directory with file names that include and start with 
>chronological tagging.

Yes.  Just remember that the files created are compressed.  You can not
display them directly with McIDAS after doing the area2png step.  You
can convert them back into regular AREA files by running area2png's
counterpart, pnga2area.

>I can then do 'ls ... > filename.file' and parse that 
>to use in a script for doing 'mv AREAxxxx name.from.filename.file'.
>
>Just thinking out loud here.......

I am not sure that I follow, but that is not important at this stage.

Later...

Tom