20011026: 20011025: GINI conversion to AREA (cont.)

>From: Jason Levit <address@hidden>
>Organization: Center for Analysis and Prediction of Storms, University of 
>Oklahoma
>Keywords: 200110222135.f9MLZn102781 McIDAS-X ADDE GINI AREA

Jason,

re: selecting remap targets
>  Ok, this make sense.  I'm assuming we have to carefully select a good
>image, because the GVAR nav data changes from image to image?  

Yes, sort of.  Selecting a well navigated target cuts down on one source
of error in the remap.

>  Actually, we were trying to read data from a McIDAS AREA format that
>contained TANC information.  This was done after the re-mapping from
>GINI to McIDAS AREA.

The process of doing an IMGCOPY from one dataset to another does not
remap the image.  It simply copies it.  Perhaps we are misunderstanding
each other because we are not viewing the various operations the same.
Remapping in McIDAS is the process of converting one projection to another.
an IMGCOPY from a GINI dataset to an AREA dataset does not do anything
with the navigation information other than reformulate it to fit the
symantics of the destination dataset.  An IMGREMAP, on the other hand,
can change the navigation.

>  Is it possible that we're talking about two different kinds of GINI data?

No.

>The GINI data I'm using is directly from the NOAAPort feed, and has not
>been through McIDAS yet.

The GINI data that I demonstrated access to in my last email is identical
to what you are receiving in your NOAAPORT ingest system.  The reason
for this is that we are receiving the data through our NOAAPORT ingest
system.  The only thing that was done in McIDAS was the writing of an
ADDE server that understands the format of the GINI data.  This then
allows one to use all of the existing McIDAS tools (and they are
considerable) to display/analyze/copy/remap the GINI data.

>Are you talking about GINI data that is contained
>in McIDAS AREA format?

No.  The GINI imagery served are in their original format, just like the
ones on your system.

>I'm probably just confused (happens often).

I am sure that it is just the newness of working with McIDAS.  An
opinion of mine developed over time is that a lot of people simply
don't "get" McIDAS at first.  After working with it, however, people
usually discover just how powerful it can be in their research
activities.  This opinion was borne out by observing just how many
people included McIDAS display output in their talks at the recently
held AMS Conference on Satellite Meteorology.

re: which commands were you using
>  Sure, here's the commands I used:
>
>  imgcopy.k GINICAPS/AREA MYDATA/IMAGES.1000 SIZE=ALL
>
>  (GINICAPS/AREA contains a McIDAS AREA file we received from SPC)
>
>  imgremap.k GINICAPS/GINI MYDATA/IMAGES.1000
>
>  (GINICAPS/GINI contains a raw GINI file we received through NOAAPort)

In the above, there was no effort made to specify the type of image
being copied or remapped.  By type, I mean the originating satellite
platform (e.g., GOES-East or GOES-West), or band (e.g., 0.65 um visible,
3.9 um infrared, 6.8 um water vapor, 10.7 um infrared, or 12.0 um
infrared).  So, if ALL of the images you get from SPC are in
GINICAPS/AREA and ALL of the GINI images you get from NOAAPORT are in
GINICAPS/GINI, then the images that will be used in the commands above
will be variable.  One time the input SPC image could be a VIS (0.65 um)
another time it could be a WV (6.8 um), etc.  The same comment holds
true for the GINI images.

I strongly suggest that you setup more specific datasets for both sets
of images.  What I have in mind is a clearer separation of images
by originating platform and band.  The example I sent you last night
demonstarted what I have in mind.  For instance, 

GINIALL/GE4KIR - the set of all GOES-East 4 km IR (10.7 um) images in
                 the ADDE group GINIALL

GINIALL/GW1kVIS - the set of all GOES-West 1 km VIS (0.65 um) images
                  in the ADDE group GINIALL, etc

The reason for the GINIALL group on adde.ucar.edu is that we keep 15
days of GINI images online at all times.  The images for "today"
(where "today" is the current day from 0Z until 23:59:59Z) are
split into three ADDE datasets: GINICOMP, GINIEAST, and GINIWEST.
This is simply an organizational construct.  If we are talking
about current day images, then the following equivalences are
true:

GINIALL/GW1KVIS <-> GINIWEST/GW1KVIS
GINIALL/GE4KIR  <-> GINIEAST/GE4KIF
 etc.

Splitting the full set of GINI images into three datasets allows the
end user to "point" at different servers (point meaning
'DATALOC ADD GINIEAST xxx', 'DATALOC ADD GINIWEST yyy', etc.)
to get their data.  This spreads the data request around to different
servers.  Shares the "wealth" (service) as it were.

I would be most happy to help you setup your ADDE datasets into
logical partitions of the data.  This could be done very quickly,
and it would get you on your way.  Please let me know if you are
interested.

>  Results after imgremap.k command:
>
>  imgremap.k: The portion of the image requested does not exist
>  imgremap.k: Failed to open source image for read

This is because you were probably trying to remap a GINI image from
GOES-West into an SPC image from GOES-East.  In fact, it could be the
case that the GINI image tried in the remap was one over Hawaii and the
destination was over the Eastern US.  Since there are no overlaps in
data points between the two, IMGREMAP will tell you so (the portion of
the image does not exist) and exit.

re: CAPS has existing code to convert a GVAR navigated ...
>Our code currently reads the McIDAS AREA files, converts the data into
>albedo, then is read into a cloud analysis scheme an the numerical model.

OK.

>I was noticing this yesterday as well, that it appears we could create
>the model grid in McIDAS, and re-map the satellite data into the model
>grid.

Satellite data can be sampled into a model grid, yes (I want to get away
from the use of re-map since it is not actually what is going on).

>If that data could be made into a netCDF file, then all we'd need
>to do is write a little routine to read the netCDF file, and we could bypass
>a lot of the infrastructure we describe here.

Hmm...  McIDAS has an ADDE output image server that can write image
data into a netCDF, but it does not have an ADDE output grid server
that can write into a netCDF.  It has the input netCDF server, but that
is not exactly what you are looking for.  The work necessary to write
a McIDAS grid into an output netCDF wouldn't be too much, so the project
is not a large one.

>Again, I really appreciate all of your patience and help.  You're helping
>to answer questions that have plagued us for years, and we've had a hard
>time tracking down documentation.  Thanks a million - really!

Hopefully, the trials in tracking down documentation are not McIDAS related.
All McIDAS documentation is online and viewable by anyone (open standards).
TeraScan data, on the other hand, is not well documented.

>  I'll work through the example you mentioned when I get to work (I'm at
>my Windows machine at home) and let you know how it goes.

Fair enough.  Again, I am willing to setup your ADDE datasets for
SPC and GINI data so that you don't keep running into remap errors
like the one you describe above.

Please let me know if _anything_ I have tried to explain is not clear
enough.  Once you understand the McIDAS setup, I think you will see
that the package can provide you with a set of tools that will make
your work easier, not harder.

Tom

>From address@hidden Fri Oct 26 10:19:09 2001
>Subject: Re: 20011026: 20011025: GINI conversion to AREA (cont.)

re: specific setup for datasetx

  Acutally, the GINICAPS/AREA and GINICAPS/GINI only contain each one file
right now for testing, and both are GOES-08 images.  Here's my lines in
"CAPSGINI.BAT":

DSSERVE ADD GINICAPS/GINI       GINI    TYPE=IMAGE DIRFILE=/work/cube/test/gini/
*.vis "JASON TEST

DSSERVE ADD GINICAPS/AREA       AREA    TYPE=IMAGE DIRFILE=/work/cube/test/area/
goes08vis_* "JASON TEST

re: offer of help setting up datasets

  We certainly welcome any help we can get.  I'll talk about this with
Kevin Thomas (he's the new data ingest guru) once we're done testing
all of this out.  I'm sure he'd appreciate insight!  Let's make sure
we can get this working before taking on the set-up, if that's ok.
I agree with you about setting up different data sets, and we'll certainly
do this with you once we get operational and real-time.

re: images were probably of a different type

  That makes sense, though both images are GOES-08.  Here's their information:

mcidas@cube$ imglist.k GINICAPS/GINI FORM=EXP
Image file directory listing for:GINICAPS/GINI
 Pos Satellite/         Date       Time      Center      Res (km)   Image_Size
     sensor                                 Lat  Lon    Lat   Lon
 --- -------------  ------------  --------  ---- ----  ----- ----- ------------
   1  G-8 IMG       25 OCT 01298  16:45:00    41   88
    Band: 1   0.65 um Visible - Cloud Cover              1.0   1.0  5120 x 5120
     proj:    0 created: 2001298 194011  memo: GINI:: Sat 11  SecID  1  ChID  1
     type:VISR     cal type:BRIT
     offsets:  data=     768 navigation=  256 calibration=    0 auxillary=    0
     doc length:   0   cal length:   0   lev length:   0 PREFIX=   0
     valcod:          0 zcor:  0 avg-smp: N
     start yyddd: 2001298  start time:164500  start scan:    0
     lcor:    1  ecor:     1  bytes per pixel: 1  ss: 70
     Image Center Point Res (derived)  Lat:   0.98 (km)  Lon:   0.98 (km)

mcidas@cube$ imglist.k GINICAPS/AREA FORM=EXP
Image file directory listing for:GINICAPS/AREA
 Pos Satellite/         Date       Time      Center      Res (km)   Image_Size
     sensor                                 Lat  Lon    Lat   Lon
 --- -------------  ------------  --------  ---- ----  ----- ----- ------------
   1  G-8 IMG       25 OCT 01298  17:02:00    36   96
    Band: 1   0.65 um Visible - Cloud Cover              1.0   2.0  2800 x 4200
     proj:    0 created: 2001298 171330  memo: RT GVAR
     type:VISR     cal type:BRIT
     offsets:  data=    2816 navigation=  256 calibration=    0 auxillary=    0
     doc length:   0   cal length:   0   lev length:   0 PREFIX=   0
     valcod:          0 zcor:  0 avg-smp: N
     start yyddd: 2001298  start time:170142  start scan:  372
     lcor: 3060  ecor:  8061  bytes per pixel: 1  ss: 70
     Image Center Point Res (derived)  Lat:   1.47 (km)  Lon:   1.33 (km)

imglist.k: done

re: trials in tracking down documentation

  Yes, you are correct, it's actual documentation about the numbers in
the files, and not McIDAS.  After some searching on the web, we've finally
found some documents that describe the formats.  Basically, it's finding
documentation on the mathematics of the orbits, etc., so we can correctly
re-map the data to the numerical model grid that drove us nuts ("Gould"
number info and how to use it was weird).
 
re: were explanations not clear enough

  Your explanations have been extremely clear.  Thanks again.  As you say,
once I get past the learning curve I'm sure I'll be using McIDAS a whole
lot more...I love it's display system!

  Jason

----------------------------------------------------------------------------
Jason J. Levit, N9MLA                  Research Scientist,
address@hidden             Center for Analysis and Prediction of Storms
Room 1022                            University of Oklahoma
405/325-3503                         http://www.caps.ou.edu/