20010709: hdf to area

>From: Darren Gallant <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200107091620.f69GKI102721

Darren,

>I have a 2-d array containing albedo*100% values from GOES-10 Channel 1.
>The array is 501 (lines) by 1501 (samples). The array consists of 501 *
>1501 bytes, i.e. an element of the array contains one byte. The image is a
>GOES-10 Visible.  The sector consists of lines 4500-5000 and samples
>17000-18500. The resolution is approx 1km or the instruments native
>resolution for the visible channel. What information do I need to display
>and navigate this image?

OK, there are two questions here.  Given that you can create a 2D
array of data values, you can pretty easily put those values into a
skeletal AREA file and then display it in McIDAS.

The question related to navigating the image, on the other hand is more
difficult.  I can imagine that one could take an existing GOES-10 VIS
imave of the same vintage and use its navigation as a first guess about
the navigation for your image.  In order to get things right, however, 
you would have to tweak various navigation parameters.  For relatively
simple navigations like Polar Stereographic, Lambert Conformal, Mercator,
and even METEOSAT (which is an idealized satellite projection), this
is pretty easy.  GOES, on the other hand, is a three axis stabilized
platform that has a quite messy navigation model, so the tweaking would
be hard especially if there are no landmarks in the image to tweak to.

>I can import the image into idl as a binary array
>and display it albeit w/o navigation.

You can do the same in McIDAS using MAKEAREA.

>I would like to create a navigated
>Mcidas area file from the binary array. I have attached the binary image 
>converted to GIF. Any assistance would be greatly appreciated.

The second step of importing arrays of image data into an AREA (the
first step is using MAKEAREA) is using MAKNAV to add navigation.  MAKNAV,
however, is not tooled to be able to write GOES navigations.

You mentioned HDF in the email subject line.  Is the original data in
HDF format?  If so, which HDF format (e.g., HDF EOS, etc.)?  The reason
I ask is that Dave Santek of SSEC made the following comment to another
user back on June 14:

(re: is there an HDF to McIDAS converter)

  Yes there is....we have MODIS HDF ADDE servers.
  
  You will need the latest version of McIDAS installed (7.8)
  and the HDF library installed (4.1r3 or 4.1r4).
  
  This will be available from the MUG web page [sometime over
  the next few weeks].

The other reason is that I have had correspondence with a person in Hawaii
about what they use to convert HDF imagery (GOES-8 and GOES-10) to McIDAS
AREA format.  If you are interested in pursuing this with the person
directly, I will dig up his email address and send it along.  If you 
decide to pursue this, I would greatly appreciate you keeping Unidata
Support in the loop by CCing us on your exchanges.  Please let me know.

Tom

>From address@hidden Wed Jul 11 11:07:32 2001
>Subject: [Fwd: HDF to McIDAS Area converter] (fwd)

Tom,

FYI
We are supposed to be getting this converter. I haven't seen any code yet.
In the mean time I have been working on a HDF to area file converter just
in case. 

We're also working on saving the raw (GVAR) files from our Terascan
server. Unfortunately, their software makes this rather difficult. Its
pretty easy to archive TDF or HDF files, but no utility exists to save the
raw pass files. You can configure Terascan to keep the pass files, but you
must find a way to rename the files properly before they are overwritten
and this information is difficult to obtain. This is similar in some
respects to Mcidas AREAXXXX files, except that the XXXX are reserved
beforehand. In terascan you can't tell a AVHRR pass file from GOES pass
file without consulting a registry. Futhermore, the files are named
pathXX and the data/time of the image is not obvious. Moreover, 
diskspace on this system is scare and its a slow machine. So there are
performance issues as well. I don't expect raw GVAR data anytime soon. 

In your last email to me you mentioned importing a 2-D image file into
Mcidas is relatively easy but navigating this image might be difficult.

The HDF file generated by Terascan's function tdftohdf includes what
appears to be the information needed to navigate this image. I'm not sure
its the type of info MAKNAV would use, but I wondering if something in
GVAR.c could use it. I'm thinking of using routines in GVAR.c to create
the areafile directly and hopefully create a navigated image. I'm assuming
GVAR.c, since its designed to work with the raw files, must extract the
GOES position info you mention inorder to derive the navigation for the
areafile. Its this info I think is contained in the HDF header. 

To answer your question, I don't know what HDF EOS is so I'm not sure this
is what Terascan's tdftohdf generates. I do know the HDF files generated
by Terascan contain far more information in the header portion than the
HDF files Anthony Guillory mentions. I think the ascii files he mentions
and Terascan's HDF headers contain similar information. I've attached an
example. Could you look it over and see if it has the information needed
to navigate this image? 


I'm assuming MSFC HDF to area convertor makes use of HDF library routines
to access the image data. This is what I've done so far. Once you can
access the image data it shouldn't be too difficult to write an area file.
Creating a navigated areafile appears to be the tricky part. May hopes are
that my HDF files contain what GVAR.c needs inorder to create a navigated 
area file and that I can extract these lines of code from GVAR.c. 

Of course I can always wait for the MSFC software. 

Thanks for your assistance 

Darren        

Darren R. Gallant               UCAR/JOSS
Programmer/Technician           Joint OfFice Of Science Support
303-497-2634                    P.O. Box 3000
address@hidden                Boulder,CO 80307-30000

---------- Forwarded message ----------
Date: Mon, 25 Jun 2001 10:58:40 -0600
From: Steve Williams <address@hidden>
Reply-To: address@hidden
To: Darren R. Gallant <address@hidden>,
     Gregory J. Stossmeister <address@hidden>,
     Scot M. Loehrer <address@hidden>, Jose Meitin <address@hidden>,
     Ronald A. Murdock <address@hidden>, Steve D. Roberts <address@hidden>
Subject: [Fwd: HDF to McIDAS Area converter]

All,
Looks like good news from Anthony Guillory (NASA/MSFC). I'll proceed
to obtain a copy of the software. Hope this will help us.

Steve

> "Guillory, Anthony" wrote:
> 
> Hi Steve,
> 
> Good to hear from you and yes it has been a long time since we have run into
> each other.
> 
> Yes, we do have an HDF to McIDAS converter. It should in theory work for most
> datasets, except with the caveat that it expects navigation data to be in a
> separate file and different format (ascii). And as for calibration - it
> assumes it is GOES-8 so the cal. is internal to McIDAS so we don't do anything
> with that.  We have a version for GMS that reads coefs. from an ascii file.
> If that's the kind of thing you guys are looking for we can get it to you.
> 
> Ok, now for my question/favor.  We have resurrected CHARM (remember that?).
> See http://wwwghcc.msfc.nasa.gov/charm for an idea of what CHARM (II) looks
> like.  We (Gary Jedlovec and I) are wondering if you have or know of the last
> known location of the CHARM I data (digital or hardcopy).  We like to at least
> put all of it in one location.
> 
> Anthony
> 
> Anthony R. Guillory
> NASA/GHCC
> 320 Sparkman Drive NW
> Huntsville, AL 35805-1912
> (256) 961-7894
> (256) 961-7394 FAX
> address@hidden
>

>netcdf g10.2001163.2015 {
>dimensions:
>       gvar_ch1_line = 501 ;
>       gvar_ch1_samp = 1501 ;
>
>variables:
>       byte gvar_ch1(gvar_ch1_line, gvar_ch1_samp) ;
>               gvar_ch1:long_name = "gvar_ch1" ;
>               gvar_ch1:units = "albedo*100%" ;
>               gvar_ch1:valid_range = '\0', '\377' ;
>               gvar_ch1:_FillValue = '\377' ;
>               gvar_ch1:scale_factor = 0.4000000059604645 ;
>               gvar_ch1:scale_factor_err = 0. ;
>               gvar_ch1:add_offset = 0. ;
>               gvar_ch1:add_offset_err = 0. ;
>               gvar_ch1:calibrated_nt = 21 ;
>       float gvar_ch1_line(gvar_ch1_line) ;
>               gvar_ch1_line:long_name = "gvar_ch1_line" ;
>       float gvar_ch1_samp(gvar_ch1_samp) ;
>               gvar_ch1_samp:long_name = "gvar_ch1_samp" ;
>
>// global attributes:
>               :projection = 0 ;
>               :et_affine = 1., 0., 0., 1., 0., 0. ;
>               :projection_name = "sensor_scan" ;
>               :satellite = "goes-10" ;
>               :sensor = 12 ;
>               :sensor_name = "gvissr" ;
>               :pass_date = 10612 ;
>               :start_time = 200930.991 ;
>               :time_adjust = 0. ;
>               :attitude = 0., 0., 0. ;
>               :sensor_tilt = 0. ;
>               :scan_samples = 30680 ;
>               :scan_rates = 6.986899563318778, 214358.0786026201, 0. ;
>               :miss_algn_mat = 1., 0., 0., 0., 1., 0., 0., 0., 1. ;
>               :use_orbele = 0 ;
>               :step_line_ang = 2.8e-05 ;
>               :step_samp_ang = 1.6e-05 ;
>               :center_line = 7958. ;
>               :center_samp = 15297. ;
>               :interp_points = 30 ;
>               :scan_time = 0., 79.005, 158.01, 237.015, 316.02, 395.025, 
> 474.03, 553.035, 632.04, 711.045, 790.05, 869.0549999999999, 
> 948.0599999999999, 1027.065, 1106.07, 1185.075, 1264.08, 1343.085, 1422.09, 
> 1501.095, 1580.1, 1659.105, 1738.11, 1817.115, 1896.12, 1975.125, 2054.13, 
> 2133.135, 2212.14, 2291.145 ;
>               :x_pos = 15387076.57378964, 15160660.52231571, 
> 14933741.27945716, 14706326.37679931, 14478423.36237856, 14250039.80043246, 
> 14021183.27114768, 13791861.37040948, 13562081.70954868, 13331851.91508957, 
> 13101179.62849664, 12870072.50592097, 12638538.21794609, 12406584.44933326, 
> 12174218.89876701, 11941449.27859869, 11708283.31459114, 11474728.74566206, 
> 11240793.3236272, 11006484.81294291, 10771810.99044899, 10536779.64510974, 
> 10301398.57775599, 10065675.60082592, 9829618.538106307, 9593235.224472033, 
> 9356533.505626557, 9119521.237841414, 8882206.287695441, 8644596.531813681 ;
>               :y_pos = 39256484.18243232, 39344479.26177602, 
> 39431168.47433591, 39516548.94284523, 39600617.83347522, 39683372.35592907, 
> 39764809.76353481, 39844927.3533361, 39923722.46618231, 40001192.48681653, 
> 40077334.84396249, 40152147.01040982, 40225626.50309801, 40297770.88319882, 
> 40368577.75619707, 40438044.77197036, 40506169.62486692, 40572950.05378216, 
> 40638383.84223372, 40702468.8184351, 40765202.85536756, 40826583.87085094, 
> 40886609.82761265, 40945278.73335536, 41002588.64082293, 41058537.64786532, 
> 41113123.89750154, 41166345.57798135, 41218200.92284533, 41268688.21098363 ;
>               :z_pos = 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0. ;
>               :x_head = -0., -0., -0., -0., -0., -0., -0., -0., -0., -0., 
> -0., -0., -0., -0., -0., -0., -0., -0., -0., -0., -0., -0., -0., -0., -0., 
> -0., -0., -0., -0., -0. ;
>               :y_head = 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0. ;
>               :z_head = -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., 
> -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., 
> -1., -1., -1., -1., -1. ;
>               :sensor_tilt_arr = -1.570796326794897, -1.570796326794897, 
> -1.570796326794897, -1.570796326794897, -1.570796326794897, 
> -1.570796326794897, -1.570796326794897, -1.570796326794897, 
> -1.570796326794897, -1.570796326794897, -1.570796326794897, 
> -1.570796326794897, -1.570796326794897, -1.570796326794897, 
> -1.570796326794897, -1.570796326794897, -1.570796326794897, 
> -1.570796326794897, -1.570796326794897, -1.570796326794897, 
> -1.570796326794897, -1.570796326794897, -1.570796326794897, 
> -1.570796326794897, -1.570796326794897, -1.570796326794897, 
> -1.570796326794897, -1.570796326794897, -1.570796326794897, 
> -1.570796326794897 ;
>               :sensor_twist = 3.141592653589793, 3.141592653589793, 
> 3.141592653589793, 3.141592653589793, 3.141592653589793, 3.141592653589793, 
> 3.141592653589793, 3.141592653589793, 3.141592653589793, 3.141592653589793, 
> 3.141592653589793, 3.141592653589793, 3.141592653589793, 3.141592653589793, 
> 3.141592653589793, 3.141592653589793, 3.141592653589793, 3.141592653589793, 
> 3.141592653589793, 3.141592653589793, 3.141592653589793, 3.141592653589793, 
> 3.141592653589793, 3.141592653589793, 3.141592653589793, 3.141592653589793, 
> 3.141592653589793, 3.141592653589793, 3.141592653589793, 3.141592653589793 ;
>               :satellite_roll = 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0. ;
>               :satellite_pitch = 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0. ;
>               :satellite_yaw = 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0. ;
>               :nutation_mat = 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 
> 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 
> 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 
> 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 
> 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 
> 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1. ;
>               :history = "gsubset include_vars=gvar_ch1 range_pairs=4500 5000 
> 17000 18500 instantiate=yes\n",
>    "/gvar/imager/HighRes/g10.2001163.2015/opt/terascan/bin/gvarin 
> input_mode=single instrument=imager co_process=gvar_HIGHRES_subproc %s 
> post_process=more_HIGHRES_pproc %s on_pass_disk=yes pass_number=5 
> block_size=26624 delta_line=1 0 0 0 0 delta_sample=1 0 0 0 0 use_master=no 
> inside=yes master_coverage=0 start_spin=1 num_spins=1973 start_percent=0 
> width_percent=100 calibrate=yes byte_output=yes min_temp=188 temp_step=0.5 
> vis_step=0.4 debug=yes\n",
>    "
" ;
>}

>From address@hidden Wed Jul 11 11:07:32 2001
>Subject: [Fwd: HDF to McIDAS Area converter] (fwd)

Tom,

FYI
We are supposed to be getting this converter. I haven't seen any code yet.
In the mean time I have been working on a HDF to area file converter just
in case. 

We're also working on saving the raw (GVAR) files from our Terascan
server. Unfortunately, their software makes this rather difficult. Its
pretty easy to archive TDF or HDF files, but no utility exists to save the
raw pass files. You can configure Terascan to keep the pass files, but you
must find a way to rename the files properly before they are overwritten
and this information is difficult to obtain. This is similar in some
respects to Mcidas AREAXXXX files, except that the XXXX are reserved
beforehand. In terascan you can't tell a AVHRR pass file from GOES pass
file without consulting a registry. Futhermore, the files are named
pathXX and the data/time of the image is not obvious. Moreover, 
diskspace on this system is scare and its a slow machine. So there are
performance issues as well. I don't expect raw GVAR data anytime soon. 

In your last email to me you mentioned importing a 2-D image file into
Mcidas is relatively easy but navigating this image might be difficult.

The HDF file generated by Terascan's function tdftohdf includes what
appears to be the information needed to navigate this image. I'm not sure
its the type of info MAKNAV would use, but I wondering if something in
GVAR.c could use it. I'm thinking of using routines in GVAR.c to create
the areafile directly and hopefully create a navigated image. I'm assuming
GVAR.c, since its designed to work with the raw files, must extract the
GOES position info you mention inorder to derive the navigation for the
areafile. Its this info I think is contained in the HDF header. 

To answer your question, I don't know what HDF EOS is so I'm not sure this
is what Terascan's tdftohdf generates. I do know the HDF files generated
by Terascan contain far more information in the header portion than the
HDF files Anthony Guillory mentions. I think the ascii files he mentions
and Terascan's HDF headers contain similar information. I've attached an
example. Could you look it over and see if it has the information needed
to navigate this image? 


I'm assuming MSFC HDF to area convertor makes use of HDF library routines
to access the image data. This is what I've done so far. Once you can
access the image data it shouldn't be too difficult to write an area file.
Creating a navigated areafile appears to be the tricky part. May hopes are
that my HDF files contain what GVAR.c needs inorder to create a navigated 
area file and that I can extract these lines of code from GVAR.c. 

Of course I can always wait for the MSFC software. 

Thanks for your assistance 

Darren        

Darren R. Gallant               UCAR/JOSS
Programmer/Technician           Joint OfFice Of Science Support
303-497-2634                    P.O. Box 3000
address@hidden                Boulder,CO 80307-30000

---------- Forwarded message ----------
Date: Mon, 25 Jun 2001 10:58:40 -0600
From: Steve Williams <address@hidden>
Reply-To: address@hidden
To: Darren R. Gallant <address@hidden>,
     Gregory J. Stossmeister <address@hidden>,
     Scot M. Loehrer <address@hidden>, Jose Meitin <address@hidden>,
     Ronald A. Murdock <address@hidden>, Steve D. Roberts <address@hidden>
Subject: [Fwd: HDF to McIDAS Area converter]

All,
Looks like good news from Anthony Guillory (NASA/MSFC). I'll proceed
to obtain a copy of the software. Hope this will help us.

Steve