20040121: full resolution GOES data through McIDAS ADDE

>From: Tiziana Cherubini <address@hidden>
>Organization: University of Hawaii
>Keywords: 200401220117.i0M1Hfp2025292 GOES 10-bit data access

Hi Tiziana,

>we are currently receiving the three channels GOES10
>satellite data through ldm (8 bit resolution).

The following is for our tracking system if nothing else:

There are now all 5 imagery channels in the IDD.  The short IR (3.9 um)
and long IR (13.3 um for GOES-East and 12.0 um for GOES-West) channels
was added at the beginning of January.  The IDD "broadcast" frequency
for all imager channels was also increased from hourly to half-hourly.
The data still, however, contain 1-byte values and the oversampling in
the horizontal dimension has been removed.

>For our assimilation pourposes we'd like to have access to the
>GOES10, 10 bit resolution data (all five channels).

OK.

>Last week I meet Tom Yoksas in Seattle at the AMS and he told
>me that those data are disseminated through IDD.

Not quite.  I commented that the full resolution data is available
through remote access by McIDAS and the Unidata IDV using its ADDE.  I
suggested that you contact me to follow-up on where you could get this
data in a timely manner for your assimilation use.

>       I'd would appreciate if you could explain us what is
>the procedure to have access to the 10 bit resolution data.

I know of two locations where one can get the full resolution GOES data
by remote McIDAS ADDE access:

- a Unidata machine hosted in the Data Center at the SSEC at
  the University of Wisconsin-Madison

- the SATEPS office of NOAA outside of Washington, D.C.

There may be more sites that have the data, but these are the best I
can do at the moment.

Access to the imagery at both locations above depend on ADDE access to
dataservers at those locations.  Any ADDE-enabled application will give
you access to that data.  Two of the applications distributed and
supported by the UPC are: McIDAS and the Unidata IDV.

If you are familiar with McIDAS concepts of ADDE access to data, the
following will make sense to you:

ADDE server - unidata2.ssec.wisc.edu
ADDE Group  - WEST

In a McIDAS-X session, you would "point" at the server for the dataset
and then be able to view, copy, etc. the data remotely:

DATALOC ADD WEST unidata2.ssec.wisc.edu
DSINFO IMAGE WEST

        Dataset Names of Type: IMAGE in Group: WEST

Name         NumPos   Content
------------ ------   --------------------------------------
ALL           2500    GOES-10 all imager data
CONUS         2500    GOES-10 continental U.S.
FD            2500    GOES-10 full disk
NH            2500    GOES-10 northern hemisphere
OTHER         2500    GOES-10 misc. (E.g. SRSO 1-minute imagery)
SH            2500    GOES-10 southern hemisphere
TEST          2500    GOES-10 all imager data

DSINFO -- done

IMGLIST WEST/CONUS.-10
Image file directory listing for:WEST/CONUS
 Pos Satellite/         Date       Time      Center   Band(s)
     sensor                                 Lat  Lon
 --- -------------  ------------  --------  ---- ---- ------------
5235  G-10 IMG      23 JAN 04023  20:45:00    33  132 1-5
5234  G-10 IMG      23 JAN 04023  20:30:00    26  140 1-5
5233  G-10 IMG      23 JAN 04023  20:15:00    33  132 1-5
5232  G-10 IMG      23 JAN 04023  20:00:00    26  140 1-5
5231  G-10 IMG      23 JAN 04023  19:45:00    33  132 1-5
5230  G-10 IMG      23 JAN 04023  19:30:00    26  140 1-5
5229  G-10 IMG      23 JAN 04023  19:15:00    33  132 1-5
5228  G-10 IMG      23 JAN 04023  19:00:00    26  140 1-5
5227  G-10 IMG      23 JAN 04023  18:45:00    33  132 1-5
5226  G-10 IMG      23 JAN 04023  18:30:00    26  140 1-5
5225  G-10 IMG      23 JAN 04023  18:00:00     0  140 1-5
IMGLIST: done

IMGDISP WEST/CONUS BAND=1 REFRESH='EG;MAP SAT' MAG=-7 -14
Beginning Image Data transfer, bytes= 461408
IMGDISP: loaded frame  2
EG;MAP SAT
IMGDISP: done
Erased graphic frame(s) 2-2
MAP: Completed frame 2


And so on.  What you would be looking to do is run a shell script that
runs the necessar McIDAS commands at the appropriate time to copy the
full resolution wavelength channel that you want into a local file.
Here is a simple exaple of setting this up from the 'mcidas' account:

DATALOC ADD WEST unidata2.ssec.wisc.edu

DSSERVE ADD MYDATA/IMAGES AREA 1 9999 "All images in McIDAS AREA format

IMGCOPY WEST/CONUS MYDATA/IMAGES.2000 BAND=1 LATLON=19 155:30 SIZE=1000 1000

This copies the VIS channel (BAND=1) to a local file (MYDATA/IMAGES.2000
will be the file AREA2000 by virtue of how the dataset was defined by
the DSSERVE command) that is 1000 lines "tall" and 1000 elements
"wide".  The image file created will have the full 10-bit resolution
and the oversampling in the horizontal dimension.

Now, it would be up to you to determine what coverage you want, and
where you want the resultant image centered.  I can work with you on
this.

If you continue to work in the same account, you will not have to
continue to set the pointing at the source dataset (the DATALOC
command), now will you have to continually redefine the local dataset
(DSSERVE command).  So, if you continue to work in the 'mcidas' account
as I mentioned at the beginning of this note, you could easily roll the
one McIDAS command into a cron-initiated shell script.  To make this
easy for folks, I include two example scripts in my McIDAS
distributoin.  These will be found in the ~mcidas/data directory after
the installation:

mcrun.sh
mcbatch.sh

'mcrun.sh' will run a series of McIDAS commands for you after setting
the needed McIDAS environment variables.  'mcbatch.sh' runs a McIDAS
BATCH file after setting the needed McIDAS variables.  To use the
scripts, your job is to:

1) make a copy of the script naming it something else (so your work
   won't get overwritten by a new McIDAS installation)

2) edit your copy of the script and set the McIDAS environment
   variables to match your McIDAS installation (this can be as
   simple as doing nothing if you follow the Unidata McIDAS installation
   instructions)

3) kicking the script off by cron at the appropriate time(s) during
   the hour for the imager band(s) you want

Again, I can work with you on getting this working.

Since I belive that the University of Hawaii has a relationship with
NOAA, I recommend that you also pursue getting access to the McIDAS
ADDE server that they maintain within the SATEPS group.  As a first
step, I would contact Brian Hughes <address@hidden> and explain
that I told you about the ADDE servers that SATEPS maintains; that the
UH has a relationship with NOAA; and that you had been led to believe
that you could get ADDE access to the datasets that they maintain for
your research/educational activities.  If Brian is still the
appropriate contact, he will then forward you a form that must be
filled out before you can get the access.  If Brian is no longer the
appropriate contact, he will either forward your request to another
person, or send you an email about who to contact.

One more dataset that you will most likely be interested in.  NASA has
an ADDE server site that has global MODIS data.  They recently opened
up this site to whoever wants the data.  Since you are looking for high
resolution imagery, it strikes me that the MODIS data would be a
logical dataset to investigate.

>Your help is greatly appreciated.

No worries.

>Aloha,

It was nice to meet you in Seattle! Cheers,

>Tiziana.
>---------------------------------------------------------------
> Tiziana Cherubini, Ph.D.             Department of Meteorology
> Research Meteorologist               University of Hawaii
> Mauna Kea Weather Center             2525 Correa Rd. HIG 367
> http://mkwc.ifa.hawaii.edu           Honolulu, Hawaii 96822
>
> tel.: (808) 956-4593                 fax : (808) 956-2877
>---------------------------------------------------------------

Tom Yoksas
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.

>From address@hidden Fri Jan 30 17:10:33 2004

Hi Tom,

    thanks a lot for your detailed email. I'll start working on
this very soon. I'll contact you again if I run (and it always
happen, I'm afraid) into some problems.
    Thanks a lot again. It was nice for me too meeting you in
Seattle.

Aloha,

Tiziana.

---------------------------------------------------------------
 Tiziana Cherubini, Ph.D.             Department of Meteorology
 Research Meteorologist               University of Hawaii
 Mauna Kea Weather Center             2525 Correa Rd. HIG 367
 http://mkwc.ifa.hawaii.edu           Honolulu, Hawaii 96822

 tel.: (808) 956-4593                 fax : (808) 956-2877
---------------------------------------------------------------