[McIDAS #TPD-549903]: AREA to NETCDF conversion

Hi Jingfeng,

re:
> I'm glad to get your quick response.

No worries.

re: what user is your script running as

> If I understand it correctly, I was running as a regular user, not as the 
> root.

I was really asking if you were running as a regular user or the user 'mcidas'.

Question:  what version of McIDAS are you running?  (output of SEE VERSION.TXT
           from a McIDAS command and text window)

re: are your trying to run the scrip out of a cron job

> I am not sure what a cron job is. I run the shell script executable called
> "goes7_vis_area_to_ncdf.sh" as a regular job,
> 
> $ ./goes7_vis_area_to_ncdf.sh

OK.

What are the environment settings when you run your shell script (output
of 'env')?  This is very important!

> which went through some thousand files located at a sub-directory as indicated
> in the script.

OK.

re: how is your NETCDF dataset setup

> The VIS data from GOES7 was downloaded from NOAA.GLASS web page. The data sets
> are in AREA format with names like "goes07.1994.225.0001.AREA_VIS". Since I
> only need the data covering a limited domain in Amazon, the first step was to
> open a McIDAS window to view the image and select the desired domain. An
> original data file was copied to sub-directory call ./mcidas/data where a file
> call RESOLV.SRV looks like this:
> 
> N1=GOES7,N2=VIS,TYPE=IMAGE,RT=N,K=AREA,R1=2001,R2=2500,C=GOES-East Band 1 - 
> Visible,
> N1=GOES7,N2=IR,TYPE=IMAGE,RT=N,K=AREA,R1=2501,R2=3000,C=GOES-East Band 2 - 
> SWIR,
> N1=NETCDF,N2=ALBEDO,TYPE=IMAGE,RT=N,K=NCDF,R1=3001,R2=3500,
> N1=NETCDF,N2=TEMPERATURE,TYPE=IMAGE,RT=N,K=NCDF,R1=3501,R2=4000,
> N1=TEMP,N2=ALBEDO,TYPE=IMAGE,RT=N,K=AREA,R1=6001,R2=6020,
> N1=TEMP,N2=TEMPERATURE,TYPE=IMAGE,RT=N,K=AREA,R1=6021,R2=6060,

Very good.

Assuming that ~/mcidas/data/RESOLV.SRV is the version of RESOLV.SRV that
is being used by the McIDAS routines run in your script (it may not be depending
on how your environment is setup), this tells me how the NETCDF/ALBEDO dataset
is defined.

Again, assuming that your environment is such that ~/mcidas/data/RESOLV.SRV 
really
is the ADDE server mapping table that is being used, it also tells me how your 
GOES7/VIS
dataset is defined.  In your ~/mcidas/data/RESOLV.SRV, GOES7/VIS is defined to 
be
the set of images in McIDAS AREA format whose names range from AREA2001 - 
AREA2500.
GOES7/VIS.131 refers to the 131st element of the set; it looks like it will 
reference
the file AREA2131, not AREA0131.  If your script really is using 
~/mcidas/data/RESOLV.SRV,
the portion of your script that does the copy is incorrect:

cp ${files} ~/mcidas/data/AREA0131

Instead, it should be:

cp ${files} ~/mcidas/data/AREA2131

Since you did not say that the resulting output was the same for each file
processed, I am guessing that a different copy of RESOLV.SRV is being used
by the McIDAS routines being run in your script.  To determine if this is
the case, please do the following:

<as the same user that is running the Bourne shell script>
-- start a McIDAS session and run:

DMAP RESOLV.SRV

-- also run:

DSSERVE LIST GOES7

Please send me the output of each command.

> According to RESOLV.SRV, the original data file, say "goes07.....VIS" was
> copied to ./mcidas/data/ called VIS.131.

Actually, the GOES7/VIS entry from the RESOLV.SRV above references files
whose names range from AREA2000 - AREA2500.

> In MciDAS, I used IMGDISP to view the
> visible image and select a sub-domain, and then copied that part of data set
> into a new data file called VIS.132 using the command "IMGCOPY GOES7/VIS.131
> GOES7/VIS.132 LATLON=-7 68 PLACE=ULEFT RES=1 SIZE=1000 1000".

OK.  This will copy the image in element GOES7/VIS.131 to the element 
GOES7/VIS.132.
The input file should be AREA2131 and the output file should be AREA2132.

> The next command
> "IMGREMAP GOES7/VIS.132 GOES7/VIS.133 PRO=MERC RES=1 SIZE=1000 1000" did
> projection and saved the projected image (over the desired domain) as VIS.133.

The input file here is AREA2132 and the output is AREA2133.

> As the last step, VIS.133 was converted into NETCDF format using "IMGCOPY
> GOES7/VIS.133 NETCDF/ALBEDO.1 SIZE=1000 1000".

Yes, AREA2133 will be converted into the first element of the NETCDF/ALBEDO
dataset.

> In summary, the netcdf data set was created using McIDAS "IMGCOPY" function.

Yes, but it is likely being created from a file that is different than the
one you think.  This would account for you getting non-zero values from
a VIS file when it should be dark.

re: how is your GOES7 dataset setup

> I don't know how GOES7 dataset setup. I used it as is.

If ~/mcidas/data/RESOLV.SRV is being used, then your output above told me
how the dataset is defined.

re: does your GOES7/VIS.133 image (created by the imgremap.k execution) look 
correct

> It looks good to me viewing in McIDAS.

Yes, but does the image change for each iteration of the DO loop in your script?

re: what _exactly_ do you mean

> That's how I realized I got problem.  At night time I expect to see nothing
> from VIS channel, but I got the converted VIS dataset in NETCDF format with
> values up to 192 (maximum is 255, see the middle panel of attached figure),
> which makes no sense.

I agree.

> In addition, all converted VIS images are exactly the
> same as shown in the middle panel of the figure

Ah Ha!  So my theory about what is going on is, in all probability, correct!
You are converting the contents of the same file to a netCDF over and over 
again.

> So I conclude that the converted VIS image in NETCDF is incorrect.

The conversion is correct.  What is incorrect is your script.

> I hope my explanation is clear. Let me know if it does not. Many thanks.

Your explanation was very clear, thanks!

The question to you now is:  does my explanation make sense to you?

I think that the problem is being caused by your inexperience in McIDAS ADDE
dataset definitions.  It would be useful for you to look through the online
help for the McIDAS ADDE dataset definition command, DSSERVE (HELP DSSERVE).
It would also be helpful if you worked your way through the online McIDAS
Learning Guide section on creating ADDE datasets:

http://www.unidata.ucar.edu/software/mcidas/2006/mclearn/toc.html

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TPD-549903
Department: Support McIDAS
Priority: Normal
Status: Closed