[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[netCDF #OML-578835]: AREA to NetCDF conversion
- Subject: [netCDF #OML-578835]: AREA to NetCDF conversion
- Date: Tue, 18 Sep 2007 14:55:56 -0600
Hi Tom,
re:
> Hey Tom... I noticed you must have generated all those netcdf
> files on cacimbo. Did you do that with a script or individually?
As you found out later (next email), I did this with a script last night.
> If it was a script, can you tell me where it is? IMGCPY seems
> pretty straightforward, but I'm not clear about getting the
> images into an ADDE dataset.
I created the script 'area2ncdf.sh' in the ~mcidas/workdata directory
as an edited copy of ~mcidas/data/mcrun.sh:
<as 'mcidas'>
cd $MCDATA
cp ../data/mcrun.sh area2ncdf.sh
-- edit area2ncdf.sh and put in the IMGCOPY command
I had not written to you about this because I intended to enhance the
script to create more descriptive output file names than the ones
produced currently. If you do not need this enhancement, I will stop
now.
> I turned the netcdf files over to my student, and the look almost
> identical to the G8 file we created with AreaToNetCDF
> (which is good!).
Yes, this is very good :-)
> I figured this would be something that we would spend a couple
> of weeks just trying to figure out how to do. Thanks for jumping
> on there!
No worries. I'm glad to help if/when I can.
> Actually, I think I found it. I saw your script in the workdata directory.
> Thanks!
Exactly. Now here is what I did as a precursor to creating and running the
script:
<as 'mcidas'>
cd /data/goesarchive
mkdir AREA netcdf
mv goes12*.BAND_04 AREA
cd $MCDATA
dsserve.k ADD GARCH/G12IR AREA DIRFILE=/data/goesarchive/AREA/goes12.\*.BAND_04
\"GOES-12 10.7 um Archive images
dsserve.k ADD GARCH/NCDF NCDF 1 9999 TYPE=IMAGE \"netCDF GOES image files
redirect.k ADD NCDF\* \"/data/goesarchive/netcdf
-- test the input dataset setup:
imglist.k NCDF/G12IR.ALL
-- test the conversion from AREA to netCDF:
imgcopy.k GARCH/G12IR.1 GARCH/NCDF.1 SIZE=674 1188 BAND=4 UNIT=TEMP
/home/mcidas/mcidas2007/netcdf/ncdump/ncdump
/data/goesarchive/netcdf/NCDF0001.nc
What I think is lacking in the 'area2ncdf.sh' script is:
- getting the size of the input image so that the SIZE= keyword specification is
not hardwired
- getting a number of useful pieces of information from the input image (e.g.,
date,
time, etc.) and using them in an informative name for the output file name
For instance, I would have preferred to change the output file names after
their
creation to something like:
goes12_20030429_0245_band4.nc
etc. In order to do this, I have to sharpen up my Bourne shell scripting
skills :-)
The information I was wanting to use comes from an expanded listing output.
For
instance:
imglist.k GARCH/G12IR.1 FORM=ALL
Image file directory listing for:GARCH/G12IR
Pos Satellite/ Date Time Center Res (km) Image_Size
sensor Lat Lon Lat Lon
--- ------------- ------------ -------- ---- ---- ----- ----- ------------
1 G-12 IMG 29 APR 03119 02:45:00 0 75
Band: 4 10.7 um IR Surface/Cloud-top Temp 16.12 9.15 674 x 1188
proj: 0 created: 2007241 64416 memo: RT GVAR
type:GVAR cal type:RAW
offsets: data= 3328 navigation= 256 calibration= 2816 auxiliary= 0
doc length: 228 cal length: 0 lev length: 0 PREFIX= 232
valcod: 119024500 zcor: 0 avg-smp: N
start yyddd: 2003119 start time: 24514 start scan: 306
lcor: 2469 ecor: 5881 bytes per pixel: 2 ss: 78
Resolution Factors (base=1): Line= 16.0 Element= 16.0
imglist.k: done
The output file would be renamed after its creation by imgcopy.k using
something
like (pseudocode):
mv /data/goesarchive/netcdf/NCDF0001.sh
/data/goesarchive/netcdf/goes12_20030429_0245_band4.nc
etc.
Now on to other comments:
I was taking a look at how you are processing data using LDM's pqact, and I
noticed that
in some cases you are processing the same data (e.g., NIMAGE) three times.
Evidently
you want to store the data for use in GEMPAK and also write it off to some
larger
store. Here is one instance of the latter case:
NIMAGE ^TIGQ52 .... ([0-3][0-9])([0-2][0-9])([0-5][0-9])
FILE -close
data/gempak/images/sat/goeseast/w_atlantic/06sounding/WATSND06_(\1:yyyy)(\1:mm)\1_\2\3
There is nothing wrong in processing the same data more than once in different
ways, but
in the NIMAGE and UNIWISC (aka MCIDAS) cases, you could actually be stepping
out the output
data files. Consider:
-- pqact.conf line 1967
MCIDAS ^pnga2area Q. (U[^ACXY1]) (.*) (.*)_IMG (0.65)um (.*) (........) (....)
PIPE -close
/usr/local/ldm/ldm-mcidas/bin/pnga2area -vl
/usr/local/ldm/logs/ldm-mcidas.log
/usr/local/ldm/data/gempak/images/sat/\3/\5/VIS/VIS_\6_\7
-- and pqact.conf line 3428
UNIWISC ^pnga2area Q. (U[^ACXY13]) (.*) (.*)_IMG (0.65)um (.*) (........) (....)
PIPE -close
/usr/local/ldm/ldm-mcidas/bin/pnga2area -vl logs/ldm-mcidas.log
-a etc/SATANNOT
-b etc/SATBAND
data/gempak/images/sat/\3/\5/VIS/VIS_\6_\7
These two actions decode the same product into the same output file since
/usr/local/ldm/data/gempak/images/sat is the same directory as
~ldm/data/gempak/images/sat.
This kind of double processing of the same product into the same output file
can have
two kinds of problems:
- unnecessary use of CPU resources
- if the actions were in two different pqact.conf files, the write by one
invocation
of pnga2area could stomp on the other invocation of pnga2area
I suggest that it would be a good idea to do some housekeeping in the way that
you are processing products using pqact. In particular, I would:
- move processing actions needed by McIDAS into a pqact.conf_mcidas
pattern-action file
- move processing actions needed by GEMPAK into a pqact.conf_gempak
pattern-action file
- leave all other actions in the pqact.conf file
After doing this more, I would modify ~ldm/etc/ldmd.conf to have three pqact
invocations:
#
# McIDAS-XCD actions
#
exec "xcd_run MONITOR"
exec "pqact -f FSL2|FNEXRAD|NLDN|UNIDATA etc/pqact.conf_mcidas"
#
# GEMPAK actions
#
exec "pqact -f FSL2|FNEXRAD|NIMAGE|NLDN|UNIDATA etc/pqact.conf_gempak"
#
# Other actions
exec "pqact -f <list of relevant feed types> etc/pqact.conf"
#
# rtstats uses the LDM to send product latency statistics to the UPC.
# For more info on rtstats please see the man page.
EXEC "rtstats -h rtstats.unidata.ucar.edu"
If you adopt this approach, you must make sure that the list of feeds specified
in the ldmd.conf 'exec' invocation contains all of the feeds for which there
are patterns in the corresponding pqact.conf_x pattern action file and no
others.
Also, you must make sure that there is no duplication of processing in the
separate
pattern-action files, or at least no duplication of the same processing in the
separate pattern-action files.
Segmenting the processing this way will help to conserve your system resources
and make your processing as efficient as possible. It may be the case that
you do not need to worry about system resources at the moment, but the volume
of data relayed by the IDD is continuously increasing, so a little tweaking now
will have significant benefits in the future.
Also, since I am not able to logon as 'ldm', I was not able to review how you
have data scouring setup. Data scouring is one of the big areas where one can
significantly save on system resources by doing some planning.
Last item:
The reason I asked you to run 'sh ./mcinet2007.sh install mcadde' as 'root' was
to re-enable access to cacimbo via ADDE. Since this is now working nicely, I
am wondering if you are amenable to cacimbo once again acting as one of the
cooperating a community servers?
Cheers,
Tom
****************************************************************************
Unidata User Support UCAR Unidata Program
(303) 497-8642 P.O. Box 3000
address@hidden Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage http://www.unidata.ucar.edu
****************************************************************************
Ticket Details
===================
Ticket ID: OML-578835
Department: Support McIDAS
Priority: Normal
Status: Closed