[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
20000629: LDM setup at INM (cont.)
- Subject: 20000629: LDM setup at INM (cont.)
- Date: Thu, 29 Jun 2000 13:17:13 -0600
>From: Pepo Juega <address@hidden>
>Organization: Instituto Nacional de Meteorologia
>Keywords: 200006220829.e5M8TZT11974 LDM ldm-mcidas
Pepo,
>Understood the FILE action.
OK. I thought you did, but I felt obligated to include comments
regarding the '-d', '-l', and '-v' flags. All email transactions
to/from support get put into our inquiry tracking system. The more
completely a question gets answered, the easier it is for the person
searching the tracking system to apply the information to his/her
situation. If at any time anyone does not want his/her
comments/questions put into the tracking system, all they have to do is
let us know and we will exclude them.
>Changed my pqact.conf at alfa:
>----------------------------------------------------
># MCIDAS products:
>#
># GRID files:
>#
>MCIDAS ^.*GRID.*
> FILE -close data/GRIB/%m%d%H%M.grid
>
># AREA files:
>#
>MCIDAS ^.*AREA.*
> FILE -close data/AREA/%m%d%H%M.area
>----------------------------------------------------
>ldmadmin pqactcheck
>...ldm/etc/pqact.conf is syntactically correct
Looks good.
>I fed manually from meteosat:
>pqinsert -vx -f MCIDAS -q $HOME/data/ldm.pq AREA7028
>Mapping 25595904
>pqinsert: 3e6a273cac4f483c2f757d1663509031 263424 20000629135738.024
>MCIDAS 000 AREA7028
>
>...and... I got 06291357.area on alfa !!!!!
Success! The really nice thing is the fact that there can be many, many
machines to which the product will be sent and delivery is guaranteed
up until the point that the product gets scoured out of the sending
machine's queue.
>Also tried out the zip/unzip business with good results
Very good.
>So, YOU got things moving al last. Now, as long as you suggested,
>I will be very interested in the rest i.e.: an 'intelligent'
>file namimg schema, and PNG.
Good.
re: PNG compression/uncompression routines
>Please tell me all about it.
I developed the PNG compression and uncompression routines for McIDAS
AREA files for our use in our Internet Data Delivery (IDD:
http://www.unidata.ucar.edu/projects/idd/index.html) system. I chose
the PNG technique after reasonably extensive comparisons of it against
a number of other compression methods. It was the overall winner even
though I chose to not compress the file headers and comment cards. A
"graphical" comparison of an AREA file and PNG compressed AREA file is
shown here:
AREA file PNG compressed AREA file
+--------------------+ +----------------------+
| area directory | | area directory |
+--------------------+ +----------------------+
+--------------------+ +----------------------+
| navigation codicil | | navigation codicil |
+--------------------+ +----------------------+
+--------------------+ +----------------------+
| calibration codicil| | calibration codicil |
+--------------------+ +----------------------+
+--------------------+ +----------------------+
| aux block | | aux block |
+--------------------+ +----------------------+
+--------------------+ +----------------------+
| | | AREA file comment |
| | | cards |
| image lines | +----------------------+
| | +----------------------+
| | | |
+--------------------+ | PNG compressed |
+--------------------+ | image |
| AREA file comment | | |
| cards | | |
+--------------------+ +----------------------+
The PNG compression is only done on the image line portion of the
AREA file. The comment cards, if they exist, are moved to just
after all of the other directories/codicils in the PNG compressed file.
The compression and uncompression routines can extract information from
the AREA file header and use it as indicies into the McIDAS SATANNOT
and SATBAND files to extract descriptive information that can be
used for file naming. This turns out to be very important/useful
for Unidata GEMPAK users and will be very useful for McIDAS users once
they upgrade to Version 7.70 (I am working on my release right now).
Our ldm-mcidas package source contains PNG interface routines that make
it very easy to read/write the PNG compressed portion of the image.
>I don't know too much about McIDAS 7.7
>yet, as we haven't got the release.
One of the nice things about 7.7 is the ability to get away from the
arcane McIDAS file naming conventions. For instance, suppose that
you have a set of infrared images in AREA file format from METEOSAT.
The pre-7.7 way of creating a dataset was to:
o configure your McIDAS-X session to know how to find the AREA files
that the images are stored in. As you well know, this is done
by use of either or both of REDIRECT and MCPATH. REDIRECT allowed
one to locate AREA files on an image-by-image basis. MCPATH
was typically used to locate groups of files in a single directory
(this is not strictly true, but it is effectively true).
o define a dataset something like:
DSSERVE ADD METEOSAT/IR AREA 1525 1600 "METEOSAT IR images
It has always been my goal to not have to name the images (or MD or
GRID) files AREAnnnn. A more descriptive name might be something
like:
METEOSAT_IR_20000630.1530
7.7 now allows this. Assuming that you had a set of images in AREA file
format named like this example and located in, say, the /data/mcidas
directory, the DSSERVE command to setup the dataset would be:
DSSERVE ADD METEOSAT/IR AREA DIRFILE=/data/mcidas/METEOSAT_IR*
REDIRECT and MCPATH do not come into play at all with this scheme, AND
you can have more than 9999 files!
>But you can imagine that what
>I intend is to transfer McIDAS images to a bunch (=~60) of hungry
>users. We do not want all of them to make ADDE requests to meteosat
>at the very same minute every half hour!
Right.
>So, we decided to build
>regional McIDAS image servers, and now what we do is plain old ftp
>sessions to feed them offline the McIDAS ports. Once the images rest
>at their regional server, they can remap, reload, etc. with ease.
The LDM will be very useful for this activity. This setup is analagous
to our IDD, and that is moving on the order of 250 GB per day when the
reception at all sites is aggregated.
>Now, this will be tricky with the LDM I got now. I tried to feed
>alfa from meteosat simulating arrival of a new meteosat slot, and
>what I get from that is three bands (VIS+IR+WV) which means 3
>images to send over. The transfer is so fast that the name for the
>file is the same, and the last overwrites the first two.
If/when you start using the PNG compress/uncomress routines, you can
name the output files so that they won't match for different bands.
>Besides,
>I have no way to differentiate bands. Of course, I could start by
>giving them different names at meteosat, like VIS001 IR001...
>and prepare a different pattern to be recognized, but once you
>told me, I would prefer this improved PNG feature.
I think that you will like this.
>BTW the band
>information, being also at the image header, could be used to
>implement the file naming strategy, right?
Absolutely correct. This is exactly what we are doing right now to
send image products created by the CIMSS group in SSEC to our
sites. As a quick example of using the PNG compression routine, I
offer the following simple Korn shell script that we are using to create
PNG compressed CIMSS products that are then sent through our IDD:
#!/bin/ksh
cd $HOME/mcidas/png
area=$1
typeset -Z4 area
dialprod=$2
file=`area2png -n -f $HOME/mcidas/data/AREA${area} -a
/home3/mcidas/data/SATANNOT -b /home3/mcidas/data/SATBAND "pnga2area Q0
${dialprod} $area \\SAT \\BAND \\RES %Y%m%d %H%M"`
pqinsert -f MCIDAS "${file}"
date=`date -u `
echo "${file}" " [ sent $date ] " >> png_file.log
/bin/rm -f "${file}"
An example file name generated by the file=`area2png ...` line in the script
is:
pnga2area Q0 CB 1110 GOES-9_IMG UNKBAND 10km 19970709 1800
The relevant pieces of this are:
Q0 - this is not really relevant for you. We use it to
tag the image with the portion of the hour from which
it was scanned:
Q0 - any time of the hour
Q1 - H to H+15
Q2 - H+15 to H+30
Q3 - H+30 to H+45
Q4 - H+45 to H+50
CB - an arbitrary two letter "product code". This was added
for support of the McIDAS routing table, ROUTE.SYS,
that some of our sites still use. You probably don't
use this notion, so you should ignore it in this example
\\SAT GOES-9_IMG - this comes from a lookup of the satellite sensor source
number (McIDAS AREA header word 2 (zero based) in the
McIDAS satellite annotation file SATANNOT
\\BAND UNKBAND - the band number for this image is not one of the ones
available in the McIDAS satellite band information file,
SATBAND. Since there is no information, area2png expands
this as UNKBAND
\\RES 10km - base resolution of the satellite from the McIDAS satellite
band information file, SATBAND. If the information
on the resolution is not in SATBAND, this is the
LINRES value in the AREA file header (header word 11
(zero based)) expressed in km
%Y%m%d 19970709 - date information from the AREA file header (header word
3 (zero based))
%H%M 1800 - time information from the AREA file header (header word
4 (zero based))
A PNG file with the name:
"pnga2area Q0 CB 1110 GOES-9_IMG UNKBAND 10km 19970709 1800"
is created from the input AREA. Its name is then used as the header
when the product is 'pqinsert'ed into the LDM product queue.
The receiving machine can key on this product with pqact entries that
might look like:
# broadcast header format:
# pnga2area Q0 Cx AREA SAT BAND RES CCYYMMDD HHMM
MCIDAS ^pnga2area Q. CA (.*) (.*) (.*) (.*) (........) (....)
PIPE
pnga2area /var/data/mcidas/CIMSS/\2_CTP_\4_\5_\6
I have made binary distributions of the ldm-mcidas package available
for a variety of operating systems:
Operating System pub/binary subdirectory
----------------------+---------------------------------------------------
DEC OSF/1 4.0E osf1_4.0-alpha
HP-UX 11.0 hpux_11.00-hp9000
IBM AIX 4.3 aix_4.3-rs6000
RedHat 6.[012] Linux linux_2.2-i686
SGI IRIX 6.5 irix_6.5-mips
SGI IRIX64 6.5 irix64_6.5-mips
Sun SOLARIS 5.7 SPARC sunos_5.6-sparc
Sun SOLARIS 5.7 SPARC sunos_5.7-sparc
Sun SOLARIS 5.7 x86 sunos_5.7-i86pc
Sun SOLARIS 5.8 x86 sunos_5.8-i86pc
In all cases, you can ftp the distribution (using binary mode!) by
anonymous FTP to our ftp server, ftp.unidata.ucar.edu. The latest
release is always accessible in ldm-mcidas.tar.Z; this is a link to the
most recent version which is currently ldm-mcidas-7.6.3.tar.Z.
If your machine(s) is(are) not using one of the above OSes, you can build
the distribution from source. In this case, you should grab the file
ldm-mcidas.tar.Z (a link to the current source distribution
ldm-mcidas.7.6.3.tar.Z) from the pub/ldm-mcidas directory using
anonymous ftp on the same ftp server, ftp.unidata.ucar.edu. In order
to build the package from source, you will need to have installed:
o LDM
o netCDF
since modules need include files from these packages for compiling and
the libraries from these for linking. The good news for you is that
you already have the LDM built and the netCDF is included as part of
the McIDAS-X distribution!
After you grab the distribution, you can investigate how the PNG
compression and uncompression routines work. You can do this through
the area2png.1 and pnga2area.1 man page files; by running either
routine with no arguments (or with -h); or you can look at my
ldm-mcidas web pages:
http://www.unidata.ucar.edu/packages/mcidas/mcidd/ldm-mcidas.html
If things don't make sense, please let me know.
>Txs a lot Tom, I'm in debt with you. Ask for anything. Whenever
Well, you probably don't really mean this, but you could:
o fly me to Spain for a month :-)
o send vast amounts of money to my private Swiss bank account :-)
Seriously though, I would be interested in you considering giving
Unidata users access to one of your ADDE servers so they could
occasionally look at the METEOSAT data that you undoubtedly have in
abundance.
Tom