20000707: LDM setup at INM (cont.)

>From: Pepo Juega <address@hidden>
>Organization: Instituto Nacional de Meteorologia
>Keywords: 200006220829.e5M8TZT11974 LDM ldm-mcidas

Pepo,

>Everything has been running smoothly for a couple of days. I got
>a  semi-operational setup, having both meteosat and grid feeding
>alfa in real time.

Excellent.  I would be most happy if your note ended right here :-)

>But I found a few images (about 2 out of 48)
>going to the wrong destination McIDAS DataSet, and also, these
>misplaced images were not complete. (Zero-filled)

OK.

>So I suspect LDM being too fast to allow pnga2area to complete its
>tasks.

This is the likely explanation.  The way that the LDM works (actually
pqact) on the receiving end is as follows:

o a product is received
o the pattern-action file, pqact.conf, is checked to see what to do
o a new process is started by pqact unless one is already designated
  to handle the product that just came in

It may seem that the last item in this list would take care of
serialization of product decodes, but this would only be true if the
decoder did not exit when it was finished with a product.  This is not
how I wrote pnga2area, however.

>I will try to explain further.
>
>meteosat feeds off the SDI by receiving a mail msg whenever a
>new slot is received. It then IMGCOPYs from SDI the image received,
>It does some IMGREMAPs to the original, and then it does a area2png
>and a pqinsert into the LDM queue to send to alfa.
>
>e.g.:
>imgcopy.k MET/FDV LDM/VISN.1 DOC=YES BAND=1 SIZE=SAME
>
>yimgremap.k LDM/VISN.1 LDM/VISNSINPS.1 PRO=PS 60 15 RES=9.5 
>            LATLON=42 17 DSIZE=740 1112
>
>file=`area2png -n -a /export/home/data/SATANNOT 
>-b /export/home/data/SATBAND -f /export/home/ldm/data/AREA/AREA1051 
>-l /export/home/ldm/logs/ldmd.log "VISNSINPS_ %Y%m%d %H%M"`
>
>pqinsert -v -f MCIDAS  -q /export/home/ldm/data/ldm.pq  \
>         -l /export/home/ldm/logs/ldmd.log "${file}"
>
>(AREA1051 corresponds to LDM/VISNSINPS.1)

Excellent!  This is exactly how I envisioned the compress/inject process
to work.

>alfa's pqact.conf receives this as:
>#####
># MCIDAS PNG format:
>#        VIS..._       CCYYMMDD   HHMM
>MCIDAS  (^VISNSINPS_) (........) (....)
>        PIPE    -flush  -close
>        png2area        data/AREA/VIS/\1_\2.\3  VISN_SIN_PS
>
>and png2area (notice it's not the original pnga2area!) contains:

Wonderful!  Wrapping pnga2area up into a script is exactly what
I would have done.

>#!/bin/sh
>
>cat > $1
>pnga2area -a /export/home/data/SATANNOT -b /export/home/data/SATBAND 
>-f $1 -v -l /export/home/ldm/logs/ldmd.log data/AREA/AREA8888
>
>imgcopy.k LDM/PNG.1 LDM/$2 SIZE=ALL
>rm -f $1
>
>(AREA8888 correspond to LDM/PNG.1)

OK, so here is your problem.  Since the received, decoded product is
always being written to the same output file, you have the possibility
of two pnga2area processes intefering with each other.  This situation
becomes worse as the receiving machine becomes slower.  I see from your
analysis below that you already deduced that.

>...so this means I want this product inserted into LDM/VISN_SIN_PS
>and it actually does get inserted, except those two sporadic occasions
>along the day.
>
>What I think is going on, is that as long as I always use LDM/PNG.1 
>as the intermediate image for all products received, it sometimes 
>gets overwritten by a 'faster' pnga2area being received inmediately
>after the first.

I agree with you.  I think that this is exactly what is happening.

>I looked at the documentation, and that is why I
>also use the -flush option, thinking that a blocking write() would
>make succesive calls to pnga2area wait for the first to finish before
>proceeding.

Unfortunately, no.  pqact is simply starting up another png2area when
a new product is received.

>See the following from alfa's ldmd.log file:
>Jul 07 07:35:39 pnga2area[22362]: Starting Up
>Jul 07 07:35:39 pnga2area[22362]: output file pathname:
>data/AREA/AREA8888
>Jul 07 07:35:39 pnga2area[22365]: Starting Up
>Jul 07 07:35:40 pnga2area[22365]: output file pathname:
>data/AREA/AREA8888
>Jul 07 07:35:40 pnga2area[22362]: unPNG::   127537    308208  2.4166
>Jul 07 07:35:40 pnga2area[22362]: Exiting
>Jul 07 07:35:40 pnga2area[22365]: unPNG::   166227    308288  1.8546
>Jul 07 07:35:40 pnga2area[22365]: Exiting
>
>22362 and 22365 all happen at the very same second! 

This output confirms your deduction.

>No time for IMGCOPY to finish with 362 when 365 overwrites AREA8888?

Right.  Since both pnga2area invocations are being told to write to
the same output file, data/AREA/AREA8888, they are stepping on each
other.

>If there is no way out, I will have to arrange having different
>intermediate AREAs for unPNGing, but I wonder if there is a way
>out.

I don't know of a way out of this dilemma.  I can tell you that this
problem will go away when you upgrade to McIDAS-X 7.70. The reason
it will go away is 7.7's ability to access data in AREA file format
that does not follow the AREAnnnn naming convention.  This means that
you will be able to write to a file named like
data/\SAT_\BAND_\RES_%Y%m%d.%H%M (or any of a myriad of other forms),
and these names can be unique.  The 7.7 dataset definition will be
something like:

DSERVE ADD LDM/PNG AREA DIRFILE=/export/home/ldm/data/METEOSAT* "METEOSAT images

I wrote pnga2area to accommodate just this kind of capability.

>P.S.: I received Don's notice of being placed at the Workshop's
>waiting list. 

I hope that you understand that we must wait list folks that are
not in our primary support category (which is universities in WMO
Region IV).  If there are seats left in the course(s) at the end
of registration, we will fill them with folks from the waiting list(s)
on a first received, first accepted basis.

On a totally different note, what do you do for radar?  Does Spain have
a network of radars that are producing "products" that you can access
along with satellite imagery?  Have you written an ADDE server for these
products?

>Txs,

You are welcome.

Tom