20050124: Unidata McIDAS-XCD Trouble (cont.)

>From: Unidata Support <address@hidden>
>Organization: UCAR/Unidata
>>Keywords: 200501201933.j0KJXOv2020815 McIDAS-XCD ADDE

Hi Again Jim,

OK, the McIDAS-X/-XCD rebuild finished and I reinstalled the full
distribution.  While on accas I found that XCDDATA had been defined to
be 'pileus.geo.msu.edu'.  I changed this and then reran the XCD BATCH
setup scripts.  I also deleted the ADDE dataset definition file,
~mcidas/workdata/RESOLV.SRV and recreated the datasets.

After all of this I expected things to start working again, but they
are not, or, at least, they didn't appear to be any different than they
were prior to my rebuild.

So, I started watching data being ingested through your LDM and found
that you are requesting IDS|DDPLUS, HDS, UNIWISC, and FSL data from U.
Wisconsin's f5.aos.wisc.edu.  I tried changing that to a Unidata
machine, but nothing improved.  I then started to think about the fact
that you are inserting data from a NOAAPORT receiver into your LDM
queue using pqing, AND you are calling this data 'WMO'.  I believe that
what is going on is the following:

- textual products from the 'WMO' stream are the same as ones
  you are requesting from UW.

- the LDM MD5 checksum assigned to products is compared to products
  already in the LDM queue and the product is only inserted if
  a product with the same MD5 checksum is not already in the queue.
  The MD5 test does _not_ include any notion of the datastream this
  product belongs to!

- the products being sent from UW are, therefore, being rejected since
  they have already been put into the LDM queue in your 'WMO' datastream

- the XCD action in ~ldm/etc/pqact.conf says to send all textual
  products from the IDS|DDPLUS IDD stream to 'xcd_run DDS'.

Since the products in 'WMO' from your NOAAPORT ingest machine get put
into your LDM product queue before you receive the same ones from UW,
you are essentially not receiving any data that should get processed by
XCD decoders.  This is why your McIDAS MDXX files are not being updated
as they should.

The solution to this dilemma is insert products from your NOAAPORT
system into the LDM queue using datastream names that correspond to
those used in the IDD.  Note that if you attempt to send binary (i.e.,
HDS and NNEXRAD) products from your WMO stream into the McIDAS XCD text
processing action ('xcd_run DDS'), the decoding should fail in
unexpected ways.

An alternative solution would be if you could figure out a regular
expression for the products in your 'WMO' stream that selected just the
textul products, then use that regular expression in a redone pqact.conf
action that is something like:

WMO     <your regular expression for text products>     PIPE
        xcd_run DDS
WMO     <your regular expression for grib message products>     PIPE
        xcd_run HDS

The final solution would be to stop your pqing process so that you
would get IDS|DDPLUS products from UW.

This was a tough one to understand!

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.