20040223: LDM on weather3.admin.niu.edu

Gilbert,

> To: Unidata Support <address@hidden>
> cc: address@hidden
> From: Gilbert Sebenste <address@hidden>
> Subject: Re: 20040221: Help! LDM on weather3.admin.niu.edu going crazy... 
> Organization: UCAR/Unidata
>Keywords: 200402212235.i1LMZlrV009065

The above message contained the following:

> You see, I was testing out a NWWS feed from David Magee, just to see if I 
> could ingest it. Bingo! It worked. He sends it out under the "PPS" feed.
> 
> Problem.
> 
> Whenever a zone forecast comes in on my machine, it runs a script called 
> "prepend_file" to append the incoming forecast to a particular file, such 
> as "TX.feb23". Basically, it just does a "cat" and appends the output to 
> the end of the file. No biggie.
> 
> BUT...
> 
> When the NWWS feed was running, I was getting *two* of those hitting the 
> script at the same time...one from NOAAPORT, the other from NWWS.

Odd.  The product-queue module of the LDM won't insert a data-product
with the same signature (i.e., MD5 checksum) as one that's already in
the product-queue.

Could it be that David Magee's ingester isn't computing the MD5 checksum
the same way as the NOAAPORT ingester?

> Thus, the "prepend_file" script went into an endless loop.  I finally
> discovered that this morning, along with 6 billion copies of the state
> zone forecast in there. It just did a repeated dump of the forecast
> until disk space was full.

Assuming that you only have one pqact(1) process active, then the two
identical data-products should be sent serially to the standard input of
the "prepend_file" script (assuming their signatures differed) rather
than simultaneously.  This should result in two data-products being
appended to the file rather than the script going into an endless loop.

> Lovely.
> 
> Now, I am not sure if this IS the cause of the problem. But:
> 
> 1. It only happened with state zone forecasts, as that is the only data 
> that uses that script
> 
> 2. The other larger file sizes I saw with the hourly weather roundups 
> were due to multiple copies of those products being sent over NWWS, a 
> known problem

Assuming that these data-products all have the same signatures, then
only one should be inserted into your product-queue and, subsequently,
retrieved by the pqact(1) process.

> 3. The problem began on the day when I changed my NWWS feed to read PPS, 
> not GPSSRC. My guess is that PPS *is* the same as "IDS|DDPLUS",

PPS is a subset of IDS|DDPLUS because DDPLUS is a composite of DDS and PPS.

The NOAAPORT textual data-products have the feedtype IDS|DDPLUS because
we can't tell which specific feedtype should be associated with each
data-product -- unlike the old days when each feed each had its own
channel.  Hence, we give the data-product the union of all possible
feedtypes and postpone full identification to pqact(1) and the LDM
decoders.

> which might throw the product into an endless loop, especially if
> you're running a script on it. In other words, if you request PPS, you
> also get anything under IDS|DDPLUS.

For the reason given above, all textual NOAAPORT data-products will match 
the PPS feedtype (as well as any combination of the feedtypes DDS, PPS,
and IDS).

> So, that script had to deal with 4 copies from NOAAPort, 
> plus one from the true PPS feed, all at once. Mee-yow.

I don't understand.  Why four copies?  And duplicate products should be
eliminated by the product-queue module.

> Yep, it just worked. I see that my Texas zone file, just erased, has come 
> back with 7 KB instead of 2 GB like it was doing on the very first 
> product.
> 
> Wowsers! Now, here's the fun part: this morning, for the first time, it 
> also filled up the hard drive on weather.admin.niu.edu, BUT weather2 was 
> fine. Huh? I thought duplicate products weren't sent? 

The product-queue module only allows one data-product with a given
(MD5) signature.

> Also, I was NOT ingesting PPS on weather. But if my hunch above was
> right, yes I was.  Which doesn't explain why weather bombed and
> weather2 was fine. Oy.

Because the amount of data doesn't seem consistent with even a five-fold
increase due to duplicate data-products, I suspect that the problem lies
with the decoder script.

> Anyway, thanks for your help over the weekend. I certainly wasn't 
> expecting that to happen. The funny thing is, when I did an "ldmadmin 
> watch", I couldn't see "PPS". I guess it was being renamed "IDS|DDPLUS" as 
> it was coming in.

If the IDS|DDPLUS NOAAPORT data-products were inserted into your
product-queue before the PPS data-products from David Magee and the MD5
signatures are computed correctly, then you wouldn't see any "PPS"
data-products from an "ldmadmin watch" because the product-queue module
eliminates duplicates and the pqmon(1) utility that underlies the
"ldmadmin watch" command reads from the product-queue.

> I'll watch it to make sure nothing else happens. But I think it's OK now.
> 
> *******************************************************************************
> Gilbert Sebenste                                                     ********
> (My opinions only!)                                                  ******
> Staff Meteorologist, Northern Illinois University                      ****
> E-mail: address@hidden                                               ***
> web: http://weather.admin.niu.edu                                      **
> Work phone: 815-753-5492                                                *
> *******************************************************************************

Regards,
Steve Emmerson
LDM Developer