20030430: GOES-10 Ingest Times (cont.)

>From: Arnold Hori <address@hidden>
>Organization: University of Hawaii
>Keywords: 200304282142.h3SLgA7U018448 Unidata-Wisconsin GOES-10

Hi Arnold,

>Hi and Thanks Tom;
>We have been watching the GOES-10 ingest times and noticed the times are
>back to the pre-Apr18 times.

OK.  I will be working with SSEC to see if we can push the times
up some more (i.e., not wait until 15 minutes after the nominal
satellite time before creating the sector and inserting it into
the LDM queue).

>However, there is a problem in that some of
>the images get ingested by 20minutes past the hour but our pnga2area
>decoder does not start until more than 1hour after that time.

So, the images are in your queue, but are not getting processed until
later?  If yes, then this typically indicates that your pqact is running
behind on processing.

>Attached
>are two files which illustrate these events.  the WVIR-???? have output
>from a notifyme (-f MCIDAS -p GOES-10) plus the pnga2area output to the
>ldm-mcidas.log file and the ls -l of the WV or IR mcidas file.  WVIR-slow
>show that the feed begin at about 15-20minutes after the hour, but the
>pnga2area processing did not start until up to 60minutes after the ingest
>time.

WVIR-slow

U.HAWAII vapor.soest.hawaii.edu  output from 
1) notifyme -h localhost -f MCIDAS -p GOES-10 output
2) pnga2area output to ldm-mcidas.log 
3) ls -l of GOES-10/8km/WV and GOES-10/4km/IR files 
4) Local time is 10hours after UTC; 0400z is 1800 Local 

1)
Apr 30 04:16:06 notifyme[19215]:  1407486 20030430041518.230 UNIWISC 000  
pnga2area Q1 U9 120 GOES-10_IMG 0.65um 4km 20030430 0400

2)
Apr 30 05:28:58 pnga2area[21156]: Starting Up
Apr 30 05:28:58 pnga2area[21156]: output file pathname: 
data/gempak/images/sat/GOES-10/8km/WV/WV_20030430_0400
Apr 30 05:28:59 pnga2area[21156]: unPNG::   158959    613456  3.8592
Apr 30 05:28:59 pnga2area[21156]: Exiting

3)
-rw-r--r--   1 ldm      businger  613456 Apr 29 19:28 WV_20030430_0400
               
This does indeed show that pqact is running behind:

- The notifyme shows that you received the product 48 seconds after it
  was first injected into the IDD (04:16:06 - 041518) which is not great,
  but it is not a show stopper.

- the output from pnga2area shows that it was started by pqact at
  05:28:58, which is 1:12:52 _after_ the image was received.  You will
  notice that pnga2area finished at 05:28:29, so it took about a
  second for pnga2area to do the decoding.

- the ls output agrees with the pnga2area log output

>The second files show some of the data which were processed by
>pnga2area in a timely fashion.

WVIR-fast

U.Hawaii vapor.soest.hawaii.edu
output from notifyme; pnga2area; ls -la 
of WV & IR data which got processed soon after the data were ingested
Local time is 10hours after UTC; 1500z is 1500Local

1)
Apr 30 15:15:24 notifyme[19215]:   170694 20030430151518.737 UNIWISC 000  
pnga2area Q1 UB 173 GOES-10_IMG 6.8um 8km 20030430 1500

2)
Apr 30 15:15:24 pnga2area[26001]: Starting Up
Apr 30 15:15:24 pnga2area[26001]: output file pathname: 
data/gempak/images/sat/GOES-10/8km/WV/WV_20030430_1500
Apr 30 15:15:26 pnga2area[26001]: unPNG::   170694    613456  3.5939
Apr 30 15:15:26 pnga2area[26001]: Exiting

3)
-rw-r--r--   1 ldm      businger  613456 Apr 30 05:15 WV_20030430_1500

This shows that pqact had caught up with the processing:

- the image was received 6 seconds after it was first injected into the IDD
  (this is more like it)

- pnga2area was started by pqact almost immediately after the product was
  received, and it took up to 2 seconds to finish decoding

- the ls output agrees with the pnga2area log output

>We are unsure if this is a problem at our end or if it is a transmission
>problem.

It does not appear to be a transmission problem since you received the
images in a reasonable amount of time (although, a 48 second delay is a
little high for the images when the client and upstream server are
both running LDM-6).

>Up until 2000z today/Apr30, we were using version 7.6.4 of the
>pnga2area but have just downloaded the 7.8.0 version.  We will be checking
>if this version improves the processing.  Is there anything else which we
>should be checking?

You should be using the v2002b version of the ldm-mcidas decoders,
since v2002b is the latest version.

>IF you are interested in the MKWC LAPS output you will find it at
>       http://hokukea.soest.hawaii.edu/current/laps/index.cgi
>The cloud cover Parameter Group is the field which is most impacted by
>missing WV and IR data.

OK, thanks for the link.

Back to your problem:

Is your machine heavily loaded at the time of the "slow pnga2area"
processing?  Is pqact using an excessive amount of CPU during the
times of "slow pnga2area" processing?

Is your LDM queue large enough for the volume of data you are ingesting
via the IDD?

Do you have a LOT of actions in ~ldm/etc/pqact.conf (typical of a GEMPAK
user)?  Along these lines, do you have any problematic regular expressions
in your ~ldm/etc/pqact.conf file (Steve Emmerson sent out a message
some time ago about potentially bad regular expressions in both ldmd.conf
requests and pqact.conf actions)?

If you are using a single instance of pqact in your LDM setup, and if
pqact is the culprit in the slow processing, and you do not have any
problematic regular expressions in pqact.conf, then you might consider
running a separate invocation of pqact to processes the
Unidata-Wisconsin images.  For instance, if your ~ldm/etc/ldmd.conf
file fires up one pqact instance to process all ~ldm/etc/pqact.conf
entries:

exec    "pqact"

you could change this to two invocations, one for the UNIWISC images
and the other for everything else:

exec    "pqact -f UNIWISC /usr/local/ldm/etc/pqact.uniwisc"
exec    "pqact -f ANY-UNIWISC /usr/local/ldm/etc/pqact.conf"

Of course, you will need to move (not copy) all of your UNIWISC (aka
MCIDAS) actions from ~ldm/etc/pqact.conf into ~ldm/etc/pqact.uniwisc
and then stop and restart your LDM for this to have any effect.

Tom

>From address@hidden Thu May  1 15:51:44 2003

Hi Tom;

Thanks for the analysis of yesterday.  We do a lot of gempak
decoding actions.  Last week, we also had Steve Emmerson tuneup our LDM;
he made some corrections to our pqact.conf file but said that they were
mostly in good order so I have left pqact.conf unchanged.  But I have
increased the ldm.pq queue to 50MB (from 40MB) and installed the
ldm-mcidas-2002b programs to our machine.  And since that time, we have
been getting GOES-10 files in a timely fashion.  I have not yet checked on
the LDM system load but suspect that you are correct about our cpu getting
overloaded at some times.  I have changed the time of our syscheck cron
job to have it check our LDM machine at about the time the GOES-10 files
are getting processed.

The Mauna Kea Weather Center folks are now very happy with the situation
and they also send their Mahalo Nui Loa (i.e., thanks-a-lot) to you folks
for all the assistance.

Aloha Nui Loa
Arnold H/U.Hawaii/