20050812: radar lag?

>From: Brendon Hoch <address@hidden>
>Organization: Plymouth State University
>Keywords: 200508121359.j7CDxFjo001287 IDD FNEXRAD latency pqact kill -USR2

Hi Brendon,

>We're seeing some latency in some of our national nexrad radar data, 
>files coming in are about an hour old:
>
>%pwd
>/huge/sdata/nowrad/1KN0R-NAT
>%ls -lt | more
>total 60176512
>-rw-r--r--  1 ldm  usr  14208533 Aug 12 13:29 BREF_20050812_1229
>-rw-r--r--  1 ldm  usr  14208533 Aug 12 13:24 BREF_20050812_1224
>-rw-r--r--  1 ldm  usr  14208533 Aug 12 13:19 BREF_20050812_1220
>-rw-r--r--  1 ldm  usr  14208533 Aug 12 13:14 BREF_20050812_1216
>-rw-r--r--  1 ldm  usr  14208533 Aug 12 13:09 BREF_20050812_1211
>
>Looks like the lag started around 5Z and increased from there.  Any 
>feedback is appreciated.

I just checked the data on atm.geo.nsf.gov (your IDD upstream feed host)
and see that the national 1km N0R (BREF) product is current:

IMGLIST NEXRCOMP/1KN0R-NAT.ALL
Image file directory listing for:NEXRCOMP/1KN0R-NAT
 Pos Satellite/         Date       Time      Center   Band(s)
     sensor                                 Lat  Lon
 --- -------------  ------------  --------  ---- ---- ------------
   1  RADAR         12 AUG 05224  07:14:00     TWX    27

 ...

  90  RADAR         12 AUG 05224  14:17:00     TWX    27
  91  RADAR         12 AUG 05224  14:21:00     TWX    27
  92  RADAR         12 AUG 05224  14:26:00     TWX    27
  93  RADAR         12 AUG 05224  14:31:00     TWX    27
IMGLIST: done

I ran the McIDAS IMGLIST command at 14:41Z; the most recent image is 14:31,
so the data is getting to atm on time.

The next thing I did was look at the data reception at Plymouth (on pscwx):

http://my.unidata.ucar.edu/content/software/idd/rtstats/siteindex.php?pscwx.plymouth.edu

Latency for FNEXRAD products:
http://my.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?FNEXRAD+pscwx.plymouth.edu

Histogram for latency for FNEXRAD products:
http://my.unidata.ucar.edu/cgi-bin/rtstats/iddbinstats_nc?FNEXRAD+pscwx.plymouth.edu

The histogram plot shows that pscwx is receiving 95% of the FNEXRAD
products within 8 seconds of their being injected into the IDD (and 99%
within 11 seconds).

Given that the products are being delivered to pscwx within a few
seconds of their IDD insertion time, and given that the products on
your upstream feed host are current, my conclusion is that the
processing of the data out of the LDM queue on pscwx must be slow.  If
you have lots of pqact.conf actions for a single invocation of pqact,
then this would explain it taking longer and longer to process the
products out of the queue.  So, I would check your LDM to see if it is
struggling to process products already received:

<as 'ldm'>
kill -USR2 <pid of pqact>
kill -USR2 <pid of pqact>

The two kills will put pqact into a debug logging state.  One of the
things that will be listed will be how long it is taking to process
data out of the queue.  What you want to look for are Delay and
pq_sequence messages since they will tell you the 'time elapsed' when
processing a product.

NOTE: don't leave your system in debug logging mode for long -- the
log output is _significant.

>Thanks,

No worries.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.