20010530: LDM ingest of NLDN data; ldm-mcidas decoding of same

>From: alan anderson <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200105302052.f4UKq2p15034 ldm-mcidas LDM pqact.conf

Alan,

>Hope summer is treating you ok.

It has been pretty hectic for me so far (too many things going on at
the moment).

>I am trying to get the NLDN lightning
>data into our ldm ingester and seem to be stuck.   I have obtained
>permission from Albany, and they have added us with an 'allow'
>statement on their server.  When our ldm starts, I see a request to
>striker.atmos.albany.edu  and then a     feedme OK  so I think that
>part is working ok.

OK.  This sounds promising.  A quick check of your LDM ingest status
using 'notifyme' shows that you are receiving the data from SUNYA:

/local/ldm% notifyme -vxl- -f NLDN -o 3600 -h waldo.stcloudstate.edu
Jun 01 00:03:36 notifyme[21880]: Starting Up: waldo.stcloudstate.edu: 
20010531230336.975 TS_ENDT {{NLDN,  ".*"}}
        NOTIFYME(waldo.stcloudstate.edu) returns OK
Jun 01 00:03:38 notifyme[21880]: NOTIFYME(waldo.stcloudstate.edu): OK
Jun 01 00:03:38 notifyme[21880]: 36b03f8fa3cd0004a735249bd245a29b    18228 
20010531230636.292    NLDN 000  2001151230005
Jun 01 00:03:39 notifyme[21880]: 61101470ff0fda9de7ec141981ffc230    17388 
20010531231236.837    NLDN 000  2001151230611
Jun 01 00:03:39 notifyme[21880]: 517b0b1a7f2ca77aaedc1f9d1c0c0fd3    16632 
20010531231835.657    NLDN 000  2001151231217
Jun 01 00:03:40 notifyme[21880]: 076aa5b7085a23e7b977eed59c85f742    16828 
20010531232437.165    NLDN 000  2001151231823
Jun 01 00:03:40 notifyme[21880]: 93d04775d5df866f536858eef0ddce6d    13692 
20010531233036.820    NLDN 000  2001151232429
Jun 01 00:03:41 notifyme[21880]: a2a5e0e65e543b6360efec398da4aaf3    14308 
20010531233639.239    NLDN 000  2001151233035
Jun 01 00:03:41 notifyme[21880]: 8b16ba07c7f7a2de30993c45a75f7b88    13412 
20010531234236.203    NLDN 000  2001151233641
Jun 01 00:03:41 notifyme[21880]: e4b17c492bbc52d9506eda83e1eb0e5d    11872 
20010531234837.751    NLDN 000  2001151234247
Jun 01 00:03:41 notifyme[21880]: 693fd0aef2dba30f84d02499c8971c36    12460 
20010531235437.415    NLDN 000  2001151234853
^CJun 01 00:03:42 notifyme[21880]: Interrupt
Jun 01 00:03:42 notifyme[21880]: exiting

>My pqact entry for the NLDN data is just below; basically just a copy
>from what has been there since last year some time.  I did change the
>first 2 entries as I assume I need to list something for
>
>       YYJJJHHMbMe  as listed in the pqact.conf discussion on your web
>       pg.
>
>My leading items in the pattern are  0 and 1-9  to match  year 01  -
>09 (looking ahead) I note that the example on your web site starts
>with   [12][0-9][0-9][0-9] | which I don't understand for a  2 digit
>listing of year   YY.

This pattern will match 1???, 2???, and ??: examples are 1999, 2000, 2001,
99, etc.

>Anyway, I am not getting any data;  no MDXX0071, which is the file for
>the data.  Also, doing a Route List in MCIDAS  shows   'none' for NLDN
>data received.

The existence of a lightning MD file and an entry in ROUTE.SYS showing
such a file are two different ways of looking at the same thing.
You are getting data, but it isn't being decoded.  More on that below.

>One more question about my LDM.  Looking today, I see lots of error
>lines regarding a broken pipe in my ldmd.log.  Samples listed at
>bottom.  I did not see these yesterday when I was editing pqact.conf,
>but maybe I created another error.   I did run pqactcheck and it said
>ok.

OK.  The errors seem to indicate a problem in FSL wind profiler decoding.
Did you stop and restart the LDM after doing the pqact.conf modifications?
I am thinking that the answer is yes, and further that the environment
from which you stopped and started the LDM had a PATH that did not include
the ~ldm/decoders directory where proftomd and nldn2md live.

>The machine is waldo, and Tom knows how to get in if needed.

Got it.

>pqact.conf
>NLDN    ^(0[1-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
>        PIPE    -close
>        nldn2md -d /var/data/mcidas 70 NLDN DIALPROD=LD \1\2 \3\400 DEV=CCN

This should be:

NLDN    
^([12][0-9][0-9][0-9]|[0-9][0-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
        PIPE    -close
        nldn2md -d /var/data/mcidas 70 NLDN DIALPROD=LD \1\2 \3\400 DEV=CCN


>May 30 20:32:43 waldo pqact[1126]: pipe_prodput: trying again
>May 30 20:32:43 waldo pqact[1126]: pbuf_flush (6) write: Broken pipe
>May 30 20:32:43 waldo pqact[1126]: pipe_dbufput:
>-closeproftomd-v-d/var/data/mci
>dasU6WPR691 write error
>May 30 20:32:43 waldo pqact[1126]: child 6205 exited with status 127
>May 30 20:32:43 waldo pqact[1126]: child 6203 exited with status 127   

This is failing for proftomd for some reason.  I will login and poke around.

<later>

OK, based on my hunch the the environment in which you stopped and restarted
your LDM did not have a PATH that contained the ~ldm/decoders directory, 
I did the following:

o login to waldo as ldm
o edit pqact.conf to change the NLDN entry
o stopped the LDM
o restarted the LDM

I now see that both FSL wind profiler and NLDN lightning data are being
decoded:

Jun 01 00:32:42 waldo proftomd[14176]: Starting up
Jun 01 00:32:43 waldo proftomd[14176]: Making /var/data/mcidas/MDXX0092; may 
take some time...
Jun 01 00:32:47 waldo proftomd[14176]: Decoding 2001152.0018 data into 
/var/data/mcidas/MDXX0092
Jun 01 00:32:47 waldo proftomd[14176]: Exiting
Jun 01 00:33:58 waldo nldn2md[14178]: NLDN2MD -- BEGIN
Jun 01 00:33:58 waldo nldn2md[14178]: PRODUCT CODE=LD          2001152     
002400
Jun 01 00:33:58 waldo nldn2md[14178]: Created MD file: 72
Jun 01 00:33:58 waldo nldn2md[14178]: NLDN2MD - Flash events in file: 385
Jun 01 00:33:59 waldo nldn2md[14178]: NLDN2MD -- DONE

The corresponding entries in the McIDAS routing table are in agreement with
this:

<as 'mcidas'>
cd workdata
route.k LIST
  ...
  LD NLDN Lightning Flashes      71-71   MDXX0072     01152   33     none     3
  ...
  U6 FSL2 6-minute Wind profil  default  MDXX0092     01152   32     none     3
  ...
route.k: Done

Looks like things are working.  Please let me know if you find anything
else that looks amiss.

Tom