[LDM #TFK-703998]: Products not pulled into LDM queue

Justin,

I didn't notice anything untoward so far.

Would you please send me the "Starting Up" message in the NOAAPort LDM's log 
file for the connection by the secondary LDM.

> When our primary system started having problems it was actually affecting
> only a disk mount, which impacted one of our other transfer processes, I
> don't believe it actually affected the LDM since none of our LDM processes
> use that mount. But, we had to move all processes running on that system.
> 
> We shut down the LDM on the problematic system at 19:14Z
> 
> The LDM was then started by cron at 19:18Z on the newly primary system,
> here is the startup statement in the log:
> 
> May 22 19:18:22 vm-lnx-ncodf1 ldmd[31722] NOTE: Starting Up (version:
> 6.11.6; built: Feb 28 2014 08:52:01)
> May 22 19:18:22 vm-lnx-ncodf1 ldmd[31722] NOTE: Using local address
> 0.0.0.0:388
> May 22 19:18:22 vm-lnx-ncodf1 ldmd[31722] INFO: Resolving
> vm-lnx-ncodf1.ncep.noaa.gov to 140.90.100.144 took 0.000356 seconds
> May 22 19:18:22 vm-lnx-ncodf1 ldmd[31722] WARN: Adjusted pathological
> regular expression ".*IRIS" on or before line 164, file
> "/usr1/ldm/etc/ldmd.conf"
> May 22 19:18:22 vm-lnx-ncodf1 ldmd[31722] WARN: Adjusted pathological
> regular expression ".*IRIS" on or before line 167, file
> "/usr1/ldm/etc/ldmd.conf"
> May 22 19:18:22 vm-lnx-ncodf1 pqact[31725] NOTE: Starting Up
> May 22 19:18:22 vm-lnx-ncodf1 pqact[31725] INFO: Successfully read
> configuration-file "/usr1/ldm/etc/pqact.conf"
> 
> I checked and all of the clocks are in sync, all use NTP.
> 
> Here are the "missed products" in the upstream LDM...
> 
> *First* is WUUS51 KLWX 221914
> 
> Here is the noaaportIngester log entry inserting it into the queue:
> 
> May 22 19:15:00 noaaportIngester[42415] NOTE: WUUS51 KLWX 221914 /pSVRLWX
> inserted [cat 1 type 4 ccb 2/0 seq 55245423 size 2324]
> 
> And the LDM log showing it being sent to the downstream systems (but not
> our vm-lnx-ncodf1 newly primary LDM which starts at 19:18):
> 
> May 22 19:15:00 vm-lnx-sbn2 vm-bldr-ncodf2.ncep.noaa.gov(fee[45575] INFO:
> sending:       2324 20180522191500.795 IDS|DDPLUS 55245423  WUUS51 KLWX
> 221914 /pSVRLWX
> May 22 19:15:00 vm-lnx-sbn2 vm-lnx-ncodfdev.ncep.noaa.gov(fe[14924] INFO:
> sending:       2324 20180522191500.795 IDS|DDPLUS 55245423  WUUS51 KLWX
> 221914 /pSVRLWX
> 
> 
> *Second* is WUUS51 KCLE 221917
> 
> May 22 19:17:13 noaaportIngester[42415] NOTE: WUUS51 KCLE 221917 /pSVRCLE
> inserted [cat 1 type 4 ccb 2/0 seq 55251867 size 1226]
> 
> LDM:
> 
> May 22 19:17:13 vm-lnx-sbn2 vm-bldr-ncodf2.ncep.noaa.gov(fee[45575] INFO:
> sending:       1226 20180522191713.186 IDS|DDPLUS 55251867  WUUS51 KCLE
> 221917 /pSVRCLE
> May 22 19:17:13 vm-lnx-sbn2 vm-lnx-ncodfdev.ncep.noaa.gov(fe[14924] INFO:
> sending:       1226 20180522191713.186 IDS|DDPLUS 55251867  WUUS51 KCLE
> 221917 /pSVRCLE
> 
> 
> *Third* is WGUS81 KCLE 221916
> 
> May 22 19:16:57 noaaportIngester[42415] NOTE: WGUS81 KCLE 221916 /pFLSCLE
> inserted [cat 1 type 4 ccb 2/0 seq 55251234 size 1236]
> 
> LDM:
> May 22 19:16:57 vm-lnx-sbn2 vm-bldr-ncodf2.ncep.noaa.gov(fee[45575] INFO:
> sending:       1236 20180522191657.565 IDS|DDPLUS 55251234  WGUS81 KCLE
> 221916 /pFLSCLE
> 
> 
> As a point of reference the next WUUS51 product was at 19:25Z and I see it
> was sent to vm-lnx-ncodf1
> 
> May 22 19:25:35 vm-lnx-sbn2 vm-bldr-ncodf2.ncep.noaa.gov(fee[45575] INFO:
> sending:       2134 20180522192535.014 IDS|DDPLUS 55276307  WUUS51 KLWX
> 221925 /pSVRLWX
> May 22 19:25:35 vm-lnx-sbn2 vm-lnx-ncodfdev.ncep.noaa.gov(fe[14924] INFO:
> sending:       2134 20180522192535.014 IDS|DDPLUS 55276307  WUUS51 KLWX
> 221925 /pSVRLWX
> May 22 19:25:35 vm-lnx-sbn2 vm-lnx-ncodf1.ncep.noaa.gov(feed[20542] INFO:
> sending:       2134 20180522192535.014 IDS|DDPLUS 55276307  WUUS51 KLWX
> 221925 /pSVRLWX
> 
> So by 19:25Z the LDM on ncodf1 appears to  have been behaving as expected.
> 
> 
> Unfortunately we are not running the ldm metrics processing.

Pity.

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: TFK-703998
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.