[LDM #OFP-197738]: debugging LDM 6.10.1

Sue,

> I am trying to debug significant data loss using LDM to transfer files
> from NCAR/RAL to MIT/LL. I believe this started happening after upgrade
> to LDM 6.10.1 and an OS upgrade to Debian Linux squeeze. Previously,  we
> had been running 6.8.1 In trying to track missing data, I am using
> notifyme to record files making it to our outgoing LDM server. One thing
> that I noticed besides missing files was that notifyme is reporting some
> products more than once several minutes apart. Could the repeated
> product notification also be a symptom of the Linux clock issue (not
> monotonically increasing values).

I suspect that a non-monotonic clock is *not* the cause of the duplicate 
product reports from notifyme(1) -- rather, the cause is likely a too-small 
product-queue. The product-queue will reject as a duplicate a data-product that 
has the same MD5 checksum as a data-product that's already in the 
product-queue. The notifyme(1) program causes an upstream LDM process to be 
created on the relevant host. That process watches the product-queue and sends 
notifications of every new data-product it sees entered into the product-queue 
that matches the notifyme(1) selection criteria (feedtype/pattern). If a 
data-product is purged from the product-queue in order to make room for more 
data-products, and if a duplicate of that data-product arrives after the purge, 
then it will be reported by the notifyme(1)-created upstream LDM process twice.

LDM 6.10 has a feature to inform you if the product-queue is too-small. The 
relevant registry parameters are "/reconciliation-mode", "/server/max-latency", 
"/queue/size", and "/queue/slots". The relevant ldmadmin(1) command is 
"ldmadmin vetqueuesize". For more information, see 
<http://www.unidata.ucar.edu/software/ldm/ldm-current/basics/LDM-registry.html>.

Contact me if you have any questions.

> thanks,
> Sue

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: OFP-197738
Department: Support LDM
Priority: Normal
Status: Closed