[LDM #EFY-891334]: Questions about memory management and the LDM

Mike,

> At UNAVCO we're using LDM as a transfer mechanism from our ftp-in
> servers to our archiving clusters.  Perl scripts call pqinsert to add
> products to the ldm.pq in our dmx then internal ldm clients subscribe
> to appropriate feeds to extract files as a stream to feed a perl script
> to store files into a directory for further processing by more perl
> scripts.  We're running into some issues with product transfers and
> I've been assigned the task of looking into the LDM transfer issues.
> I suspect it has to do with the efficiency of the processing, and as I'm
> learning about LDM I have some thoughts about improving it's performance
> by changing our configuration.  That and updating our software package
> which is incredibly old (4.6.5).

4.6.5? Wow! That is old.

LDM 6 is almost a different beast. I'd recommend reading its documentation very 
carefully.

> One of the first things I have considered changing is the ldm.pq file
> size is set at 20G which seems large for the file sets we're transferring.
> The swap file on the machine (solaris 5.10) is only 4G and I think that it
> should be tuned to have a larger system swap than the 20G ldm.pq. The pq
> stats show that we're never using more than about a 10th of the slots, and
> the largest file we've seen is a 3.8G file sent as an uncompressed tar.

The LDM-6 product-queue is a memory-mapped file, so it doesn't use swap space 
(at least on a modern O/S). The rule-of-thumb for swap space is to have twice 
as much swap as physical memory.

The rule-of-thumb for the size of the LDM product-queue is to make it large 
enough and with enough slots so that the minimum residence time of a 
data-product is at least one hour.

> I would like to tune the ldm.pq size down to 2G and I was wondering if
> the ldm.pq were down sized to fit in the swap space at a 2G file size
> what would happen if pqinsert were used to try and add a 3.8G file?

LDM-4's pqinsert(1) would likely crash. LDM-6's will error-exit.

> I'm confident that this could help us out with our transfer issues but
> I would like to know if others with more experience using LDM have any
> more practical advice.  I'm still in the middle of mapping our product
> distribution and when completed I think I can eliminate most duplicate
> requests though I believe if we updated our software LDM does a good
> job of taking care of that for me.

The LDM won't insert a product into the queue if a product with the same 
signature (i.e., MD5 checksum) is already in the queue.

> Any feedback or recommendations you could provide would be greatly
> appreciated.

It would have been good if you had attended our LDM training workshop. The next 
one will be next summer.

Feel free to drop by. I often work from home, so you should call ahead.

We could also use Google Hangouts.

> Thanks
> 
> -Mike Rost
> 
> Software Engineer I
> UNAVCO
> 6350 Nautilus Dr
> Boulder CO 80301
> 
> 303-381-7467

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: EFY-891334
Department: Support LDM
Priority: Normal
Status: Closed