[LDM #EFY-891334]: Questions about memory management and the LDM

Mike,

> Thanks Steve: I appreciate the dialog.
> 
> I’ll see about getting into a training workshop.  I only started here mid 
> July so I’m still learning.
> 
> There are many things here that haven’t been updated in terms of software.  
> That’s what I’m starting to make my way through is the entanglement of old 
> software.  I have been reading the documentation on ldm 4.6.12 and I hope to 
> download and build it next week.

I hope you're referring to LDM 6.12.6 rather than 4.6.12.

> I’m debating right now if I should have the sysadmins bring the gcc and perl 
> up to date on the machines.  I’m sure it would help as well, but it’s a long 
> process to get that done.  At any rate it’s one battle at a time as I map out 
> some milestones for progress.

The only necessity is that the perl interpreter be version 5.

> I still have some uncertainty about the behavior of ldm.  I was under the 
> impression that memory mapped files were mapped to the virtual memory of the 
> machine(physical and swap).  Part of what is perceived with other users here 
> in their experience with ldm is that it is a problematic process on a solaris 
> machine, because when it is running it takes up every bit of the physical 
> memory.  That observation is based on the file size we have configured, so 
> when running with other processes on the server there is plenty of swapping 
> going on.  I guess what you’re telling me does make sense though in the fact 
> that although all the physical memory is consumed, there is still swap space 
> left on the drive, so conceptually the ldm.pq file being memory mapped is 
> like extending the swap space of the OS.

Virtual memory is specific to a process rather than a machine. The LDM 
product-queue is memory-mapped into the virtual memory of each process that has 
it open.

> I wasn’t getting that impression from reading the documentation online, it 
> seemed that the transfer was between the memory mapped file and the virtual 
> memory of the machine, so it made sense to have a swap space large enough to 
> augment the physical memory of the machine and expand the size of the virtual 
> memory to be large enough to put a 20G ldm.pq file into a mapping and still 
> have room left for the other OS processes.

Our advice is to have sufficient physical memory to hold the entire 
product-queue plus the O/S and its buffers. Multiply the size of the 
product-queue by 3/2 and you should be good.

> The reason we have a large file size for the queue is we would like to have 
> more than an hours worth of time for files.  Looking at our stats our mean 
> retention time right now is about five days.  Probably way too much as I 
> suspected so I’ll definitely consider trimming down the file size of ldm.pq 
> and see how that effects the performance and delivery.

5 days? Yeah, you can make it smaller.

> In the case of attempting to insert a file larger than the ldm.pq size does 
> that imply that LDM-4 will stop processing files through ldm until it is 
> restarted, and with an error thrown in LDM-6 will it continue to process or 
> will it stop ingesting files until restarted as well.

If the files are being inserted via pqinsert(1) *and* that process is running 
*inside* the process-group of the LDM server (by means of an EXEC entry in the 
LDM configuration-file) then the LDM-4 pqinsert(1) process will likely crash 
due to a SIGSEGV and take the entire LDM system down with it. An LDM-6 
pqinsert(1) processs in the LDM server's process group, however, will simply 
error-exit and the LDM system will stay up. If the pqinsert(1) process is not 
in the LDM server's process group, then neither LDM system will terminate. 
There will be a delay up to 30 seconds, however, before the inserted product is 
relayed downstream.

> The reason I would like to stay with the LDM is that it is in place, it 
> recognizes file duplication at the insertion point.  In my previous email I 
> was referring to downstream ldm’s that are requesting duplicated file sets 
> from the up stream.  I think if we can get our end to end path configurations 
> sorted out I believe we can tune the system to meet our needs.  Thanks for 
> the help and if I do get a chance I’ll see about dropping by UCAR after the 
> first of the year to get a better sense of how I could make ldm work for us.

You'd be welcome.

> Thanks
> -Mike

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: EFY-891334
Department: Support LDM
Priority: Normal
Status: Closed