Re: Notes on Linux Build for Large Queue

Daryl,

>       This well over the border of my knowledge, but I thought that
> since I am running kernel 2.4.3 and glibc-2.2.2-10 that LFS support is now
> transparent in Linux.  By adding the compile options to gcc, it would
> automatically change references to use 64 Bits.  I have no clue how it
> works nor do I want to.
> 
> I look at my queue files which are well over 2 G
> 
> ls -l data/
> total 17906352
> -rw-rw-r--    1 ldm      ldm      4113330176 Aug  7 16:50 ldm.pq
> -rw-rw-r--    1 ldm      ldm      4376367104 Aug  7 14:47 ldm.pq-4g
> -rw-rw-r--    1 ldm      ldm      6564175872 Aug  7 13:58 ldm.pq-6g
> -rw-rw-r--    1 ldm      ldm      3282219008 Aug  7 15:02 ldm.pq-works
> 
> pqmon reports that my queue size is way over 2 GB
> 
> pqmon -i 3
> Aug 07 22:31:16 pqmon: Starting Up (4678)
> Aug 07 22:31:16 pqmon: nprods nfree  nempty      nbytes  maxprods  maxfree  
> minempty    maxext  age
> Aug 07 22:31:16 pqmon:  66953     1  918646   822346992     66953        1    
> 918646 3214670608 2424
> 
> I filled the 3 GB queue this morning and could see old products get
> overwritten by watching pqmon (with Linux)
> 
> I will see how much my current 3.85 G queue holds.  I am guessing it will
> hold around 9 hours worth.  (I hope)
> 
> Thanks again.  This is facinating to me!  I am basing some of my acquired
> knowledge from this webpage.  It is a bit out-dated.
> 
> http://www.suse.de/~aj/linux_lfs.html

OK, thanks, I think I understand the situation now.

For the LDM to support large queues, it is not sufficient for the
platform on which it is compiled to only provide LFS (Large File
Support).  The platform must also support the "LP64 model", where
longs and pointers are represented with 64-bits.  This requires a
64-bit operating system, not just LFS support on a 32-bit operating
system.

It would have been possible to write the LDM code to use the
"Transitional Extensions to the Standard Unix Specification" so it
could use LFS on 32-bit platforms, but this would have required fairly
extensive changes to the LDM code.  Every occurrence of the type
"off_t" would have had to be conditionally compiled to be either
"off_t" or "off64_t", depending on whether the code was compiled for
LFS, and similarly for every I/O interface (such as open()
vs. open64()), every associated manifest constant (such as FGETLK vs
FGETLK64), every associated data type, etc.  The addition of all this
conditional compilation would have made the source significantly
harder to understand and maintain.

Instead, we have sources that work unchanged on 32-bit platforms and
64-bit platforms; only the compile flags have to change.  But the down
side is we don't provide transitional large file support for 64-bit
file offsets on 32-bit systems.

The fact that the LINUX executables appear to work for files smaller
than 4 Gbytes may just be luck.  It's possible that some assertion
will be violated the first time a difference must be calculated
between file offsets for products that are further apart than 2
Gbytes.  I'm surprised it seems to work as well as it does, so maybe
it will continue to work for a while, but that won't be because we
designed it that way or tested it on 32-bit platforms with queues
larger than 2 Gbytes.

I'll be interested to hear what happens.  I think I should at least
provide better documentation for some of the assumptions we're making
about large file support on the web page we have for that:

  http://www.unidata.ucar.edu/packages/ldm/huge-queues.html

and if your LINUX system with big queues continues to work, we can add
that "discovery" to the web page as well :-).

--Russ