Re: ldm changes (fwd)

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================

---------- Forwarded message ----------
Date: Thu, 13 Apr 2000 13:21:46 -0600
From: Russ Rew <address@hidden>
To: Robb Kambic <address@hidden>
Subject: Re: ldm changes (fwd) 

Robb,

Mike asked:

> Do you know when/if a future release of the LDM will include support
> for a product queue larger than 2GB?  I guess this would require
> converting all the (unsigned) int's to long's in the product queue
> code.  Most all of our operating systems have O_LARGEFILE support
> except Linux and that is in progress.

When I first looked at this, I though it would be easy to make 4 Gbyte
queues instead of 2 Gbyte queues, by just changing the use of the
32-bit signed type off_t to the 32-bit unsigned type size_t, which
could hold file offsets between 0 and 4 Gbytes.  However, it turns out
that the system calls for locking regions of a file only accept off_t
offsets, so this simple modification won't work.  

There are two ways I can think of to support larger product queues:

 1.  Use the O_LARGEFILE support Mike mentions.
 2.  Use multiple queues, each limited to 2 Gbytes, or

Alternative 1 would remove all practical queue size limitations
forever, but is a pretty big change.  I'm not sure how much effort it
would be to change a large program like the LDM from using 32-bit file
offsets (type off_t) to using 64-bit file offsets (type off64_t) in a
portable way, but I know lots of changes would be required.  A couple
of changes would even show through the pq library public interface,
though most other client programs that use the pq library could be
insulated from the change.

The biggest uncertainty for me is the amount of standardization for
system calls with 64-bit types among Solaris, HP-UX, AIX, OSF, and
Linux systems.  If these all adhere to the same interfaces for
seeking, locking, mmap calls, and I/O, the conversion would be fairly
straightforward (maybe a couple of week's worth of work at the most).
If there are OS dependencies or multiple ways to provide 64-bit
support, it might take considerably longer.

I haven't thought much about the practicality of using multiple 2
Gbyte queues instead.  It might be possible to hide such an
implementation in the pq layer.

--Russ