[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #SXI-868914]: Data product maximum size

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [LDM #SXI-868914]: Data product maximum size
  • Date: Tue, 25 Jun 2013 15:25:10 -0600
Hi Leonard,

> We're running LDM 6.10.1 on a mix of Linux CentOS 5 and 6 systems. All
> are 64-bit.  Builds were done on the systems the software runs on.  The
> product queues are 5GB.
> 
> Upstream has started sending data products (files) that are 2.4GB each.
> Some come through, some do not.
> 
> I've seen mention of a 2GB limit in a couple of support tickets. Since
> it is working for some products which are larger than than, I'd like to
> ask what the maximum product size is.

The metadata that accompanies a data-product has a size parameter that's 
interpreted as a 32-bit unsigned integer, so the maximum size of a data-product 
is (2^32)-1 bytes.

The approximately 2GB limit stems from the type of an "off_t" in the C runtime 
library. "off_t"s are used to randomly access the product-queue. On many 32-bit 
systems, they are a 32-bit signed integer, in which case the maximum size of 
the product-queue is limited to (2^31)-1 bytes. Because you have a 64-bit 
system, this doesn't apply and you shouldn't have any problems sending (2^32)-1 
byte products.

> Thanks!
> 
> --
> ==Leonard E. Sitongia
> High Altitude Observatory
> National Center for Atmospheric Research
> P.O. Box 3000 Boulder CO 80307  USA
> address@hidden  voice: (303)497-2454  fax: (303)497-1589

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: SXI-868914
Department: Support LDM
Priority: Normal
Status: Closed