Re: netcdf/nco bug

Mike,

> Thanks for the notes. I rebuilt NCO against:
> NETCDF_LIB=/usr/local/apps/netcdf-3.6.0-beta6/lib64/r4i4
> NETCDF_INC=/usr/local/apps/netcdf-3.6.0-beta6/include
> 
> But I still have the same problem with ncks.
> 
> If I build b+a.nc, I get a few vbls out of a but not many.
> 
> I'm stumped. I'm going to read your FAQ in depth but if you have any  
> further suggestions I'd be anxious to hear them.

In that case, I suspect this may be a problem with ncks, which will
need to be updated for use with netCDF-3.6.0.  It's author (Charlie
Zender) hasn't had a chance yet to test his utilities with the
netCDF-3.6.0 library, and make whatever changes are necessary.

Now that I think about it, I'll bet that ncks needs to be modified in
the following way:
 
 - By looking at the sizes of things and doing some arithmetic, it
   should determine whether the output netCDF file requires 64-bit
   offsets to the beginning of at least one of its variables.

 - If a 64-bit offset output file is needed, it should call
   nc_create() with the extra mode flag NC_64BIT_OFFSET.

Otherwise, it will still be creating classic-format netCDF files with
their rigid constraints required by using 32-bit file offsets.

Alternatively ncks and the other netCDF operators could add an
additional flag that indicates output files should use the 64-bit
offset format and let the user determine when and whether that is
necessary.

As a workaround for this user, you could test my theory by just
modifying the nc_create() calls in ncks to always generate files in
the new 64-bit offset format and see if it then works.  However, these
files will not be accessible by other utilities that haven't also be
linked against 3.6.0, so this might be a problem until the transition
is further along.

Until Charlie Zender actually has a chance to look at this and decide
how best to update his utilities to make use of the new netCDF large
file facilities, I would recommend against users trying to manipulate
large netCDF files with ncks linked against the 3.6.0 library.

However, it should still be possible for the model Jeff is using to
directly write the large file he wants by using 3.6.0 and the
appropriate nf90_create() flag for 64-bit offset files.

I'm CC:ing Charlie Zender on this, to let him know about the
potential problem we are creating by releasing this beta software
before third-party developers have a chance to update their
utilities.  However, I think we make it clear in the accompanying FAQ
that we discourage use of the new format unless it is absolutely
necessary due to the transition necessary to allow third-party
software to be upgraded to the new release:

  http://my.unidata.ucar.edu/content/software/netcdf/faq-lfs.html

Charlie, I'm appending a draft of the note I'll send to netcdfgroup
soon to announce this beta release.  It's really intended for
developers like you to give us feedback on what's in this release.
Sorry there are people jumping the gun on this, causing transition
problems like this one.

--Russ

 To: netcdfgroup
 Cc:
 Subject: netCDF 3.6.0-beta6 with Large File Support
 Reply-to: address@hidden
 Organization: UCAR Unidata Program
 Fcc: +outbox

 Hi,

 A new beta release of netCDF-3.6.0 is available for external testing:

   ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-beta.tar.gz

 Simplified installation instructions for the beta release are
 available at:

   http://www.unidata.ucar.edu/packages/netcdf/INSTALL-beta/

 This release is close to final, but we would appreciate feedback on
 any problems you might discover.

 NetCDF version 3.6 improves large file support, Windows compatibility,
 ease of installation, and performance using the Fortran-90 interface.
 In addition, it fixes a few bugs, mostly associated with large file
 support.

 In previous releases it was possible to create netCDF files that
 exceeded 2 GiB by observing fairly severe constraints on the structure
 of the data.  For example, you could use a single large fixed-size
 variable for the last variable with no record variables, or you could
 use the record dimension for all large variables with a large number
 of records.  Such restrictions were necessary due to the use of 32-bit
 fields in the netCDF format for offsets that point to the beginning of
 data for each variable.

 Version 3.6 introduces a new variant of the netCDF file format, while
 preserving backward compatibility with the existing format.  The new
 variant has 64-bit fields for file offsets, removing many of the size
 constraints for netCDF files.  Greg Sjaardema of Sandia Labs provided
 the initial version of the C library code to support 64-bit offsets,
 and we added changes to increase the maximum variable and record size,
 to permit use from Fortran, C++, and Java interfaces, to support
 creation of large files with the ncgen utility, and to check declared
 variable shapes for conformance with remaining size constraints.  The
 new library will access data transparently in both the classic and
 64-bit offset formats.

 Version 3.6 is configured to make use Large File Support by default if
 available on the target platform.  Building and installing from source
 has been simplified so that there is no need to set environment
 variables on most platforms.

 We are maintaining a list of questions and answers for netCDF large
 file support:

   http://www.unidata.ucar.edu/packages/netcdf/faq-lfs.html

 More information about the 3.6.0-beta release is contained in the
 RELEASE_NOTES: 

   http://www.unidata.ucar.edu/packages/netcdf/release-notes-3.6.0-beta6

 or the new draft documentation:

   http://www.unidata.ucar.edu/packages/netcdf/newdocs/netcdf/index.html

 If you have other questions or encounter problems with the beta
 release, please contact address@hidden.  Thanks!

 -- Russ Rew
 -- Ed Hartnett