Re: 20050218: restoring headers in large netcdf files

>To: address@hidden
>cc: Bjarne B?chmann <address@hidden>,
>cc: Peter Gylling J?rgensen <address@hidden>
>From: Karsten Bolding <address@hidden>
>Subject: restoring headers in large netcdf files
>Organization: Bolding & Burchard Hydrodynamics
>Keywords: 200502190949.j1J9ncv2029855 netCDF large file header restore

> I've now added NF_64BIT_OFFSET when creating the .nc file and I do
> indeed get:
> kbk@sill01:/data/kbk/dk_03nm/2003/03$ !od
> od -An -c -N4 dk_03nm.3d.000.nc 
>    C   D   F 002
> 
> (I actually had to use 'ior(NF_CLOBBER,NF_64BIT_OFFSET)' instead of just
> 'or(..' with the Intel Fortran compiler in a Linux machine - v. 8)

OK, I'm not sure what the difference is between the intrinsic functions
OR(N,M) and IOR(N,M).  Maybe they are two names for the same Boolean
bitwise OR function.

> >   Was it possible to create netCDF files larger than 2 GiBytes before
> >   version 3.6?
> > 
> >   Yes, but there are significant restrictions on the structure of large
> >   netCDF files that result from the 32-bit relative offsets that are
> >   part of the classic netCDF format. For details, see NetCDF Classic
> >   Format Limitations in the User's Guide:
> >      
> > http://www.unidata.ucar.edu/packages/netcdf/netcdf/NetCDF-Classic-Format-Limitations.html
> > 
> > Also to have files with bad headers, I think you would have had to
> > close the CDF2 file and then later reopen it with nf_open() and either
> > enter define mode with nf_redef() or change an attribute value with
> > nf_put_att_TYPE().  If you didn't do all of these things, then I don't
> > think the bug I was referring to could affect you.  
> > 
> > However, it's possible you have identified a different bug, either in
> > the netCDF library or in NCO.  I'm very interested in helping diagnose
> > the problem.
> 
> Before v3.6.0 when you broke the 2GB limit the program stopped with an
> error message from NetCDF (don't remember the exact words now) but now
> it seems you can actually make a file more than 2GB as described in the
> link you give above.

I checked the CDL of the file you sent, and it is the right form to be
representable as a CDF1 ("classic format") file as well as a CDF2
("64-bit offsets") file.  The header and fixed-size variables take
2309824 bytes, and each record takes 12211204 bytes, so your 744
record file should have been 9087445600 bytes, about 8.46 GiBytes.

You didn't have to use the new CDF2 version of the format, and if you
want to be able to use your with other utilities that have not yet
upgraded to netCDF 3.6 or later, it would be better to leave them as
CDF1 files.  I wasn't recommending that you should write them as CDF2
files (by using NF_64BIT_OFFSET).  I was instead saying they would
have had to be written as CDF2 files for you to have experienced the
potentially serious bug I mentioned, where a CDF2 file gets changed
into a CDF1 file when you reopen it for writing and add a new variable
or rewrite an attribute.

> > > Here is a listing of the files:
> > > phy-43:~# ls -lh /nodes/{01,02,03,04}/kbk/dk_03nm/2003/01/*3d*
> > > -rw-r--r--  1 kbk kbk 8.5G Feb 19 00:15 
> > > /nodes/01/kbk/dk_03nm/2003/01/dk_03nm.3d.000.nc
> 
> <snip> 
> 
> > I would be interested in the output of "ncdump -h" for one of these
> > files.  Could you send me what you get when you run
> > 
> >   $ ncdump -h dk_03nm.3d.000.nc
> 
> I've attached the result of the above command to this file.
> 
> > 
> > for example?  I want to determine if they are properly structured CDF1
> > files that meet the constraints for > 2 GiB files without using the
> > 64-bit offsets in the CDF2 format.
> > 
> 
> <snip>
> 
> > netcdf test {
> > // Last fixed variable > 2 GiB.  Expect to fail when using library
> > // not compiled with Large File Support.
> > dimensions:
> >     x = 1024;
> >     y = 1024; 
> >     z = 1025;
> > variables:
> >     float x(x);
> >     float y(y);
> >     float z(z);
> >     short var(x, y, z);
> > data:
> >   var = 1, 2, 3;
> > }
> > 
> 
> As I wrote I've updated to the latest beta version and when I do a
> make extra_text I get:
> ./quick_large_files -f ./example.nc
> *** Creating ./example.nc for 64-bit offset large file test...ok
> :
> :
> *** Now a 64 bit-offset file that just fits...ok
> 
> *** All large file tests were successful.
> *** Success ***
> ls -l ./example.nc 
> -rw-r--r--  1 kbk kbk 12884902184 Feb 21 09:24 ./example.nc
> rm ./example.nc
> 
> but when I do:
> kbk@orca:/data/kbk$ ncgen -v 2 -b large.cdl
> I only get:
> kbk@orca:/data/kbk$ ls -l large.nc
> -rw-r--r--  1 kbk kbk 2149593328 Feb 21 09:26 large.nc
> i.e. limited to 2GB
> 
> After a few minutes !! I actually kill the program - since the file size
> does not change.

Oops, sorry, I should try these examples out before sending them in
replies.  When I run ncgen for the above file, I get

  $ ncgen -v 2 -b large.cdl
  ncgen: large.cdl line 14: out of memory

when it reaches 2 GiB also, but that's a problem with ncgen trying to
allocate enough memory to hold the whole variable, not a problem
trying to write the file.

I've just tried this to produce a CDF1 file that's bigger than 2 GiB
for you to test with NCO.  Put this in "large.cdl":

netcdf large {
dimensions:
        x = 1024;
        y = 1024; 
        z = 1024;
        time = UNLIMITED;
variables:
        byte var(time, x, y, z); // 1 GiB per record
        int time(time);
data:
  time = 1, 2, 3;
}

Then run

  ncgen -b large.cdl; ls -l large.nc

For me this took a little over 3 minutes to produce a file over 3 GiB
in size:

  -rw-r--r--   1 russ     ustaff   3221225648 Feb 22 16:47 large.nc

Now this is a CDF1 (classic) netCDF file, so even older versions of
NCO should be able to handle it, as long as NCO is built with large
file system support.  If you attempt the NCO operation 

  ncks -v time large.nc

I would think it should work, if NCO is built with large file support.
But I don't have NCO built with large file support to try this with
right now.  The version I have from 2003 gives what looks like an
unrelated error:

  $ /scratch/russ/nco-2.7.1/bin/ncks -v time large.nc time.nc
  ncks: ERROR unable to determine method for remote retrieval of large.nc

I will try to build a version of nco with large file support and try
this out to make sure it works as it should, or if not to report it as
an nco bug.

> So I can't really check whether I can use nco on the file large.nc. I
> however tested it on the output files generated with the new library and
> I do get a:
> kbk@phy-43:/nodes/01/kbk/dk_03nm/2003/04$ ncks -v elev dk_03nm.3d.000.nc
> elev.nc
> ERROR: program exiting through nco_err_exit() which will now call
> exit(EXIT_FAILURE)
> nco_err_exit(): ERROR nco_open()
> Not a netCDF file
> 
> i.e. I do not a version of nco with support for large files compiled in
> so I'll have to update nco for further processing. 

--Russ