[netCDF #MOB-465267]: Increasing maximum record size

Hi Sergey,

I've finally gotten time to try to reproduce the problem you are seeing.

First I built a separate version of the netCDF 3.6.1 library (which I
will subsequently call the "biglib" version), by changing the two
macros in netcdf.h to allow more variables and attributes:

  #define NC_MAX_ATTRS  40000    /* max global or per variable attributes */
  #define NC_MAX_VARS   40000    /* max variables per file */

I was skeptical this would be a factor or that I would see any
difference with biglib from the standard 3.6.1, but I nevertheless ran
tests with both versions of the library, and appropriate ncdump and
ncgen programs making use of the corresponding library.  I made a
symbolic link to the ncgen that could handle 40000 variables
"ncgen-big", and the standard ncgen from 3.6.1 just "ncgen".

In the first test, I used ncgen to try to generate a netCDF file with
two large fixed-size 4-byte variables, each dimensioned for
43x43x14x143x231, and noted that no error messages were generated with
either the standard 3.6.1 library or biglib.  In each case the file
was generated using

  $ ./ncgen -b -x -v2 sergeyf2.cdl && ls -l sergeyf2.nc
  -rw-r--r--   1 russ     ustaff   6841210256 Feb  6 08:46 sergeyf2.nc
  $ ./ncgen-big -b -x -v2 sergeyf2.cdl && ls -l sergeyf2.nc
  -rw-r--r--   1 russ     ustaff   6841210256 Feb  6 08:46 sergeyf2.nc

("-b" means generate a binary netCDF file corresponding to the CDL
input.  "-v2" means use version 2 of the format, with 64-bit offsets.
"-x" just means set NOFILL mode so I don't have to wait forever for
the big files to be created with fill values.)

I also noted that if the declaration for either variable is changed to
be type double (8 bytes) instead of type float (4 bytes), an error
results immediately (because the variable is larger than 4 Gbytes and
there are record variables):

  $ ./ncgen -b -x -v2 sergeyf2a.cdl && ls -l sergeyf2a.nc
  ncgen: One or more variable sizes violate format constraints
  $ ./ncgen-big -b -x -v2 sergeyf2a.cdl && ls -l sergeyf2a.nc
  ncgen: One or more variable sizes violate format constraints

but that if I don't define a record dimension (change TSTEP from
UNLIMITED to 1), then there are no record variables, so the last
fixed-size variable can be any size.  So in that case making the last
variable type double is OK:

  $ ./ncgen -b -x -v2 sergeyf2b.cdl && ls -l sergeyf2b.nc
  -rw-r--r--   1 russ     ustaff   7715808599 Feb  6 08:57 sergeyf2b.nc
  $ ./ncgen-big -b -x -v2 sergeyf2b.cdl && ls -l sergeyf2b.nc
  -rw-r--r--   1 russ     ustaff   7715808599 Feb  6 08:57 sergeyf2b.nc

Then I changed the last two variables to be record variables, by given
them each a first dimension of TSTEP:

  $ ./ncgen -b -x -v2 sergeyv2.cdl && ls -l sergeyv2.nc
  -rw-r--r--   1 russ     ustaff   6841210264 Feb  6 09:00 sergeyv2.nc
  $ ./ncgen-big -b -x -v2 sergeyv2.cdl && ls -l sergeyv2.nc
  -rw-r--r--   1 russ     ustaff   6841210264 Feb  6 09:01 sergeyv2.nc

Again, I verified that making the first of type "double" violates the
size constraints:

  $ ./ncgen -b -x -v2 sergeyv2a.cdl && ls -l sergeyv2a.nc
  ncgen: One or more variable sizes violate format constraints
  $ ./ncgen-big -b -x -v2 sergeyv2a.cdl && ls -l sergeyv2a.nc
  ncgen: One or more variable sizes violate format constraints

but making the second type "double" instead works fine:

  $ ./ncgen -b -x -v2 sergeyv2b.cdl && ls -l sergeyv2b.nc
  -rw-r--r--   1 russ     ustaff   7715808607 Feb  6 09:04 sergeyv2b.nc
  $ ./ncgen-big -b -x -v2 sergeyv2b.cdl && ls -l sergeyv2b.nc
  -rw-r--r--   1 russ     ustaff   7715808607 Feb  6 09:05 sergeyv2b.nc

So then, still skeptical that the actual number of record variables
can have any effect, I created 33033 extra record variables, all of
the form:

        float vnnnnn(TSTEP, LAY, ROW, COL) ;
                vnnnnn:long_name = "vnnnnn" ;
                vnnnnn:units = "moles/s" ;
                vnnnnn:var_desc = "Model species vnnnnn" ;

where nnnnn goes from 00000 to 33032 in steps of 1.  I inserted
statements of the form above into CDL files and used ncgen-big to try
to reproduce the problem you are seeing, using a file that has 33033
variables substituting for the "NO" and "NO2" variables in the files
sergey*.cdl.

This took some time, but produced a 10 GByte file with no error
messages:

  $ ./ncgen-big -b  -x -v2 sergeyv33k.cdl && ls -l sergeyv33k.nc
  -rw-r--r--   1 russ     ustaff   10266732364 Feb  6 11:28 sergeyv33k.nc

But if I declare the first of those last two variables

        float var1(TSTEP,VARA,VARB,LAY,ROW,COL); // 43x43x14x143x231
        float var2(TSTEP,VARA,VARB,LAY,ROW,COL); // 43x43x14x143x231

(var1) to be type double instead of float, it exceeds the format
variable size limits, and the error message properly reflects that:

  $ ./ncgen-big -b  -x -v2 sergeyv33ka.cdl && ls -l sergeyv33ka.nc
  ncgen: One or more variable sizes violate format constraints

whereas if the last record variable (var2) is declared to be of type
double instead, everything works fine, since the last record variable
can use up to 4 GBytes per record:

  $ ./ncgen-big -b  -x -v2 sergeyv33kb.cdl && ls -l sergeyv33kb.nc
  -rw-r--r--   1 russ     ustaff   11141330707 Feb  6 12:55 sergeyv33kb.nc

My concludions are:
    
  - The problem you are seeing has no relation to how many variables
    there are or whether you have changed the definition of the
    NC_MAX_VARS or NC_MAX_ATTRS macros in netcdf.h.

  - There is no record length limit that is apparent as part of the
    problem.  The netCDF record length for files with 64-bit offsets
    is over 8 Exabytes, and none of these examples are near that
    limit.  There is a documented limit of 4 GBytes per record for
    each record variable.  (See the netCDF FAQ for why this limit
    exists.)

  - The library correctly detects variables that are larger than about
    4 GBytes per record unless they occupy the only place such
    variables can be located, as the last record variable or the last
    fixed-length variable in a file with no record variables.

So the bottom line is I can't reproduce the error you are seeing.
Everything is working as expected witth netCDF version 3.6.1.  

I didn't ask this before, but is it possible that you have another
dimension in the variable you are trying to declare, to make it larger
than 4 GBytes per record?

Is it possible you have declared the variable to be of type double, so
it takes more than 4 GBytes per record?

If you want to try any of these tests to try to further diagnose the
problem, you can download the CDL files I used from this directory:

  http://www.unidata.ucar.edu/staff/russ/test/

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: MOB-465267
Department: Support netCDF
Priority: High
Status: Closed