Re: 20040818: Very large netCDF files

Tom,

> ... where do I look to see the nc_create that you mentioned when 
> the files are being written out and where is NC_64BIT_OFFSET specified?

Ed has pointed out the C call, but if you are using the Fortran-77 or
Fortran-90 interface instead, you should look for the corresponding
NF_CREATE or NF90_CREATE call and verify that it uses the appropriate
flag, as documented in the new user guides for Fortran-77 and
Fortran-90:

http://my.unidata.ucar.edu/content/software/netcdf/netcdf-4/newdocs/netcdf-f77/NF_CREATE.html

and

http://my.unidata.ucar.edu/content/software/netcdf/netcdf-4/newdocs/netcdf-f90/NF90_CREATE.html
 

(Ed is updating these, so if they don't mention the 64-bit offsets
flag, they will shortly.)  

> Russ, it would seem impractical for me to transfer all the SI 
> directories and codes (there are many) and input data files (also very 
> many different ones) for our very large run for you to try to run it 
> yourself.  There are still significant parts of it that we must simply 
> run as black boxes so I can't explain alot of the guts of it to you.  
> Let me know what you think about this.

OK, we'll try a different approach.  Dave Gill sent us a CDL file that
provides the structure of a WRF dataset, which for a single time is
2.66 GBytes.  It starts out:

  netcdf wrfinput_d01 {
  dimensions:
          Time = UNLIMITED ; // (1 currently)
          DateStrLen = 19 ;
          west_east = 1599 ;
          south_north = 1199 ;
          west_east_stag = 1600 ;
          bottom_top = 34 ;
          south_north_stag = 1200 ;
          bottom_top_stag = 35 ;
          ext_scalar = 1 ;
          DIM0009 = 5 ;
          land_cat_stag = 24 ;
          soil_cat_stag = 16 ;
          soil_layers_stag = 5 ;
  variables:
          char Times(Time, DateStrLen) ;
          float LU_INDEX(Time, south_north, west_east) ;
                  LU_INDEX:FieldType = 104 ;

and continues with a large number of variables.  We'll try
constructing a test case with this, writing a few values in one of the
last variables

          float SNOWC(Time, south_north, west_east) ;

and then reading them back in to verify the values.  If that doesn't
work, we'll have something here we can use to fix the problem.  If it
works OK, we'll send you the code for that test and see if it works
for you.  If not, then we'll be able to work on the platform-specific
problem.

Would it be better for you if we used the Fortran-90 interface or the
Fortran-77 interface instead of the C interface for the test case?
I suspect you're using Fortran-90, but I'm not sure.

--Russ