[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: large netCDF data sets

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: large netCDF data sets
  • Date: Thu, 19 Aug 2004 09:54:45 -0600
Hi,

Yesterday, I wrote:

> Using the CDL Dave Gill sent, I've got a test case that doesn't work
> as intended with the netCDF 3.6.0-beta library and large files, so it
> looks like a bug in the library.  Please stand by while we track this
> down and get a fix ...

It turns out there was a bug in the ncgen utility we were using to
generate the test, and an assertion in the netCDF library that should
have been deleted.  After fixing these bugs, everything seems to work
as expected with the CDL file wrfinput_d01.cdl.

I tried writing values in the first variables Times and LU_INDEX and
the last few variables, TMN, XLAND, SNOWC, then reading them back in a
separate program, and they look fine.  This is with one record, in a
2.66 Gbyte file.  The values in the variables TMN, XLAND, and SNOWC
are definitely beyond 2 Gbytes in the file (they start respectively at
bytes 2638488232, 2646157036, and 2653825840).

Ed will let you know when the new beta release is ready for download.

Here's a Fortran program (generated by ncgen with a little manual
simplification) that writes and reads the data, so you can try this to
see if you get the same results we do when you link it against the new
netCDF-3.6.0 beta release.  This is Fortran-77, because ncgen doesn't
generate Fortran-90 (yet).  Hopefully you can still use it:

  http://my.unidata.ucar.edu/content/staff/russ/tmp/wrf_test.f

Here's how I compiled and ran it on a Solaris platform:

  $ f90 -g -w -I/home/russ/work/include wrf_test.f -o wrf_test 
-L/home/russ/work/lib -lnetcdf

On a different platform, you may need different compiler flags, and
the above directory paths are just where I installed a test copy.

Then when I run it, it takes a while to write the file wrf_test.nc:

  $ ./wrf_test
  $ ls -l wrf_test.nc
  -rw-r--r--   1 russ     ustaff   2661494168 Aug 19 09:24 wrf_test.nc

I can verify various values were written to the file by looking at it
with the "ncdump" utility that is part of the 3.6.0 distribution.  For
example, to verify that the first value of the SNOWC variable is "17",
the nonsense value that was written to that variable in the Fortran-77
program, I use:

  $ /home/russ/work/bin/ncdump -v SNOWC wrf_test.nc | head -780
  netcdf wrf_test {
  dimensions:
          Time = UNLIMITED ; // (1 currently)
          DateStrLen = 19 ;
          west_east = 1599 ;
   ...
                  :MAP_PROJ = 1 ;
                  :MMINLU = "USGS" ;
  data:

   SNOWC =
    17, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, 

where the "head -780" is just to limit the output to the first 780
lines, which shows the start of the data.  Without this, you would
also see all the fill values in SNOWC that weren't written.

--Russ