Re: 20050218: restoring headers in large netcdf files (fwd)

>To: address@hidden
>cc: Bjarne B?chmann <address@hidden>,
>cc: Peter Gylling J?rgensen <address@hidden>
>From: Karsten Bolding <address@hidden>
>Subject: restoring headers in large netcdf files
>Organization: Bolding & Burchard Hydrodynamics
>Keywords: 200502190949.j1J9ncv2029855 netCDF large file header restore

I'm filing this to show that the problem was resolved.

------- Forwarded Message

Date:    Thu, 03 Mar 2005 09:50:34 +0100
From:    Karsten Bolding <address@hidden>
To:      Russ Rew <address@hidden>
cc:      Karsten Bolding <address@hidden>,
         address@hidden, Bjarne B?chmann <address@hidden>,
         Peter Gylling J?rgensen <address@hidden>
Subject: Re: 20050218: restoring headers in large netcdf files

Hi Russ

I know I promised a reply already last saturday but time slipped ...  
better late than never ..

<snip>
> 
> Karsten,
> 
<snip>
> 
> Having just built NCO 2.9.9 with large file support (I used
> --enable-largefiles as a configure argument when building, but that
> may be the default), I can now verify that ncks built this way does
> work fine on the above 3 GiB netCDF classic format (CDF1) file:
> 
>   $ ncks -v time large.nc time.nc
>   $ ncdump time.nc
>   netcdf time {
>   dimensions:
>         time = UNLIMITED ; // (3 currently)
>   variables:
>         int time(time) ;
> 
>   // global attributes:
>                 :history = "Wed Mar  2 14:58:55 2005: ncks -v time large.nc 
> time.nc" ;
>   data:
> 
>    time = 1, 2, 3 ;
>   }
> 
> If the ncks you have works the same way on the same file generated by
> ncgen, then I think it should work fine on the large CDF1 files you
> generated with the netcdf=3.6.0 version library.  If ncks fails on
> this file, then you need to upgrade your NCO installation.
> 
> I also generated a 64-bit offset (CDF2) netCDF file from the above
> large.cdl file, using ncgen:
> 
>   $ ncgen -v 2 -b large.cdl; ls -l large.nc
>   -rw-r--r--   1 russ     ustaff   3221225656 Mar  2 15:08 large.nc
> 
> and tested the same ncks, to make sure it would work with both CDF1
> and CDF2 files.  It worked identically, so for ncks (and any other
> application compiled with netCDF 3.6), it shouldn't matter whether you
> have CDF1 or CDF2 files.  But I would recommend just using CDF1 files
> for as long as you can, until applications are upgraded to version
> 3.6.

Intel Xeon based system
kbk@phy-43:/data/kbk$ time ncgen -b large.cdl

real    4m54.871s
user    0m4.179s
sys     0m12.293s
kbk@phy-43:/data/kbk$ ls -lh large.nc
-rw-r--r--  1 kbk kbk 3.1G Mar  3 08:42 large.nc

AMD Opteron based system
kbk@m01:~> time ./local/netcdf-3.6.1-beta3/bin/ncgen -b large.cdl

real    0m57.479s
user    0m2.315s
sys     0m12.396s
kbk@m01:~> ls -lh large.nc 
-rw-r--r--  1 kbk gast 3,1G 2005-03-03 08:53 large.nc

Note the very big difference in generation time. Both system were idle
when I did the test. I'll have to look into that but it is a different
matter.

On both architectures I can do a ncks -v time large.nc out.nc and get
the correct output.

I experienced one problem though. On the Opteron system I can't find the
correct flags to compile the netcdf library with support for the Intel
Fortran compiler. I can compile netcdf on an Intel based machine and
copy the libaries to the Opteron machine and then I can compile and link
my own Fortran programs. Do you have successfully compiled and installed
netcdf on a Opteron based system with support for the Intel compiler (v
8.1)? And if so can you provide me with the settings? 

To build nco with support for large files on the Opteron I had to build
the library without support for Fortran. I actually had to set
CFLAGS=-fPIC or the nco configure script complained.

After the above described execises I returned to the original problem
with my large output files. The short summary is that with a properly
compiled nco I can extract both the clasical and new formatted netcdf
files :-)

> 
> --Russ
> 


Karsten

-- 
----------------------------------------------------------------------
Karsten Bolding                    Bolding & Burchard Hydrodynamics
Strandgyden 25                     Phone: +45 64422058
DK-5466 Asperup                    Fax:   +45 64422068
Denmark                            Email: address@hidden

In a world without walls and fences, who need windows and gates 
----------------------------------------------------------------------

------- End of Forwarded Message