Re: 20050505:netcdf-4

>To: address@hidden
>cc: address@hidden
>From: Jim Edwards <address@hidden>
>Subject: netcdf-4 
>Organization: UCAR/Unidata
>Keywords: 200505041602.j44G2eP3008413

Hi Jim,

> You are probably aware that NCAR has a single rack bluegene rack on
> the floor.  One of the main applications running on this system is
> the SCD/CSS developed homme <http://www.homme.ucar.edu/> model.
> This model currently uses an embarrassingly parallel I/O model with
> each processor writing an individual netcdf (3) file to disk.  The
> files are later combined in a post processing step using a
> combination of ncl and nco tools.
> 
> How is the netcdf-4 effort going?  Is it at a point where we could
> consider replacing our i/o model and get rid of the postprocessing
> step?  Is there a possibilty of colaboration in development of an
> i/o interface of homme to netcdf-4?

The netCDF-4 effort is going pretty well, but we don't plan to have
the first release out until late July or August.  There will be one
or more beta releases before that, and we already have an alpha
release with parallel I/O implemented, but it is still undergoing
testing and not much documentation is available for it yet.

We recently got a similar question from someone at Sandia Labs who is
writing out 1500 netCDF files in parallel from an application and then
running a big job at the end to read them all in and combine them.
Here's the advice I gave him, which is also relevant to your
situation:

  
http://my.unidata.ucar.edu/cgi-bin/getfile?file=/content/support/help/MailArchives/netcdf/msg03033.html

and there were two follow-ups:

  
http://my.unidata.ucar.edu/cgi-bin/getfile?file=/content/support/help/MailArchives/netcdf/msg03034.html
  
http://my.unidata.ucar.edu/cgi-bin/getfile?file=/content/support/help/MailArchives/netcdf/msg03037.html

In short, if you need something right now in which the interfaces and
format probably won't change, I'd recommend pnetcdf 1.0.0-pre2 from

  http://www-unix.mcs.anl.gov/parallel-netcdf/

If you're willing to take a chance on our alpha release, which may not
build easily on BlueGene, which requires first building a current
snapshot of HDF5, in which the interfaces and format may change before
final release, and for which the documentation is minimal, you could
try our alpha3 release of netcdf-4 from

  ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4

Or you might wait until our beta release or contact Greg Sjaardema to
see if he's come up with a good solution yet.

--Russ