[netCDF #SSL-586386]: Parallel OI in netCDF4 library

Hi Steve,

> In discussions with some folks on an NSF CyberGIS project they are
> looking for a geospatial format that could support parallel IO for
> some 2D raster operations like reprojection of large imagery or
> elevation data.
> 
> They looked at HDF5 but couldn't find a common approach to handling
> the geospatial information. I asked if they had looked at netCDF and
> they stopped looking when they saw the note about needing to build
> it with HDF5 (since they already saw that as a dead end).

NetCDF-4 can be built without HDF5 (configure with --disable-netcdf-4
or just don't provide a path to an HDF5 installation).  The resulting
library will only support the classic data model and associated
classic and 64-bit offset formats.  But that's what the parallel
netCDF software from Argonne Labs and Northwestern University
supports, in addition to a new "CDF-2" format that provides 64-bit
everything but sticks to the classic data model:

  http://trac.mcs.anl.gov/projects/parallel-netcdf

It has no dependency on HDF5, and has proven to make efficent use of
parallel file systems, as the benchmarking links on that page show.

> In looking at the netCDF pages,
> http://www.unidata.ucar.edu/software/netcdf/workshops/2010/pnetcdf/index.html
> It appears parallel IO is covered in the netCDF4 library.

There is also another way to do parallel I/O in netCDF-4, based on
HDF5's parallel I/O facilities, which it calls.  One nice thing about
these two (parallel) parallel I/O systems is they use the same API,
with only a single flag in the nc_create_par() function needed to
specify which parallel I/O system you want to use.  So if you are
using the full netCDF-4 with HDF5, you can pretty easily use both and
compare them.

> Is the netCDF4 library an option you would recommend for this
> purpose?  Any particular issues or concerns with the parallel
> aspects of netCDF4?

Each parallel system seems to work fine for some HPC users.  If you
can get by with the classic model and format (no compression, no
chunking), then the Argonne/Northwestern parallel-netCDF is less
complex, but I'm not sure how users would compare their performance.
The classic model from netCDF-3 is completely sufficient to support 
the coordinate systems representations of the CF conventions.

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: SSL-586386
Department: Support netCDF
Priority: Normal
Status: Closed