This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hi Steve, > In discussions with some folks on an NSF CyberGIS project they are > looking for a geospatial format that could support parallel IO for > some 2D raster operations like reprojection of large imagery or > elevation data. > > They looked at HDF5 but couldn't find a common approach to handling > the geospatial information. I asked if they had looked at netCDF and > they stopped looking when they saw the note about needing to build > it with HDF5 (since they already saw that as a dead end). NetCDF-4 can be built without HDF5 (configure with --disable-netcdf-4 or just don't provide a path to an HDF5 installation). The resulting library will only support the classic data model and associated classic and 64-bit offset formats. But that's what the parallel netCDF software from Argonne Labs and Northwestern University supports, in addition to a new "CDF-2" format that provides 64-bit everything but sticks to the classic data model: http://trac.mcs.anl.gov/projects/parallel-netcdf It has no dependency on HDF5, and has proven to make efficent use of parallel file systems, as the benchmarking links on that page show. > In looking at the netCDF pages, > http://www.unidata.ucar.edu/software/netcdf/workshops/2010/pnetcdf/index.html > It appears parallel IO is covered in the netCDF4 library. There is also another way to do parallel I/O in netCDF-4, based on HDF5's parallel I/O facilities, which it calls. One nice thing about these two (parallel) parallel I/O systems is they use the same API, with only a single flag in the nc_create_par() function needed to specify which parallel I/O system you want to use. So if you are using the full netCDF-4 with HDF5, you can pretty easily use both and compare them. > Is the netCDF4 library an option you would recommend for this > purpose? Any particular issues or concerns with the parallel > aspects of netCDF4? Each parallel system seems to work fine for some HPC users. If you can get by with the classic model and format (no compression, no chunking), then the Argonne/Northwestern parallel-netCDF is less complex, but I'm not sure how users would compare their performance. The classic model from netCDF-3 is completely sufficient to support the coordinate systems representations of the CF conventions. --Russ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: SSL-586386 Department: Support netCDF Priority: Normal Status: Closed