[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #SVC-505747]: netCDF-4 performance using HDF5 on top of MPIIO

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #SVC-505747]: netCDF-4 performance using HDF5 on top of MPIIO
  • Date: Wed, 20 Mar 2013 21:49:19 -0600
Hi Beom-Soo Kim,

> I've employed netCDF-4 using hdf5 on top of MPIIO (netCDF4-parallel) to a
> few Models.  Some models have moderate performance, some models have not.
> 
> I have some questions.
> 
> First, one of models calls nf90_def_var after calling nf90_redef. In this
> case, I cannot read output by ncdump in netCDF version 4.0.x.  It can
> be read from netCDF version 4.1.x. However, while reading this one by
> ncdump, it takes long time than original output (Not using hdf5 on top
> of MPIIO).  Original output can be read by ncdump in netCDF version 4.0.x.
> Is this bug?

If your netCDF version 4.0.x was built with any of HDF5 versions 1.8.0 through 
1.8.4, you may be seeing symptoms of this HDF5 bug, which was fixed in versions
after 1.8.4:

  http://www.hdfgroup.org/newsletters/bulletin20100226.html

As described in that bulletin, the bug occurred when more than 8 attributes were
added to a variable, as might have happened after you called nf90_def_var().

NetCDF versions 4.1.x were not affected by this HDF5 bug, because they should
have been built with later versions of HDF5, according to our recommendations.

> Second, also this model has poor performance.
> Is there a poor performance issue in this case?

I'm not sure why you are seeing poor performance with parallel I/O in this case.
There have been some improvements to parallel I/O in later versions of netCDF
and HDF5.  The most recent release is netCDF version 4.2.1.1 using HDF5 version
1.8.10-patch1, but I don't know whether upgrading would improve your parallel
I/O performance.

--Russ

> Regards,
> Beom-Soo, Kim
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: SVC-505747
Department: Support netCDF
Priority: Normal
Status: Closed