[netCDF #SVC-505747]: netCDF-4 performance using HDF5 on top of MPIIO

Hi,

> The http://www.hdfgroup.org/newsletters/bulletin20100226.html is not my case.
> The netCDF 4.0.x was built with hdf5 1.8.4.1.
> And I got the below error message when I try to run ncdump command.
> 
> "Can't open HDF5 attribute"
> 
> You can see the message with test code(f90tst_parallel.f90) which is in 
> netCDF source directory.
> I've added the functions in the test code.
> 
> ! With the classic model netCDF-4 file, enddef must be called.
> Call handel_err(nf90_enddef(ncid))
> 
> --- added
> call handle_err(nf90_redef(ncid))
> call handel_err(nf90_def_dim(ncid,"test",nz,z_dimid))
> call handel_err(nf90_def_var,"test",nf90_int,z_dimid,zvarid))
> call handel_err(nf90_enddef(ncid))
> 
> If you compile with netCDF4 using hdf5 on top of MPIIO, you can see the error 
> message as running ncdump in netCDF 4.0.x.
> 
> And when converting output created with parallel I/O with netCDF4 and not 
> parallel I/O with netCDF4 to text form by ncdump in netCDF version 4.1.x 
> through 4.2.0, the output created with parallel I/O with netCDF4 takes quit 
> long time than the output created with serial netCDF4.
> So I guess it's something to do with poor performance.

All I can suggest is upgrading to version 4.2.x or the upcoming 4.3.0 release.  
The 4.0.1 release had lots of bugs that
have since been fixed, and the same is true of older HDF5 1.8.x releases.  For 
example, we recently encountered a
similar bug with files written using parallel-netcdf not permitting adding new 
variables or other metadata correctly 
using serial netCDF-4, and a fix for that bug will be in netCDF 4.3.0:

   Fix bug with some pnetcdf files getting corrupted by calling nc_enddef in a 
file open for writing
   https://bugtracking.unidata.ucar.edu/browse/NCF-234

Anyway, thanks for reporting the problem.

--Russ

> Regards,
> Beom-Soo
> -----Original Message-----
> From: Unidata netCDF Support [mailto:address@hidden]
> Sent: Thursday, March 21, 2013 12:49 PM
> To: Beom-Soo Kim
> Cc: address@hidden
> Subject: [netCDF #SVC-505747]: netCDF-4 performance using HDF5 on top of MPIIO
> 
> Hi Beom-Soo Kim,
> 
> > I've employed netCDF-4 using hdf5 on top of MPIIO (netCDF4-parallel)
> > to a few Models.  Some models have moderate performance, some models have 
> > not.
> >
> > I have some questions.
> >
> > First, one of models calls nf90_def_var after calling nf90_redef. In
> > this case, I cannot read output by ncdump in netCDF version 4.0.x.  It
> > can be read from netCDF version 4.1.x. However, while reading this one
> > by ncdump, it takes long time than original output (Not using hdf5 on
> > top of MPIIO).  Original output can be read by ncdump in netCDF version 
> > 4.0.x.
> > Is this bug?
> 
> If your netCDF version 4.0.x was built with any of HDF5 versions 1.8.0 
> through 1.8.4, you may be seeing symptoms of this HDF5 bug, which was fixed 
> in versions after 1.8.4:
> 
> http://www.hdfgroup.org/newsletters/bulletin20100226.html
> 
> As described in that bulletin, the bug occurred when more than 8 attributes 
> were added to a variable, as might have happened after you called 
> nf90_def_var().
> 
> NetCDF versions 4.1.x were not affected by this HDF5 bug, because they should 
> have been built with later versions of HDF5, according to our recommendations.
> 
> > Second, also this model has poor performance.
> > Is there a poor performance issue in this case?
> 
> I'm not sure why you are seeing poor performance with parallel I/O in this 
> case.
> There have been some improvements to parallel I/O in later versions of netCDF 
> and HDF5.  The most recent release is netCDF version 4.2.1.1 using HDF5 
> version 1.8.10-patch1, but I don't know whether upgrading would improve your 
> parallel I/O performance.
> 
> --Russ
> 
> > Regards,
> > Beom-Soo, Kim
> >
> >
> Russ Rew                                         UCAR Unidata Program
> address@hidden                      http://www.unidata.ucar.edu
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: SVC-505747
> Department: Support netCDF
> Priority: Normal
> Status: Closed
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: SVC-505747
Department: Support netCDF
Priority: Normal
Status: Closed