[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #VFX-630532]: Writing multiple variables at once using NetCDF4 parallel

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #VFX-630532]: Writing multiple variables at once using NetCDF4 parallel
  • Date: Tue, 09 Jun 2015 20:45:49 -0600
Charles,

> Sorry to bear bad news, but it appears that you tweaked the loop over
> processor so now EVERY processor defines EVERY variables.
> 
> I guess I can live with that. But if I ask every processor to create only
> the variables it will actually write I end up with:
> 
> doutriaux1@sofia:[build_parallel]:[6434]> ncdump -h try.nc
> ncdump: try.nc: NetCDF: HDF error
> 
> Does the def part HAS TO BE collective? Can I have a def mode has
> independent as well? I¹ll try this.

Yes, according to my understanding, HDF5 requires that defining any
metadata for the file (even including attributes) has to be collective.

Here's an excerpt from the HDF5 FAQ that explains why:

  ... The HDF library has a requirement that all metadata updates be done 
  collectively so all processes see the same stream of metadata updates. 
  This is how HDF5 was designed. Breaking the collective requirements for 
  metadata updates has been discussed previously and we know that it is 
  worth having for certain scenarios, but it is just not possible at the 
  moment without a lot of research and funding.
  ( https://www.hdfgroup.org/HDF5/faq/parallel.html#attr )

So now I guess I'm the bearer of bad news. But defining netCDF-4
variables collectively is not generally a costly operation, compared
with writing or accessing the data. Unless you determine that this
is a bottleneck for your application, maybe it should not be a concern ...

--Russ

> Thanks
> 
> C.
> 
> 
> 
> On 6/9/15, 1:14 PM, "Unidata netCDF Support"
> <address@hidden> wrote:
> 
> >Oops again, please ignore the previous "solution" I attached. Looking
> >at the CDL, it's apparent I messed up the dims used to define the
> >variables.  This one is right, and also uses your original strided
> >writes.
> >
> >--Russ
> >
> >Russ Rew                                         UCAR Unidata Program
> >address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> >Ticket Details
> >===================
> >Ticket ID: VFX-630532
> >Department: Support netCDF
> >Priority: Normal
> >Status: Closed
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: VFX-630532
Department: Support netCDF
Priority: Normal
Status: Closed