[netCDF #LTC-671596]: Multithreaded IO with libnetcdf4

Netcdf-3 (classic) files are not thread safe either.

There is one more complication. As of hdf5 version 1.10,
the HDF5 library supports single-writer,multiple-reader
semantics on datasets. I suspect that this is where the
performance improvements are coming from. Of course, this
is operating in a multi-threaded environment to make use
of multiple-readers.

My hope is that when I add thread-safety
to netcdf-c, I can expose that semantics as well. 

> 
> Thank you for the detailed reply Now it is more clear. So the basic
> message is that any opertions on .nc4 filers are not thread-safe. Right?
> 
> What about reading classic netcdf files? Are those calls thread-safe?
> 
> BTW it looks like cdo does multithreaded IO on .nc4 files. Otherwise
> i can not explain the cdo performance. For that they probably
> use HDF5 library directly...
> 
> BR,
> Rostislav
> 
> 
> 
> On Thu, Aug 09, 2018 at 12:34:13PM -0600, Unidata netCDF Support wrote:
> >If you have the hdf5 source directory and do  the command
> >    ./configure --help
> >you will see this:
> >>  --enable-threadsafe     Enable thread-safe capability. Not compatible with
> >>                          the high-level library, Fortran, or C++ wrappers.
> > >                         [default=no]
> >Since the netcdf-c library makes extensive use of the high_level library 
> >(libhdf5_hl)
> >this makes it technically unusable by us.
> >Making the netcdf-c library thread safe is on our to-do list,
> >but it cannot be done until some other parallelism issues are resolved.
> >In any case, our first attempt will be to serialize access to the netcdf
> >API (which is what HDF5 does, I believe). So even if the client program
> >is multi-threaded, each call to netcdf will end up being serialized.
> >At some point, we hope to extend this to allow a limited degree of
> >multi-threaded operation, but this not going to happen soon because it
> >requires significant code analysis.
> >My point about using multiple threads inside a compressor was perhaps
> >unclear. The idea is that when HDF5 calls the (de-)compressor to
> >(de-)compress a chunk, it does not care how the compressor operates.
> >If internally, the compressor were to spin up multiple threads to do
> >the compression and then kill off those threads when the compression
> >of the chunk completed, HDF5 would not care. However, I think the cose
> >of using multiple threads inside the compression code would only be
> >justified if the chunk size were large.
> >
> >
> >>
> >> Thank you! I did not quite get your point...
> >> The whole idea is to use compression provided by HFD5 via libnetdef.
> >> Or is there any way to get a compression part of read/write out of 
> >> libnetcdf?
> >>
> >> Indeed, HDF5 can be compiled thread-safe.
> >> https://support.hdfgroup.org/HDF5/doc/TechNotes/ThreadSafeLibrary.html
> >> I wonder if at least NF90_GET_VAR can be called from concurrent threads.
> >>
> >> BR,
> >> Rostilsav
> >>
> >>
> >>
> >> On Wed, Aug 08, 2018 at 11:59:14AM -0600, Unidata netCDF Support wrote:
> >> >Unfortunately, the netcdf library is not thread safe
> >> >because it depends on the HDF5 library, which is not thread safe.
> >> >It is unknown if accessing different variables in different
> >> >threads will work correctly.
> >> >The only possibility I can see is as follows.
> >> >1. create a thread pool
> >> >2. use large chunk sizes for your variables
> >> >3. inside your compression code, you can use threads from the pool
> >> >to compress pieces of each chunk that is passed to you.
> >> >In other words only use threading inside of your own compression code.
> >> >
> >> >
> >> >>
> >> >> We have recently started uding compression features of NetCDF4.
> >> >> It gives substantial gain in space at expence of CPU resources
> >> >> on compression/decompression. For few other applications we
> >> >> use multithreaded compression/decompression to cope with compression
> >> >> costs. In some cases it turns out to be also faster than dealing with
> >> >> uncompressed data due to disk io.
> >> >>
> >> >> Are NF90_PUT_VAR and NF90_GET_VAR functions thread-safe if called for
> >> >> different variables?
> >> >>
> >> >> Or is there any way to distribute the compression/decompression overhead
> >> >> in Netcdf4 over few threads?
> >> >>
> >> >> Thank you!
> >> >>
> >> >> BR,
> >> >> Rostislav
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Rostislav Kouznetsov
> >> >>
> >> >> **************************************
> >> >> Atmospheric composition unit
> >> >> Finnish Meteorological Institute
> >> >> Erik Palmenin aukio 1, 00560
> >> >> (PL-503 FI-00101)
> >> >> Helsinki Finland
> >> >> Phone: +358 29 539 4630
> >> >> **************************************
> >> >>
> >> >>
> >> >
> >> >=Dennis Heimbigner
> >> >  Unidata
> >> >
> >> >
> >> >Ticket Details
> >> >===================
> >> >Ticket ID: LTC-671596
> >> >Department: Support netCDF
> >> >Priority: Normal
> >> >Status: Open
> >> >===================
> >> >NOTE: All email exchanges with Unidata User Support are recorded in the 
> >> >Unidata inquiry tracking system and then made publicly available through 
> >> >the web.  If you do not want to have your interactions made available in 
> >> >this way, you must let us know in each email you send to us.
> >> >
> >> >
> >>
> >> --
> >> Rostislav Kouznetsov
> >>
> >> **************************************
> >> Atmospheric composition unit
> >> Finnish Meteorological Institute
> >> Erik Palmenin aukio 1, 00560
> >> (PL-503 FI-00101)
> >> Helsinki Finland
> >> Phone: +358 29 539 4630
> >> **************************************
> >>
> >>
> >
> >=Dennis Heimbigner
> >  Unidata
> >
> >
> >Ticket Details
> >===================
> >Ticket ID: LTC-671596
> >Department: Support netCDF
> >Priority: Normal
> >Status: Open
> >===================
> >NOTE: All email exchanges with Unidata User Support are recorded in the 
> >Unidata inquiry tracking system and then made publicly available through the 
> >web.  If you do not want to have your interactions made available in this 
> >way, you must let us know in each email you send to us.
> >
> >
> 
> --
> Rostislav Kouznetsov
> 
> **************************************
> Atmospheric composition unit
> Finnish Meteorological Institute
> Erik Palmenin aukio 1, 00560
> (PL-503 FI-00101)
> Helsinki Finland
> Phone: +358 29 539 4630
> **************************************
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: LTC-671596
Department: Support netCDF
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.