[netCDF #LTC-671596]: Multithreaded IO with libnetcdf4

If you have the hdf5 source directory and do  the command
    ./configure --help
you will see this:
>  --enable-threadsafe     Enable thread-safe capability. Not compatible with
>                          the high-level library, Fortran, or C++ wrappers.
 >                         [default=no]
Since the netcdf-c library makes extensive use of the high_level library 
(libhdf5_hl)
this makes it technically unusable by us.
Making the netcdf-c library thread safe is on our to-do list,
but it cannot be done until some other parallelism issues are resolved.
In any case, our first attempt will be to serialize access to the netcdf
API (which is what HDF5 does, I believe). So even if the client program
is multi-threaded, each call to netcdf will end up being serialized.
At some point, we hope to extend this to allow a limited degree of
multi-threaded operation, but this not going to happen soon because it
requires significant code analysis.
My point about using multiple threads inside a compressor was perhaps
unclear. The idea is that when HDF5 calls the (de-)compressor to
(de-)compress a chunk, it does not care how the compressor operates.
If internally, the compressor were to spin up multiple threads to do
the compression and then kill off those threads when the compression
of the chunk completed, HDF5 would not care. However, I think the cose
of using multiple threads inside the compression code would only be
justified if the chunk size were large.


> 
> Thank you! I did not quite get your point...
> The whole idea is to use compression provided by HFD5 via libnetdef.
> Or is there any way to get a compression part of read/write out of libnetcdf?
> 
> Indeed, HDF5 can be compiled thread-safe.
> https://support.hdfgroup.org/HDF5/doc/TechNotes/ThreadSafeLibrary.html
> I wonder if at least NF90_GET_VAR can be called from concurrent threads.
> 
> BR,
> Rostilsav
> 
> 
> 
> On Wed, Aug 08, 2018 at 11:59:14AM -0600, Unidata netCDF Support wrote:
> >Unfortunately, the netcdf library is not thread safe
> >because it depends on the HDF5 library, which is not thread safe.
> >It is unknown if accessing different variables in different
> >threads will work correctly.
> >The only possibility I can see is as follows.
> >1. create a thread pool
> >2. use large chunk sizes for your variables
> >3. inside your compression code, you can use threads from the pool
> >to compress pieces of each chunk that is passed to you.
> >In other words only use threading inside of your own compression code.
> >
> >
> >>
> >> We have recently started uding compression features of NetCDF4.
> >> It gives substantial gain in space at expence of CPU resources
> >> on compression/decompression. For few other applications we
> >> use multithreaded compression/decompression to cope with compression
> >> costs. In some cases it turns out to be also faster than dealing with
> >> uncompressed data due to disk io.
> >>
> >> Are NF90_PUT_VAR and NF90_GET_VAR functions thread-safe if called for
> >> different variables?
> >>
> >> Or is there any way to distribute the compression/decompression overhead
> >> in Netcdf4 over few threads?
> >>
> >> Thank you!
> >>
> >> BR,
> >> Rostislav
> >>
> >>
> >>
> >> --
> >> Rostislav Kouznetsov
> >>
> >> **************************************
> >> Atmospheric composition unit
> >> Finnish Meteorological Institute
> >> Erik Palmenin aukio 1, 00560
> >> (PL-503 FI-00101)
> >> Helsinki Finland
> >> Phone: +358 29 539 4630
> >> **************************************
> >>
> >>
> >
> >=Dennis Heimbigner
> >  Unidata
> >
> >
> >Ticket Details
> >===================
> >Ticket ID: LTC-671596
> >Department: Support netCDF
> >Priority: Normal
> >Status: Open
> >===================
> >NOTE: All email exchanges with Unidata User Support are recorded in the 
> >Unidata inquiry tracking system and then made publicly available through the 
> >web.  If you do not want to have your interactions made available in this 
> >way, you must let us know in each email you send to us.
> >
> >
> 
> --
> Rostislav Kouznetsov
> 
> **************************************
> Atmospheric composition unit
> Finnish Meteorological Institute
> Erik Palmenin aukio 1, 00560
> (PL-503 FI-00101)
> Helsinki Finland
> Phone: +358 29 539 4630
> **************************************
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: LTC-671596
Department: Support netCDF
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.