[netCDF #MJW-484586]: a few more questions on fletcher 32 checksum

Hi Randy,

> I could not find a lot at the NetCDF website on how to use [the
> Fletcher32 checksum functions]
> 
> I did run across setting some constants ...
> 
> #define
> NC_NOCHECKSUM<http://www.unidata.ucar.edu/software/netcdf/docs_snapshot/html/netcdf_8h.html#ac42e062a3be9d>
> a20b62239f383760b0c> 0
> In HDF5 files you can set check-summing for each
> variable. <http://www.unidata.ucar.edu/software/netcdf/docs_snaps>
> hot/html/netcdf_8h.html#ac42e062a3be9da20b62239> f383760b0c>
> 
> #define
> NC_FLETCHER32<http://www.unidata.ucar.edu/software/netcdf/docs_snapshot/html/netcdf_8h.html#a8598aa723c600>
> cc97e76a5f11ea8abd4> 1
> In HDF5 files you can set check-summing for each
> variable. <http://www.unidata.ucar.edu/software/netcdf/docs_snaps>
> hot/html/netcdf_8h.html#a8598aa723c600cc97e76a5> f11ea8abd4>
> 
> And I did come across a couple of C functions ...
> 
> EXTERNL int nc_def_var_fletcher32 (int ncid, int varid, int fletcher32)
> EXTERNL int
> nc_inq_var_fletcher32<http://www.unidata.ucar.edu/software/netcdf/docs_snapshot/html/group__variables.>
> html#ga2e4999ccf1df31d8aebcc0989913bccc> (int ncid, int varid, int 
> *fletcher32p)
> 
> Learn the checksum settings for a variable.

Those are the netCDF functions you would use.  You shouldn't need to
call any HDF5 functions.  You call nc_def_var_fletcher32 for each
variable that you want checksummed.  You ordinarily wouldn't need to
call nc_inq_var_fletcher32, because once a variable is defined as
checksummed, accessing it always checks the checksums, and writing to
it always updates the checksums, chunk-wise (that is each chunk has a
checksum, and writing to a chunk will cause that chunks checksum to be
updated).  Whether a variable is checksummed or not should be
transparent to readers, except that they will get an error if a chunk
is ever read with an incorrect checksum.

> So it looks like you can cause HDF5 libraries to calculate the
> checksums on writes, and append the checksums to the "chunk" you are writing.
> 
> Is there a means to enable/disable using the checksum on reads ?

No, as far as I understand, if you set checksumming for a variable, it
will always be checked on reads and updated on writes.

> Note that I found the function calls in the HDF5 library at
> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Proper> ty-SetEdcCheck ...
> 
> Sometimes, it is desirable to skip checksumming during data reading
> in order to get good performance.? There is a 
> function H5Pset_edc_check to offer this option:
> herr_t H5Pset_edc_check(hid_t dxpl, H5P_EDC_t check)

That's never called in the netCDF-4 library implementation, so it's
not available for use.  If there is a real need for it, it might be
possible to add in the future.  Note that the checksum is only checked
when a chunk is read into the chunk cache, so if a reading process
reads data from the same chunk while it's still in the cache, there's
no performance hit on the read.

> Do I have any misunderstandings ?

It sounds like your understanding is right.

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: MJW-484586
Department: Support netCDF
Priority: Normal
Status: Closed