[netCDFJava #CRA-892492]: Using Java NetCDF With a Large HDF5 Database - Performance Questions

Creighton -

The Java-based HDF5 reader in netcdf-Java should not be
reading the whole file in order to get the header info.
So we need some more information in order to diagnose your problem.
1. Can you send us the header information using NCdumpW?
2. If you have access to the C-library version of netcdf-c,
   can you do an ncdump -h command on your file and see if that
   also takes a long time?

WRT to your specific questions:
* Is there a way to speed up the process of opening a large (17 GB) HDF5 file?
The questions above will help us diagnose this.

* Or is there a way to provide some sort of index file to speed up
reading the Variable information when the file is open?
Internally, HDF5 files use B-trees, which are indices, so that is
already addressed by the HDF5 file format.

* Is there a way to read an HDF5 database when split across multiple
files (at a file-system level) with sequentially named segments
(e.g. Db_01.h5, Db_02.h5, etc)?
Short answer: no
Longer answer: The very newest HDF5 C library (1.10) supports
an approximation to this. However, there is no pure Java implementation
of the 1.10 file format.

=Dennis Heimbigner
 Unidata


   
=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: CRA-892492
Department: Support netCDF Java
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.