[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #IUX-351028]: Setting Lustre file system parameters within NetCDF

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #IUX-351028]: Setting Lustre file system parameters within NetCDF
  • Date: Sun, 07 Aug 2011 18:47:00 -0600
Hi Dan,

> Please consider this a request for enhancement.
> 
> I have an application that generates files with sizes ranging from zero
> bytes to 7+ GB all in the same directory.  When running under a Lustre
> file system, the only option the user has to optimize I/O performance is
> to set the stripe count and stripe size of the working directory.  All
> files copied into or creating within that working directory inherit the
> settings of the parent directory.
> 
> One option is to make more directories.
> 
> Another solution would be to include the ability of setting the stripe
> size/count at file creation within the NetCDF library.

There may be another option: use the Lustre lfs utility with the
setstripe option to set the stripe count and stripe size for files or
directories before creating netCDF files.  Since you can override the
directory settings for specific files within a directory using the lfs
utility, would it work to invoke the "system" library function with a
command line that invoked the "lfs setstripe" command with the desired
options, after an nc_open() or nc_create() call?  If that works, it
could be made into a little utility function with just the filename
and desired parameters.

Adding platform-specific parameters to nc_open() or nc_create() would
create a maintenance problem for us, because it would require
additional testing on a Lustre platform, to which we have no access.

We'll consider your enhancement request, especially if there are more 
requests for this kind of functionality, but if a solution is available that
doesn't require support in our library, that would be preferable for us.

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: IUX-351028
Department: Support netCDF
Priority: High
Status: Closed