[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #SID-581519]: posix_fadvise

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #SID-581519]: posix_fadvise
  • Date: Thu, 21 Apr 2016 21:51:54 -0600
Hello Leo,

The file pointer to the open netCDF file is buried in the library; the netCDF C 
api uses an 'ncid' as the way to identify files which writing to/reading from 
netCDF files.  Adding a way to invoke posix_fadvise is an interesting thought, 
but it is not currently possible. 

Sorry I can't provide a more helpful answer,

-Ward

> Hi Ryan,
> 
> would it be possible if I wrote the code in C and used the netCDF C API?
> That would help me as well. I just need a file pointer that can be used
> as input
> parameter for posix_fadvise.
> 
> I believe this feature will become more and more important since
> unnecessary caching of large files can really slow down a system quite a
> bit.
> 
> Leo
> 
> Am 21.04.2016 um 17:51 schrieb Unidata Python Support:
> > Leopold,
> >
> > The python netcdf4 library wraps the netcdf4 C library. The C library 
> > handles all of the calls that open files, so there is no Python file object 
> > available to use with posix_fadvise() in Python.
> >
> > As far as I can tell, I don't think what you want to achieve would be 
> > possible.
> >
> > Ryan
> >
> >> Linux provides the function posix_fadvise to instruct the kernel to
> >> discard the IO cache for an open file after use with the parameter
> >> POSIX_FADV_DONTNEED. This is useful if many large files are read in
> >> (only once). I can use fadvise with files opened with a python "open"
> >> statement but not with the file objects returned by netcdf4.Dataset.
> >>
> >> f=netCDF4.Dataset(fn,'r')
> >> m=fallocate.posix_fadvise(f, 0, siz,fallocate.POSIX_FADV_DONTNEED)
> >>
> >> fails.
> >> Can you instruct me how to use fadvise with netcdf files?
> >
> > Ticket Details
> > ===================
> > Ticket ID: SID-581519
> > Department: Support Python
> > Priority: Normal
> > Status: Closed
> >
> 
> --
> Ao. Prof. Dr. Leopold Haimberger
> Department of Meteorology and Geophysics
> Althanstrasse 14 1090 Wien
> AUSTRIA
> Tel. +43 1 4277 53712
> Mobile: +43 664 260 5378
> Fax +43 1 4277 9537
> www: http://imgw.univie.ac.at/
> 
> 


Ticket Details
===================
Ticket ID: SID-581519
Department: Support netCDF
Priority: Normal
Status: Closed