[netCDF #KZJ-320086]: Short read are not managed?

Hi Dr. Panitz,

> with great interest I read about the problems with the short read
> described in the ticket #PDZ-683250.  But I have to admit that I did
> not understand everything written there due to "missing background".
> 
> However, I would like to point out that I encountered the same problem
> when running a regional climate model on a CRAY XC40.  The model reads
> data from NetCDF files that are stored on a Lustre Filesystem.
> 
> The problem is identical to that described in the ticket.
> The content of the files is not  corrupted.
> The typical size of a file is about 40 Mbyte.
> 
> I use the following module that had been installed by CRAY staff:
> cray-netcdf/4.3.2
> 
> Unfortunately, I cannot say how NetCDF had been installed (which options
> had been used).
> 
> Of course, I am in contact with a member of the CRAY staff. His assumption
> is that the problems only occurs if  "there is lot of traffic on the
> filesystem".  This assumption seems to be consistent with my experience
> that the problem does not occur with every read but in an unsystematic
> manner.
> 
> Thus, I would be very interested in any response to the ticket mentioned
> above and in any solution.  Furthermore,  I have the questions:
> 
> - does any option exist for NetCDF installation  that could solve the
> "short read" problem?

No, sorry, there is currently no such option. Although we don't currently
test each release on a Cray platform, it's possible that an old configure
option we used to support, "--enable-ffio", might work for platforms
that support FFIO library when --disable-netcdf-4 is also specified , 
but our test platforms don't support FFIO and get compile-time 
errors for ---enable-ffio..

> - would the definition of the "bufrsize" parameter in the call of
> "nf90_open" help? If yes, which size would be appropriate,
> respectively, how can it be determined?

I don't know if it would help, but you might try a smaller size than
you're currently using. I know of no way to determine whether a
smaller size would work or what it should be other than testing with
an example that demonstrates the problem, and that may be 
impractical if it requires generating a high I/O load to see the 
problem.

We would like to have a solution for this problem, but have no
expertise in tuning netCDF on the Cray for Lustre file systems.
I see that Googling "cray lustre netcdf" turns up lots of web sites
and even an HDF5 paper on this subject: "Tuning HDF5 for Lustre 
File Systems" from The HDF Group.

> Thank your very much for any help.

Sorry I couldn't provide any useful help.

--Russ

> Best regards
> 
> Hans-Juergen Panitz
> 
> 
> --
> Karlsruher Institut für Technologie (KIT)
> Institut für Meteorologie und Klimaforschung
> Bereich Troposphäre (IMK-TRO)
> 
> Dr. Hans-Jürgen Panitz
> 
> Hermann-von-Helmholtz-Platz 1
> D-76344 Eggenstein-Leopoldshafen
> 
> Phone: xx49-(0)721-608 22802
> Fax  : xx49-(0)721-608 24377
> E-Mail: address@hidden
> 
> www.kit.edu
> www-fzk.imk.uni-karlsruhe.de
> 
> KIT - Universität des Landes Baden-Württemberg und
> nationales Großforschungszentrum in der Helmholtz-Gemeinschaft
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: KZJ-320086
Department: Support netCDF
Priority: High
Status: Closed