[netCDF #LKF-153632]: Problem adding vector to variable with unlimited dimension

Hi Sourish,

Interesting problem.  Here's what I've discovered:

The same problem occurs when the C API is used instead of the Fortran
API, so it's not a bug in the Fortran netCDF library.

According to h5dump, the original dataset used big-endian
representations for all numeric quantities in the file.  If a
big-endian 64-bit 1.0 is converted to little-endian by reversing the
bytes, it prints as 3.03865194161742e-319, the value you are seeing
when you expect 1.0.

If in your step (2), I use nccopy instead of Unix cp, that is instead of

  $ cp ~/Downloads/state_trajectory_0015.nc4 test_output_copy.nc4

I use

  $ nccopy ~/Downloads/state_trajectory_0015.nc4 test_output_copy.nc4

then the resulting file (on my x86_64 platform) uses little-endian
representations for numeric quantities, looks identical to the
big-endian version under ncdump, and both your Fortran and the
equivalent C programs work as expected to append the 1.0 vector.

So it looks like there are some circumstances under which appending
64-bit floating-point data to a big-endian array on a little-endian
platform doesn't properly convert from native little-endian to the
destination big-endian representation.

I still don't know how general this problem is and whether it occurs
only in the netCDF-4 to HDF5 layer or below that.  If it was a more
general bug, it seems like it would have been reported already, but
yours is the first example.  So far, I've only seen this bug with the
file that you've made available, so it's also possible there's
something weird about how that file was created.

This is just an interim progress report on the bug.  I'll need to get
access to a big-endian test platform to see if we can discover under
exactly what circumstances it occurs.

--Russ

> I've reproduced the problem, following your instructions, and see the
> same wrong ncdump output on my Fedora platform using gfortran. I will
> translate to an equivalent C program to try determine if the bug is in
> the netCDF Fortran implementation or in the C library it calls (or 
> even in the HDF5 library that calls, though that seems very unlikely).
> 
> Thanks for reporting this!
> 
> --Russ
> 
> > I have a netcdf-4 file with three unlimited dimensions, and three
> > variables. If I attempt to add a vector to one of the variables along
> > its unlimited dimension, the values written to the file are not the
> > values that I'm trying to write. Here are the steps required to
> > reproduce the problem:
> > 
> > (1) Compile the attached program test2.f90. On my system, I used 'ifort
> > -r8 -o test_m1qn3 test2.f90 -lnetcdff'.
> > 
> > (2) Download the file state_trajectory_0015.nc4 from
> > 
> >
> https://www.wetransfer.com/downloads/b8b003345b74d23ebd84604b42274b7120140110191947/e2c03ea7361755f>
> 3a333ff80f99284c020140110191947/96b8cd
> > 
> > and copy it to test_output_copy.nc4.
> > 
> > (3) Run the program, './test_m1qn3'.
> > 
> > (4) Look at the variable 'x_c' in test_output_copy.nc4 using 'ncdump -v
> > x_c test_output_copy.nc4'. I expect the last elements to be all ones,
> > since that's what is written from the Fortran program, but instead the
> > last elements are, e.g.,
> > 
> > x_c =
> > {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3.03865194161742e-319},
> > {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3.03865194161742e-319},
> > {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3.03865194161742e-319},
> > {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3.03865194161742e-319},
> > {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3.03865194161742e-319},
> > ...
> > ...
> > 
> > Any ideas on what is going wrong? My system is
> > 
> > $ uname -a
> > Linux int2.cartesius.surfsara.nl 2.6.32-279.5.2.bl6.Bull.36.x86_64 #1
> > SMP Fri May 24 09:42:11 CEST 2013 x86_64 x86_64 x86_64 GNU/Linux
> > 
> > and the netcdf version is
> > 
> > $ nf-config --version
> > netCDF-Fortran 4.2

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: LKF-153632
Department: Support netCDF
Priority: Normal
Status: Closed