Wei and Barbara, I've attached a simplified demonstration for the HDF5 issue, wrrds.c, that writes a 75 MB file of floats, then reads them back, one value at a time. It finishes fine when linked to HDF5-1.8.12, but gets an HDF5 error when linked with HDF5-1.8.13. In either case, it takes about 5 minutes to run on my OSX laptop: $ uname -a Darwin milton.local 13.4.0 Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64 $ NCDIR=~/installs/nc432-1812 $ cc -o wrrds-1812 wrrds.c -I$NCDIR/include -L$NCDIR/lib -lnetcdf $ ./wrrds-1812 $ ./wrrds-1812 /tmp/wrrds.nc created /tmp/wrrds.nc written /tmp/wrrds.nc: reading values one-at-a-time successfully read file /tmp/wrrds.nc $ NCDIR=~/installs/nc432-1813 $ cc -o wrrds-1813 wrrds.c -I$NCDIR/include -L$NCDIR/lib -lnetcdf $ ./wrrds-1813 $ ./wrrds-1813 /tmp/wrrds.nc created /tmp/wrrds.nc written /tmp/wrrds.nc: reading values one-at-a-time error: NetCDF: HDF error reading var[0][16][130][1161] The fact that it fails the same way on yellowstone and a Mac laptop means its likely platform independent. --Russ > Here is NCAR's Yellowstone system info: > > uname -a > Linux geyser05 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 > x86_64 x86_64 x86_64 GNU/Linux > > Barbara, and Russ, > > Attached is a C code which I can reproduce this problem on NCAR's > Yellowstone > by read the NCL generated NetCDF file. > > Thank you all for check on this issue. > > Wei > > > > Wei > > > ================================================ > 1850 Table Mesa Dr. > Boulder, CO 80307 > Phone: 303-497-8924 > > On Fri, Sep 26, 2014 at 11:06 AM, <address@hidden> wrote: > > > Hi Wei, > > > > I probably should have figured that out! > > Joe Lee will look at this and we'll see what the next step is. > > I'll bring it up in the HDF5 meeting on Monday, as well. > > > > -Barbara > > > > > > On Fri, 26 Sep 2014, Wei Huang wrote: > > > > Barbara, > >> > >> I am a NCL developer in the NCL team. > >> As I mentioned in my last email, it seems that the problem > >> shows up when we tried to read netCDF4classic file quite some > >> million times. And the same script which worked with HDF5-1.8.12. > >> This read end-up only read each point at time with the way > >> the user coded in NCL. We know it is not an efficient way to read > >> data, but our code should not fail, and so HDF5 should not fail either. > >> > >> If you need, I can FTP the data file to you, and you can try to read this > >> data point by point with HDF5, > >> which may reproduce the problem. > >> If Russ wants to investigate more, I can FTP the data as well. > >> > >> Thanks for looking at this issue. > >> > >> Wei > >> > >> ================================================ > >> 1850 Table Mesa Dr. > >> Boulder, CO 80307 > >> Phone: 303-497-8924 > >> > >> On Fri, Sep 26, 2014 at 7:57 AM, <address@hidden> wrote: > >> > >> Hi Wei, > >>> > >>> I'm not sure we'll be able to determine the issue with your NCL > >>> scripts. However, I did find someone here who has used NCL before. > >>> I will forward your scripts to him, in case he can help. > >>> > >>> He suggested you contact the ncl-talk mailing list, as they may be > >>> familiar with this issue. It looks like you have to subscribe to > >>> ncl-talk. See: > >>> > >>> https://www.ncl.ucar.edu/Support/email_lists.shtml > >>> > >>> In the mean-time, we will see if we can determine anything from the > >>> scripts. > >>> > >>> -Barbara > >>> address@hidden > >>> > >>> > >>> > >>> I have been able to reproduce this issue with a simple NCL scripts. > >>> > >>>> Attached are two NCL scripts, where wrt.ncl created a data file. > >>>> Then read.ncl read a sub-set of the data in a loop. > >>>> > >>>> The read script can demonstrate that HDF5 error start to appear > >>>> loop index 414. (where the first 414 read just normal). > >>>> > >>>> I understand that it seems not fair to send you NCL scripts, > >>>> but that is how we got the problem, and little bit more frustrated > >>>> is that it seems the same code worked fine on my Mac, > >>>> but failed on NCAR's yellowstone. > >>>> > >>>> I hope I can give you more info, but that is what I got so far. > >>>> > >>>> Thanks, > >>>> > >>>> Wei > >>>> > >>>> > >>>> ================================================ > >>>> 1850 Table Mesa Dr. > >>>> Boulder, CO 80307 > >>>> Phone: 303-497-8924 > >>>> > >>>> On Thu, Sep 25, 2014 at 8:16 AM, <address@hidden> wrote: > >>>> > >>>> > >>>> Hi Wei, > >>>>> > >>>>> Thanks for the update. > >>>>> I'll wait to hear from you. > >>>>> > >>>>> -Barbara > >>>>> address@hidden > >>>>> > >>>>> > >>>>> On Wed, 24 Sep 2014, Wei Huang wrote: > >>>>> > >>>>> Barbara, > >>>>> > >>>>> > >>>>>> I am able to repeat this issue with a very short NCL script. > >>>>>> We'll do more debug and will let you know if we still have problem. > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Wei > >>>>>> > >>>>>> ================================================ > >>>>>> 1850 Table Mesa Dr. > >>>>>> Boulder, CO 80307 > >>>>>> Phone: 303-497-8924 > >>>>>> > >>>>>> On Wed, Sep 24, 2014 at 3:01 PM, Wei Huang <address@hidden> wrote: > >>>>>> > >>>>>> Barbara, > >>>>>> > >>>>>> > >>>>>>> The problem came from a user running NCL. > >>>>>>> We got the data (two netcdf files, total of 35G), and could reproduce > >>>>>>> the > >>>>>>> issue when we compile NCL with HDF5-1.8.13, and NetCDF-4.3.2. > >>>>>>> But there is no problem when we use HDF5-1.8.12, and NetCDF-4.3.2. > >>>>>>> Then we try to reproduce the problem without read those two large > >>>>>>> files, > >>>>>>> but it seems NCL works with HDF5-1.8.13 and NetCDF-4.3.2 with > >>>>>>> simple code. > >>>>>>> > >>>>>>> This kind of put us in a place of no where, which we do not know > >>>>>>> where > >>>>>>> and how to debug this problem. That is why I send the question to > >>>>>>> netcdf-support. > >>>>>>> I do not know if want to send you the data and NCL script to debug > >>>>>>> this issue. If you feel the data is too much, and the problem is from > >>>>>>> NCL, > >>>>>>> could you please check with your developers to see if they have any > >>>>>>> idea? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> > >>>>>>> Wei > >>>>>>> > >>>>>>> ================================================ > >>>>>>> 1850 Table Mesa Dr. > >>>>>>> Boulder, CO 80307 > >>>>>>> Phone: 303-497-8924 > >>>>>>> > >>>>>>> On Wed, Sep 24, 2014 at 12:39 PM, <address@hidden> wrote: > >>>>>>> > >>>>>>> Hi Wei, > >>>>>>> > >>>>>>> > >>>>>>>> Yes, please do send us a way to reproduce the issue. (I don't know > >>>>>>>> what > >>>>>>>> the problem is off-hand.) > >>>>>>>> > >>>>>>>> Thanks! > >>>>>>>> -Barbara > >>>>>>>> > >>>>>>>> ====================== > >>>>>>>> Barbara Jones > >>>>>>>> The HDF Group Helpdesk > >>>>>>>> address@hidden > >>>>>>>> ====================== > >>>>>>>> > >>>>>>>> > >>>>>>>> Hi Wei, > >>>>>>>> > >>>>>>>> > >>>>>>>> I will forward this question to address@hidden, because the > >>>>>>>>> symptoms > >>>>>>>>> are > >>>>>>>>> something that doesn't work with HDF5-1.8.13 but did work with > >>>>>>>>> HDF5-1.8.12. > >>>>>>>>> > >>>>>>>>> They may ask you for a way to duplicate the problem, unless they > >>>>>>>>> can > >>>>>>>>> tell what > >>>>>>>>> the problem is from the HDF5 error messages you have supplied. > >>>>>>>>> > >>>>>>>>> --Russ > >>>>>>>>> > >>>>>>>>> We try to create a netcdf file with write mode: netcdf4classic. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> It works with a test program, but failed with a real case, with > >>>>>>>>>> the > >>>>>>>>>> error > >>>>>>>>>> message below. > >>>>>>>>>> > >>>>>>>>>> We use HDF5-1.8.13 with netcdf4.3.2. > >>>>>>>>>> > >>>>>>>>>> If compiled with HDF5-1.8.12, and netcdf4.3.2, the real case > >>>>>>>>>> worked. > >>>>>>>>>> > >>>>>>>>>> We have checked the real case code few times, but did not find > >>>>>>>>>> anything > >>>>>>>>>> wrong. > >>>>>>>>>> > >>>>>>>>>> Thanks for any help on this issue. > >>>>>>>>>> > >>>>>>>>>> Wei > >>>>>>>>>> > >>>>>>>>>> -------------------------------- > >>>>>>>>>> > >>>>>>>>>> HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0: > >>>>>>>>>> #000: H5P.c line 303 in H5Pcreate(): unable to create property > >>>>>>>>>> list > >>>>>>>>>> major: Property lists > >>>>>>>>>> minor: Unable to create file > >>>>>>>>>> #001: H5Pint.c line 1747 in H5P_create_id(): unable to atomize > >>>>>>>>>> property > >>>>>>>>>> list > >>>>>>>>>> major: Property lists > >>>>>>>>>> minor: Unable to register new atom > >>>>>>>>>> #002: H5I.c line 895 in H5I_register(): can't insert ID node into > >>>>>>>>>> skip > >>>>>>>>>> list > >>>>>>>>>> major: Object atom > >>>>>>>>>> minor: Unable to insert object > >>>>>>>>>> #003: H5SL.c line 995 in H5SL_insert(): can't create new skip list > >>>>>>>>>> node > >>>>>>>>>> major: Skip Lists > >>>>>>>>>> minor: Unable to insert object > >>>>>>>>>> #004: H5SL.c line 687 in H5SL_insert_common(): can't insert > >>>>>>>>>> duplicate > >>>>>>>>>> key > >>>>>>>>>> major: Skip Lists > >>>>>>>>>> minor: Unable to insert object > >>>>>>>>>> > >>>>>>>>>> ================================================ > >>>>>>>>>> 1850 Table Mesa Dr. > >>>>>>>>>> Boulder, CO 80307 > >>>>>>>>>> Phone: 303-497-8924 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Russ Rew UCAR Unidata > >>>>>>>>>> Program > >>>>>>>>>> > >>>>>>>>>> address@hidden > >>>>>>>>> http://www.unidata.ucar.edu > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Ticket Details > >>>>>>>>> =================== > >>>>>>>>> Ticket ID: MRA-585540 > >>>>>>>>> Department: Support netCDF > >>>>>>>>> Priority: Normal > >>>>>>>>> Status: Closed > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>> > >>>>> > >>>>> > >>>>> > >>>> -- > >>> > >>> > >> > > -- > > > > Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: MRA-585540 Department: Support netCDF Priority: Normal Status: Closed
Attachment:
wrrds.c
Description: Binary data