Re: 19990711: extra header space

>To: address@hidden
>From: Michael Nolta <address@hidden>
>Subject: extra header space
>Organization: Princeton
>Keywords: 199907110517.XAA12464 netCDF

Hi Michael,

You wrote:

> I'm having a little trouble creating extra space in the header using the
> '__' functions. The little test program I wrote isn't behaving as I
> expected. It creates a file with an integer array, fills the array, and
> then reenters define mode to add a "title" global attribute. Looking at
> the output file, it seems that adding the "title" attribute causes the
> "x" variable data to be rewritten, overwriting the extra header space in
> the process. What am I doing wrong?
 ...
> Perhaps a few crude diagrams will help.
> If I comment out the section adding the "title" attribute, I get a file
> that looks like this (not to scale ;-):
> 
>       **000000000000000000abcdefghijklmnopqrstuvwxyz
> 
> where * is the header information, 0 is blank, and abc... is the data.
> Writing "title" gives me a file that looks like this:
> 
>       ********abcdefghijklmnopqrstuvwxyzopqrstuvwxyz
> 
> What I expected to get was:
> 
>       ********000000000000abcdefghijklmnopqrstuvwxyz
> 
> i.e., "title" is written into the blank space and the data is left
> untouched. What appears to happen is that the header expands and then the
> data is written a second time immediately preceding the (non-blank) header
> info.
> 
> Basically, what I'm trying to do is add some room for expansion to the
> header, so I can add to it later and avoid copying the data section.

There's now a fix for this bug in our new netCDF version 3.5 beta
release, and I've tested the fix extensively enough to be fairly
confident that it introduces no new bugs.  In case you have the time
to test it, the new 3.5 beta release is available from

    ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-3.5-beta1.tar.Z

Here's the addition to the RELEASE_NOTES describing the fix:

    Fixed a number of bugs related to attempts to support shrinking
    the header in netCDF files when attributes are rewritten or
    deleted.  Also fixed the problem that nc__enddef() did not work as
    intended in reserving extra space in the file header, since the
    extra space would be compacted again on calling nc_close().  The
    header of a netCDF file should not shrink.  Any extra space in the
    header resulting from calling nc__enddef() (or nf__enddef()) to
    reserve space, from deletion of attributes, or from rewriting
    attributes with fewer values will persist after the file is
    closed.  This extra header space will be available for later
    efficient addition of new dimensions, attributes, or variables.
    (Note: renaming something to a shorter name, however, does not
    create any usable extra space in the file header.)

I've also appended an equivalent patch for the current netcdf 3.4 version of
libsrc/nc.c, in case that's more convenient, but I haven't tested this
extensively with netCDF 3.4.

Thanks again for the bug report.

--Russ
_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu

diff -c nc.c~ nc.c
*** nc.c~       Fri Jul 23 15:15:27 1999
--- nc.c        Fri Jul 23 15:15:36 1999
***************
*** 205,215 ****
        if(ncp->vars.nelems == 0) 
                return;
  
!       index = (off_t) ncp->xsz;
!       ncp->begin_var = D_RNDUP(index, v_align);
!       if(ncp->begin_var - index < h_minfree)
!       {
!               ncp->begin_var = D_RNDUP(index + (off_t)h_minfree, v_align);
        }
        index = ncp->begin_var;
  
--- 205,221 ----
        if(ncp->vars.nelems == 0) 
                return;
  
!       /* only (re)calculate begin_var if there is not sufficient space in 
header
!          or start of non-record variables is not aligned as requested by 
valign */
!       if (ncp->begin_var < ncp->xsz + h_minfree ||
!           ncp->begin_var != D_RNDUP(ncp->begin_var, v_align) ) 
!       {
!         index = (off_t) ncp->xsz;
!         ncp->begin_var = D_RNDUP(index, v_align);
!         if(ncp->begin_var < index + h_minfree)
!         {
!           ncp->begin_var = D_RNDUP(index + (off_t)h_minfree, v_align);
!         }
        }
        index = ncp->begin_var;
  
***************
*** 229,238 ****
                index += (*vpp)->len;
        }
  
!       ncp->begin_rec = D_RNDUP(index, r_align);
!       if(ncp->begin_rec - index < v_minfree)
!       {
!               ncp->begin_rec = D_RNDUP(index + (off_t)v_minfree, r_align);
        }
        index = ncp->begin_rec;
  
--- 235,251 ----
                index += (*vpp)->len;
        }
  
!       /* only (re)calculate begin_rec if there is not sufficient
!          space at end of non-record variables or if start of record
!          variables is not aligned as requested by r_align */
!       if (ncp->begin_rec < index + v_minfree ||
!           ncp->begin_rec != D_RNDUP(ncp->begin_rec, r_align) )
!       {
!         ncp->begin_rec = D_RNDUP(index, r_align);
!         if(ncp->begin_rec - index < v_minfree)
!           {
!             ncp->begin_rec = D_RNDUP(index + (off_t)v_minfree, r_align);
!           }
        }
        index = ncp->begin_rec;
  
***************
*** 591,712 ****
  
  
  /*
-  * Move the "non record" variables "in". 
-  * Fill as needed.
-  */
- static int
- move_vars_f(NC *gnu, NC *old)
- {
-       int status;
-       int varid;
-       NC_var **gnu_varpp = (NC_var **)gnu->vars.value;
-       NC_var **old_varpp = (NC_var **)old->vars.value;
-       NC_var *gnu_varp;
-       NC_var *old_varp;
-       off_t gnu_off;
-       off_t old_off;
-       
- 
-       for(varid = 0; (size_t) varid < old->vars.nelems; varid++)
-       {
-               gnu_varp = *(gnu_varpp + varid);
-               if(IS_RECVAR(gnu_varp))
-               {
-                       /* skip record variables on this pass */
-                       continue;
-               }
-               /* else */
- 
-               old_varp = *(old_varpp + varid);
-               gnu_off = gnu_varp->begin;
-               old_off = old_varp->begin;
-       
-               if(gnu_off == old_off)
-                       continue;       /* nothing to do */
- 
-               assert(gnu_off < old_off);
-       
-               status = gnu->nciop->move(gnu->nciop, gnu_off, old_off,
-                        old_varp->len, 0);
- 
-               if(status != NC_NOERR)
-                       return status;
-       }
- 
-       return NC_NOERR;
- }
- 
- 
- /*
-  * Move the records "in". 
-  * Fill as needed.
-  */
- static int
- move_recs_f(NC *gnu, NC *old)
- {
-       int status;
-       size_t recno;
-       int varid;
-       NC_var **gnu_varpp = (NC_var **)gnu->vars.value;
-       NC_var **old_varpp = (NC_var **)old->vars.value;
-       NC_var *gnu_varp;
-       NC_var *old_varp;
-       off_t gnu_off;
-       off_t old_off;
-       
- 
-       for(recno = 0; recno < old->numrecs; recno++)
-       {
-       for(varid = 0; (size_t) varid < gnu->vars.nelems; varid++)
-       {
-               gnu_varp = *(gnu_varpp + varid);
-               if(!IS_RECVAR(gnu_varp))
-               {
-                       /* skip non-record variables on this pass */
-                       continue;
-               }
-               /* else */
- 
-               if((size_t) varid < old->vars.nelems)
-               {
-                       /* a pre-existing variable */
-                       old_varp = *(old_varpp + varid);
-                       gnu_off = gnu_varp->begin +
-                                (off_t)(gnu->recsize * recno);
-                       old_off = old_varp->begin +
-                                (off_t)(old->recsize * recno);
-               
-                       if(gnu_off == old_off)
-                               continue;       /* nothing to do */
-       
-                       assert(gnu_off < old_off);
- 
-                       status = gnu->nciop->move(gnu->nciop, gnu_off, old_off,
-                                old_varp->len, 0);
-       
-                       if(status != NC_NOERR)
-                               return status;
-                       continue;
-               }
- 
-               
-               /* else, a new variable */
-               if(NC_dofill(gnu))
-               {
-                       status = fill_NC_var(gnu, gnu_varp, recno);
-                       if(status != NC_NOERR)
-                               return status;
-               }
-       }
-       }
- 
-       gnu->numrecs = old->numrecs;
- 
-       return NC_NOERR;
- }
- 
- 
- /*
   *  End define mode.
   *  Common code for ncendef, ncclose(endef)
   */
--- 604,609 ----
***************
*** 727,732 ****
--- 624,631 ----
                /* a plain redef, not a create */
                assert(!NC_IsNew(ncp));
                assert(fIsSet(ncp->flags, NC_INDEF));
+               assert(ncp->begin_rec >= ncp->old->begin_rec);
+               assert(ncp->begin_var >= ncp->old->begin_var);
  
                if(ncp->vars.nelems != 0)
                {
***************
*** 741,793 ****
                                if(status != NC_NOERR)
                                        return status;
                        }
!                       else
!                       {
!                               /* ncp->begin_var <= ncp->old->begin_var */
!                               /* rare */
!                               status = move_vars_f(ncp, ncp->old);
!                               if(status != NC_NOERR)
!                                       return status;
!                       }
!               }
!               else if(ncp->begin_rec < ncp->old->begin_rec)
!               {
!                       if(ncp->begin_var < ncp->old->begin_var)
!                       {
!                               status = move_vars_f(ncp, ncp->old);
!                               if(status != NC_NOERR)
!                                       return status;
!                       }
! #ifndef NDEBUG
!                       else
!                       {
!                               assert("Number of non rec vars shrank (<) ???" 
!                                       == 0);
!                       }
! #endif
!                       status = move_recs_f(ncp, ncp->old);
!                       if(status != NC_NOERR)
!                               return status;
                }
                else
!               {
!                       /* ncp->begin_rec == ncp->old->begin_rec */
!                       /* rare and untested */
!                       if(ncp->begin_var < ncp->old->begin_var)
                        {
!                               status = move_vars_f(ncp, ncp->old);
                                if(status != NC_NOERR)
!                                       return status;
                        }
- #ifndef NDEBUG
-                       else if(ncp->begin_var > ncp->old->begin_var)
-                       {
-                               assert("Number of non rec vars shrank (==) ???" 
-                                       == 0);
-                       }
- #endif
-                       /* ncp->begin_var == ncp->old->begin_var */
-                       /* NOOP */
                }
                }
        }
--- 640,657 ----
                                if(status != NC_NOERR)
                                        return status;
                        }
!                       /* else if (ncp->begin_var == ncp->old->begin_var) { 
NOOP } */
                }
                else
!               {       /* Even if (ncp->begin_rec == ncp->old->begin_rec)
!                          and     (ncp->begin_var == ncp->old->begin_var)
!                          might still have added a new record variable */
!                       if(ncp->recsize > ncp->old->recsize)
                        {
!                               status = move_recs_r(ncp, ncp->old);
                                if(status != NC_NOERR)
!                                     return status;
                        }
                }
                }
        }