Re: [netcdfgroup] netcdf-4 open/close memory leak

Ted Mansell <address@hidden> writes:

> This may not be of any help, but in playing with the h5_read.c file in
> the hdf5 examples directory, I got it to behave similarly to this
> issue (although more like my other issue with opening 100 files and
> reading variables out of them).  Basically, if a dataset is opened
> with H5Dopen2 but not closed with H5Dclose, the memory grows but
> valgrind doesn't notice.  So I wonder if var->hdf_datasetid (dataset
> property list) is not getting closed when a new variable gets read
> after a different one was just read?  Perhaps the open/close leak is
> related if opening the netcdf4 file requires reading metatdata (e.g.,
> in function read_dataset).  It seems like H5Dclose is often just
> getting called in the exit (BAIL) section.

With netCDF, H5Dclose is called on every open dataset when the file is
closed. You can see this all in libsrc4/nc4internal.c. Here's the
function that closes everything in a group:

/* Recursively delete the data for a group (and everything it
 * contains) in our internal metadata store. */
int 
nc4_rec_grp_del(NC_GRP_INFO_T **list, NC_GRP_INFO_T *grp) 
{
   NC_GRP_INFO_T *g, *c;
   NC_VAR_INFO_T *v, *var;
   NC_ATT_INFO_T *a, *att;
   NC_DIM_INFO_T *d, *dim;
   NC_TYPE_INFO_T *type, *t;
   int retval;

   assert(grp);
   LOG((3, "nc4_rec_grp_del: grp->name %s", grp->name));

   /* Recursively call this function for each child, if any, stopping
    * if there is an error. */
   g = grp->children;
   while(g)
   {
      c = g->next;
      if ((retval = nc4_rec_grp_del(&(grp->children), g)))
         return retval;
      g = c;
   }

   /* Delete all the list contents for vars, dims, and atts, in each
    * group. */
   att = grp->att;
   while (att)
   {
      LOG((4, "nc4_rec_grp_del: deleting att %s", att->name));      
      a = att->next;
      if ((retval = nc4_att_list_del(&grp->att, att)))
         return retval;
      att = a;
   }

   /* Delete all vars. */
   var = grp->var;
   while (var)
   {
      LOG((4, "nc4_rec_grp_del: deleting var %s", var->name));      
      /* Close HDF5 dataset associated with this var, unless it's a
       * scale. */
      if (var->hdf_datasetid && !var->dimscale && 
          H5Dclose(var->hdf_datasetid) < 0)
         return NC_EHDFERR;
      v = var->next;
      if ((retval = var_list_del(&grp->var, var)))
         return retval;
      var = v;
   }

   
   /* Delete all dims. */
   dim = grp->dim;
   while (dim)
   {
      LOG((4, "nc4_rec_grp_del: deleting dim %s", dim->name));      
      /* Close HDF5 dataset associated with this dim. */
      if (dim->hdf_dimscaleid && H5Dclose(dim->hdf_dimscaleid) < 0)
         return NC_EHDFERR;
      d = dim->next;
      dim_list_del(&grp->dim, dim);
      dim = d;
   }

   /* Delete all types. */
   type = grp->type; 
   while (type)
   {
      LOG((4, "nc4_rec_grp_del: deleting type %s", type->name));      
      t = type->next;
      if ((retval = type_list_del(&grp->type, type)))
         return retval;
      type = t;
   }

   /* Tell HDF5 we're closing this group. (The only reason I check
    * hdf_grpid here is so that I can tests list code without really
    * have a HDF file open.) */ 
   LOG((4, "nc4_rec_grp_del: closing group %s", grp->name));         
   if (grp->hdf_grpid && H5Gclose(grp->hdf_grpid) < 0) 
      return NC_EHDFERR;

   /* Finally, redirect pointers around this entry in the list, and
    * nc_free its memory. */
   grp_list_del(list, grp);

   return NC_NOERR;
}



>
> Just stabbing in the dark, really.... it is highly possible that I
> have no clue whatsoever.  I did also try calling the garbage_collect
> routine, to no effect.
>

We will get it figured out eventually, I'm sure...

Thanks,

Ed

-- 
Ed Hartnett  -- address@hidden