Re: netcdf ncsync() bug

Hello,

A while ago I reported a problem I was having with ncsync() in 2.3.2.  I've
attached a reply from Russ which summarizes the problem.  In some recent
testing with netCDF 2.4-beta the problem still appears, and a quick check
of the code looks like the only change was the addition of some comments.
Was this discovered to not be a bug? or not worth fixing?  I was hoping the
fix would appear in 2.4, but perhaps the email just got lost in the volumes
which you must receive.  If nothing else, I want to cast a vote for the
continued consideration of real-time uses of netcdf.

Thanks,
Gary

----
Gary Granger                              address@hidden
Research Data Program                     http://www.atd.ucar.edu/rdp/gjg.html
National Center for Atmospheric Research  Ph: 303-497-8816  FAX: 303-497-8770

----
>From address@hidden  Mon Sep 26 10:34:14 1994
Received: from unidata.ucar.edu by atd.atd.ucar.EDU (8.6.4/ NCAR Mail Server 
04/10/90)
        id KAA28857; Mon, 26 Sep 1994 10:34:13 -0600
Received: from buddy (buddy.unidata.ucar.edu) by unidata.ucar.edu with SMTP id 
AA10058
  (5.65c/IDA-1.4.4 for <address@hidden>); Mon, 26 Sep 1994 10:34:04 -0600
Message-Id: <address@hidden>
To: address@hidden (Gary Granger)
Cc: address@hidden, address@hidden
Subject: Re: netcdf ncsync() and Zeb 
In-Reply-To: Your message of "Sun, 25 Sep 1994 19:28:40 MDT."
             <address@hidden> 
Organization: UCAR Unidata Program
Date: Mon, 26 Sep 1994 10:34:03 -0600
From: Russ Rew <address@hidden>
Status: RO

>Organization: NCAR/ATD
>Keywords: 199409250227.AA08412 ncsync bug

Hi Gary,

> I've been trying to track down a file sync'ing problem in the Zeb software
> over here in ATD.  The scenario is thus: a producer process periodically
> adds records to a netcdf file, and then after each addition copies the file
> into a consumer's file.  The data server detects the st_mtime change to the
> consumer file and tells the consumer to sync with the file.  But the sync
> does not appear to be working.  ncsync() looks like it syncs correctly when
> the file is RDWR and changes are being written to the file.  However, in
> this scenario the consumer's file is also opened RDWR, but to sync it needs
> to be able to read the changed file from the disk.  The behavior I was
> hoping for was that if there were no changes to write to the disk, ncsync()
> would do a read sync.  When the consumer keeps the file open read-only, it
> does read the disk, but then it doesn't update the handle with the xdrs
> changes.  A call to ncdiminq() after ncsync() returns the old record count
> instead of what's now in the file.  Is ncsync() intended to do this kind of
> sync'ing, or was it intended only to write sync's in RDWR mode?

The documentation says ncsync() is intended to work for reading processes,
to update changes made by another writing process.  Our nctest program
doesn't test this, so it's possible you have discovered a bug.

> I've included a copy of ncsync() from version 2.3.2 below.  Following that
> is an attempted 'fix' which works for the test scenario above.  The handle
> structures are freed before NCxdrfile_sync() and then recreated based on
> the newly read xdrs buffer.  This has more overhead for write syncs than
> the original version, since a read sync is done every time, even when a
> write sync was done first.  I could not find a way to test whether anything
> was written by NCxdrfile_sync(), at least not without changing the
> interface or violating the encapsulation of the low-level IO.  With such a
> test, ncsync() could avoid re-creating the handle and just return once the
> write sync is done.
> 
> I am wondering how far off base I am here, and if perhaps anyone sees a
> better solution to the problem.  Thanks.

I will forward this note with your patch appended to Glenn Davis, who wrote
the library and I hope still remembers the details of the ncsync()
implementation.  It appears as if this should go on our list for fixing in
the next release.  In the meantime, it sounds like you have a reasonable
workaround.

Thanks for reporting this.

--
Russ Rew                                                UCAR Unidata Program
address@hidden                                          P.O. Box 3000
http://www.unidata.ucar.edu/                          Boulder, CO 80307-3000


> Gary Granger
> NCAR/ATD/RDP
> (303)497-8816
> 
> - -------------------------------------------------------------------------
> /* Original: $Id: file.c,v 1.65 1993/04/28 21:22:07 davis Exp $ */
> 
> ncsync(id)
> int id ;
> {
>       NC *handle ;
> 
>       cdf_routine_name = "ncsync" ;
> 
>       handle = NC_check_id(id) ; 
>       if(handle == NULL)
>               return(-1) ;
> 
>       if( handle->flags & NC_INDEF )
>       {
>               NCadvise(NC_EINDEFINE, "Unfinished definition") ;
>               return(-1) ;
>       }
> 
>       if(handle->flags & NC_RDWR)
>       {
>               handle->xdrs->x_op = XDR_ENCODE ;
>               if(handle->flags & NC_HDIRTY)
>               {
>                       if(!xdr_cdf(handle->xdrs, &handle) )
>                               return(-1) ;
>                       handle->flags &= ~(NC_NDIRTY | NC_HDIRTY) ;
>               }
>               else if(handle->flags & NC_NDIRTY)
>               {
>                       if(!xdr_numrecs(handle->xdrs, handle) )
>                               return(-1) ;
>                       handle->flags &= ~(NC_NDIRTY) ;
>               }
>       } else /* read only */
>       {
>               /* assert(handle->xdrs->x_op == XDR_DECODE) ; */
>               /* free the stuff in handle that xdr_cdf allocates */
>               handle->xdrs->x_op = XDR_FREE ;
>               (void) xdr_cdf(handle->xdrs, &handle) ;
>               handle->xdrs->x_op = XDR_DECODE ;
> 
>               if(!xdr_cdf(handle->xdrs, &handle) )
>               {
>                       nc_serror("xdr_cdf") ;
>                       NC_free_cdf(handle) ; /* ?? what should we do now? */
>                       return(-1) ;
>               }
>               if( NC_computeshapes(handle) == -1)
>                       return(-1) ;
>       }
> 
>       (void) NCxdrfile_sync(handle->xdrs) ;
> 
>       return(0) ;
> }
> 
> - ----------------------------------------------------------------------------
> /* Changed: $Id: file.c,v 1.65 1993/04/28 21:22:07 davis Exp granger $ */
> 
> ncsync(id)
> int id ;
> {
>       NC *handle ;
> 
>       cdf_routine_name = "ncsync" ;
> 
>       handle = NC_check_id(id) ; 
>       if(handle == NULL)
>               return(-1) ;
> 
>       if( handle->flags & NC_INDEF )
>       {
>               NCadvise(NC_EINDEFINE, "Unfinished definition") ;
>               return(-1) ;
>       }
> 
>       if(handle->flags & NC_RDWR)
>       {
>               /* If we have handle changes to sync, encode them here */
>               handle->xdrs->x_op = XDR_ENCODE ;
>               if(handle->flags & NC_HDIRTY)
>               {
>                       if(!xdr_cdf(handle->xdrs, &handle) )
>                               return(-1) ;
>                       handle->flags &= ~(NC_NDIRTY | NC_HDIRTY) ;
>               }
>               else if(handle->flags & NC_NDIRTY)
>               {
>                       if(!xdr_numrecs(handle->xdrs, handle) )
>                               return(-1) ;
>                       handle->flags &= ~(NC_NDIRTY) ;
>               }
>       }
> 
>       /* free the stuff in handle that xdr_cdf allocates */
>       handle->xdrs->x_op = XDR_FREE ;
>       (void) xdr_cdf(handle->xdrs, &handle) ;
> 
>       /* write our changes to disk or read someone else's */
>       (void) NCxdrfile_sync(handle->xdrs) ;
> 
>       /* create the handle with any changes which may have been read */
>       handle->xdrs->x_op = XDR_DECODE ;
>       if(!xdr_cdf(handle->xdrs, &handle) )
>       {
>               nc_serror("xdr_cdf") ;
>               NC_free_cdf(handle) ; /* ?? what should we do now? */
>               return(-1) ;
>       }
>       if( NC_computeshapes(handle) == -1)
>               return(-1) ;
> 
>       /* at this point the handle and buffer should be in sync with the
>        * file on the disk, whether we wrote the file or someone else did */
>       return(0) ;
> }
> 
> 
> ------- End of Forwarded Message
>