Re: netcdf write performance

>To: address@hidden
>From: Tim Hoar <address@hidden>
>Subject: Re: 20031120: netcdf write performance
>Organization: UCAR
>Keywords: 200311202353.hAKNrmEH017770

Hi Tim,

> [for Russ' benefit:   I have a (random) time and want to either add to the
> existing unlimited dimension or insert at the appropriate (existing) time.
> Using nc_inq_dimlen is insufficient, since I need to know the corresponding
> time]
> 
> If I read the netcdf coordinate variable for time (the unlimited dimension)
> each time I encounter the above situation, things grow as follows:
> 
> Running case 2000 at Thu Nov 20 16:16:32 MST 2003
>  file_name = Mydata
>  dim1 =           10
>  dim2 =          100
>  nT   =         2000
>  t =            0  time is   1380838617
>  Model attributes written, netCDF file synched ...
>  t =          100  elapsed time is         1303
>  t =          200  elapsed time is         1488
>  t =          300  elapsed time is         1835
>  t =          400  elapsed time is         2182
>  t =          500  elapsed time is         2535
>  t =          600  elapsed time is         2889
>  t =          700  elapsed time is         3264
>  t =          800  elapsed time is         3618
>  t =          900  elapsed time is         3990
>  t =         1000  elapsed time is         4690
>  t =         1100  elapsed time is         4834
>  t =         1200  elapsed time is         5194
>  t =         1300  elapsed time is         5594
>  t =         1400  elapsed time is         5997
>  t =         1500  elapsed time is         6395
>  t =         1600  elapsed time is         6798
>  t =         1700  elapsed time is         7201
>  t =         1800  elapsed time is         7900
>  t =         1900  elapsed time is         8004
>  t =         2000  elapsed time is         8399
> 2.548u 6.880s 0:09.42 100.0%    0+0k 0+0io 253pf+0w
> 
> 
> The elapsed time is the result of the Fortran90 intrinsic "system_clock"
> and has terrific units -- some unknown integer representation of the system
> clock. What I have printed is the time spent since the last print (i.e. if
> the system is linear, the elapsed time printed would be constant).
> 
> However,
> 
> If I use exactly the same methodology but keep a local array of the times
> present in the netcdf file (i.e. I replace the call to read the current
> netcdf time variable with a local time array) -- the code performs linearly:
> 
> Running case 2000 at Thu Nov 20 16:12:35 MST 2003
>  file_name = Mydata
>  dim1 =           10
>  dim2 =          100
>  nT   =         2000
>  t =            0  time is   1378473155
>  Model attributes written, netCDF file synched ...
>  t =          100  elapsed time is          969
>  t =          200  elapsed time is          968
>  t =          300  elapsed time is          967
>  t =          400  elapsed time is          967
>  t =          500  elapsed time is          966
>  t =          600  elapsed time is          970
>  t =          700  elapsed time is          968
>  t =          800  elapsed time is          968
>  t =          900  elapsed time is          967
>  t =         1000  elapsed time is          968
>  t =         1100  elapsed time is          968
>  t =         1200  elapsed time is          971
>  t =         1300  elapsed time is          968
>  t =         1400  elapsed time is          969
>  t =         1500  elapsed time is          972
>  t =         1600  elapsed time is          971
>  t =         1700  elapsed time is          968
>  t =         1800  elapsed time is          968
>  t =         1900  elapsed time is          968
>  t =         2000  elapsed time is          968
> 0.380u 1.562s 0:01.94 100.0%    0+0k 0+0io 240pf+0w
> 
> This includes the fact that I write to the netcdf time array in
> exactly the same manner as the non-linear response case.
> 
> There is no error in the slow case, just a design decision that
> seems perfectly logical yet results in very slow code. In order to
> achieve linear performance in the production code, I will have to
> jump through some hoops.

With your current structure, reading the entire time coordinate
variable involves reading a single value from each record, so if you
have 2000 times, that's 2000 disk reads to get a table of 2000 values.
All the slowdown is due to these reads of an ever-increasing number of
records.  As you have discovered, if you cache the times in a table in
memory and replace the reads of the time variable with accesses to
this table, there is no slowdown.

This may be obvious and unappealing, but as another workaround you
could store an auxiliary fixed size variable containing a copy of the
time variable with enough capacity to hold all the times you will ever
add to one dataset.  If you update this whenever you add a new time
and read from it rather than from the time record variable, you can
get all the time values in only a few disk accesses and things won't
slow down significantly as you search through more times.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden          http://www.unidata.ucar.edu/staff/russ