Re: 950306: Memory allocation using the C++-interface

> >From: address@hidden (Mechthild Wolber)
> >Organization: .
> >Keywords: 199503061503.AA28730

Hi Mechthild,

> I want to define a 3-dimensional (144*72*16) data structure with netCDF.
> My problem is that there are segmentation faults when I try to run my
> C++ - program with such a huge structure.
> 
> My question: can I allocate buffer for my object 'm'? I think that this 
> already done by the library functions, but I although try to find a 
> possibility to run the programm.
> 
> I'm sendig my code as attachment, 

I think there is a misunderstanding about what the

       NcVar::put( const float* vals, const long* counts );

member function expects for the `counts' parameter, perhaps due to unclear
documentation.  The `counts' parameter you provide should specify the size
in each dimension of the slab of values you are providing, rather than
the entire size of the variable into which you are putting values.  The
documentation says:

    The following methods put values into the variable object for
    hyper-slabs of n dimensions.  The counts argument is a vector containing
    the size of each edge of the hyper-slab being put.  As above, the start
    corner is [0,0,...,0] by default, but can be changed by use of the
    set_cur() method.

    bool put(const char* vals, const long* counts);
    bool put(const short* vals, const long* counts);
    bool put(const long* vals, const long* counts);
    bool put(const int* vals, const long* counts);
    bool put(const float* vals, const long* counts);
    bool put(const double* vals, const long* counts);

So where you used:

     static float m_data[] = {
        950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961,
        962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973
       };
     m->put(m_data, m->edges());

The m->edges() vector specifies that you are providing a 144*72*16 block of
values, but you are actually providing only 24 values in the m_data[] array.
Hence when the library tries to access values past the 24th to write them,
it causes a segmentation violation.  The library functions don't allocate an
in-memory buffer for all the values for m, because this is an I/O library
and the values are stored on disk, not in memory.  It may not be possible to
allocate enough memory to hold all the values in a variable, since it might
be too large.  But no matter how large `m' is, you can write all the values
into it in whatever order you want an in whatever size batches you choose (1
at a time, or all at once if you have all the values in memory).

If the 24 values you were providing were actually intended to be

   m[0][0][0], m[0][0][1], m[0][0][2], ..., m[0][0][23]

for example, what you should use is something like:

     long m_edges[] = {1, 1, 24};
     m->put(m_data, m_edges);

If the 24 values you were providing were instead intended to be

   m[0][0][0], m[0][0][1], m[0][0][2],
   m[0][1][0], m[0][1][1], m[0][1][2],
   ...
   m[0][7][0], m[0][7][1], m[0][7][2],

then you should instead use something like:

     long m_edges[] = {1, 8, 3};
     m->put(m_data, m_edges);

and so on.

> #include <string.h>
> #include "netcdf.hh"
> 
> #define NUM(array) (sizeof(array)/sizeof(array[0]))
> 
> int
> main( void )
> {
>     const char* path = "netCDFdaten.nc";
>     NcNewFile nc (path); // Create and leave in define mode
> 
>     // Check if the file was opened
>     if (! nc.is_valid()) {
>       cerr << "can't create netCDF file " << path << "\n";
>       return 0;
>     }
>  
>     // Create dimensions
>     NcDim* xd = nc.add_dim("x", 144);
>     NcDim* yd = nc.add_dim("y", 72);
>     NcDim* zd = nc.add_dim("z", 16);
> 
>    
> 
>     // Create variables and their attributes
>     NcVar* m = nc.add_var("m", ncFloat, xd, yd, zd);
>     m->add_att("Erklaerung", "Messwerte zu t=1995 01 31 12:00 und p=Ozon");
>     m->add_att("Einheit", "ppv");
>     static float range[] = {0., 1500.};
>     m->add_att("valid_range", NUM(range), range);
>     m->add_att("_FillValue", -9999.0f);
> 
>     NcVar* x = nc.add_var("x", ncFloat, xd);
>     x->add_att("long_name", "Laenge");
>     x->add_att("units", "GradOW");
> 
>     NcVar* y = nc.add_var("y", ncFloat, yd);
>     y->add_att("long_name", "Breite");
>     y->add_att("units", "GradNS");
> 
>     NcVar* z = nc.add_var("z", ncLong, zd);
>     z->add_att("long_name", "Hoehe");
>     z->add_att("units", "Km");
> 
>     // Global attributes
>     nc.add_att("Zeit", "12:00");
>     nc.add_att("Datum", "1995 01 31");
>     nc.add_att("Parameter", "Ozon");
> 
>     // Start writing data, implictly leaves define mode
> 
>     float *xs = new float[xd->size()];
>     for(int i = 0; i < xd->size(); i++)
>       xs[i] = -90. + 2.5*i;
>     x->put(xs, xd->size());
> 
>     float *ys = new float[yd->size()];
>     for(i = 0; i < yd->size(); i++)
>       ys[i] = -180. + 2.5*i;
>     y->put(ys, yd->size());
>  
>     float *zs = new float[zd->size()];
>     for(i = 0; i < zd->size(); i++)
>       zs[i] = 10. + 0.5*i;
>     z->put(zs, zd->size());
> 
>     static float m_data[] = {
>       950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961,
>       962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973
>       };
>     m->put(m_data, m->edges());
>     
>     // close of nc takes place in destructor
> }

--Russ

______________________________________________________________________________

Russ Rew                                                UCAR Unidata Program
address@hidden                                          P.O. Box 3000
http://www.unidata.ucar.edu/                          Boulder, CO 80307-3000
______________________________________________________________________________