[netCDF #DIG-256328]: type 2 (char) data writing

Hi John,

I've forwarded this question to esupport, because the answer is of
more general interest and the question will probably come up again.
For that reason, the response may also be a little more elaborate than
you're looking for.

> if i create a file with NC_FORMAT_CLASSIC:
> 
> and i have a char (type=2) data variable, and i try to write to it with
> 
> nc_put_vars_uchar (grpid, varid, origin, shape, stride, valb)
> 
> or
> 
> nc_put_vars_schar (grpid, varid, origin, shape, stride, valb)
> 
> 
> i get error:
> 
> NetCDF: Attempt to convert between text & numbers

Right, the error message is correct in this case, because you are
using a function intended for writing numeric data to write text data,
and the API doesn't permit that.  This needs a little explanation (and
better documentation).

The "schar" or "uchar" used as a suffix in the function name refers to
the in-memory C type of the data you want to write, not the netCDF
type of the variable to which you are writing.

The only C declaration for 8-bit integers is "char".  There is no
"byte" declaration, although "char" can be qualified as signed or
unsigned.  So if you have some 8-bit numeric data you want to write to
a numeric netCDF variable, you must use one of the nc_put_varX_schar()
or nc_put_varX_uchar() functions, where X can be "1", "", "a", "s", or
"m", depending on what kind of write you intend.  Also the netCDF
variable you are writing to can be any netCDF numeric type: NC_BYTE,
NC_SHORT, NC_INT, NC_FLOAT, or, NC_DOUBLE, or an additional 5 numeric
types for netCDF-4 files.  The numeric data in the memory array of
8-bit values is converted to the numeric type on disk before it's
written.  But you can't mix numeric data and character strings, or ask
the netCDF library to convert numeric data to characters in a string.

If you want to write data to a netCDF variable declared to be of type
NC_CHAR, rather than NC_BYTE, you have declared your intention that
that variable is only for text data.  The functions to use in that
case are the nc_put_varX_text() family, and they all require a last
argument of type const char*.  So the API prevents you from
inadvertently writing floats or ints into what is supposed to be an
array of chars (unless you do it intentionally with casts).

So you need to replace

  nc_put_vars_uchar (grpid, varid, origin, shape, stride, valb)

with

  nc_put_vars_text (grpid, varid, origin, shape, stride, valb)

(which will generally reek havoc on an array of characters being used
for a UTF-8 encoded string, if there are any non-ASCII characters in
the array and stride is greater than 1).

> if i use
> 
> nc4.nc_put_vars_text(grpid, varid, origin, shape, stride, valb);
> 
> i get error:
> 
> NetCDF: Not a valid data type or _FillValue type mismatch
> 
> however, NC_NETCDF4 works ok with nc_put_vars_text and fails on
> nc_put_vars_uchar and nc_put_vars_uchar with
> 
> NetCDF: Attempt to convert between text & numbers

This error message is also what's intended, for the same reason as above.
The "uchar" and "schar" in the function name still only refer to the numeric
type declarations of the C data to be written, not the netCDF type of the
target variable.

> not sure from
> 
> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c.html#nc_005fput_005fvars_005f-type
> 
> what is expected for char type.

Yes, I think better documentation is needed to explain the allowed
conversions that take place when you write (or read) data and the
in-memory type differs from the netCDF on-disk type.  I thought there
used to be a section on type conversions, but now I'm not seeing it.

All plausible numeric conversions are supported, but no conversion is
permitted between text and numeric values.  The intent of the type
part of function names also needs better explanation, as the original
netCDF-4 code actually had a bunch of read and write functions that
used a "_ubyte" suffix which were identical to the existing functions
that used a "_uchar" suffix, because of a misunderstanding of which
type the suffix referred to.  NetCDF client code should not have to
know what numeric type is used in a file for numeric data.  It can
read it all as doubles by just using the functions with the "_double"
prefix.

> 1) what is uchar and schar used for ??

Only for C functions that deal with 8-bit signed or unsigned values
in-memory.  If no conversion between memory and disk is desired, then
uchar corresponds to the netCDF type NC_UBYTE and schar to NC_BYTE.

> 2) it looks like _FillValue must be messedd up, but im not sure why that
> happens.

The default _FillValue for the three netCDF types NC_CHAR, NC_BYTE,
and NC_UBYTE are all different (0, -127, and 255 respectively).
Applications such as ncdump don't use the default fill value for
NC_BYTE or NC_UBYTE data, it must be set explicitly if a fill value is
intended.  I'd have to know more about types of everything and context
to know what caused the error message above that mentioned _FillValue.

--Russ
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: DIG-256328
Department: Support netCDF
Priority: Normal
Status: Closed