[netCDF #XFB-271301]: netCDF question, char/string attributes, global or variable atts.

Greg,

Just to keep tracking this in our support system, I'm adding that the 
alternative of creating netCDF-4 classic model files instead of netCDF-4
files was possible in NCL, and still permitted the use of compression.

So that is a practical work-around for now for the lack of support for
netCDF-4 string attributes in the current netCDF-4 Fortran library.

I've added a bug tracking issue to finish the implementation of the
missing netCDF-4 enhanced data model functions, so we won't lose track
of the issue:

   https://bugtracking.unidata.ucar.edu/browse/NCFORTRAN-29
   Implement missing support for NF90_STRING primitive type for string 
attributes and data

--Russ

> > You're absolutely right about netcdf4 - I am using this version on purpose
> > to take advantage of the compression.  When I made the older v3 type, it
> > was quite large so I experimented with v4.  Guess I need to find all that
> > I can about the newer API for v4 so I can keep moving forward in the new
> > week because I need to get a particular piece of code written prior to
> > AMS.  So, I'll try.
> 
> Is there any way you can use netCDF-4 classic model files, instead of netCDF-4
> files?  The former support compression with the netCDF-3 data model and data
> types, and appear to be completely implemented in the current netCDF-4 Fortran
> library.  However (and I'm somewhat alarmed to discover this), the netCDF-4
> Fortran library is not just missing documentation and tests for the netCDF-4
> string type, but also missing implementation of support for the netCDF-4
> NF90_STRING data type.  That explains the dearth of examples and test for
> that type in the source code.
> 
> I was mistaken in thinking that a work-around was to use the NF90_VLEN type
> instead of the NF90_STRING type.  As you've discovered by digging around,
> the NF90_VLEN functions are also missing documentation, and I suspect their
> implementation is also incomplete.
> 
> This all works in the C and Java libraries, but apparently you're the first
> to try to make use of the netCDF-4 Fortran string type in the netCDF-4
> enhanced data model, rather than just using the netCDF-4 classic-model format.
> I'm also embarrassed that we didn't notice this issue earlier.
> 
> I'd like to discuss workarounds in more detail for your AMS deadline,
> including use of a little C program to make the change to append to the
> history attribute, a C program to convert that attribute to the NC_CHAR
> type that the F90 library can handle, or other alternatives, but need to
> know more about the problem first.  I'll try calling later ...
> 
> --Russ
> 
> > On Jan 11, 2014, at 2:58 PM, Unidata netCDF Support wrote:
> >
> > > Hi Greg,
> > >
> > >> I am really struggling with something in netCDF and getting a little
> > >> desperate for help - I've done google search much of the last 2 days,
> > >> checked support/mail archives, stack-overflow, etc.  I'd greatly
> > >> appreciate if you could give me additional guidance on this topic.
> > >>
> > >> I have a 64-bit offset NC file I created using NCL reading, processing,
> > >> and writing data from an original GRIBv2 file to make the NC file.  Now,
> > >> for further post-processing and deriving new data, I'm reading in that
> > >> NC file (and others), doing some math on the data, and writing a new
> > >> output NC file, using F90 (ifort compiler on Yellowstone).
> > >>
> > >> I am having a bear of a time with character variables.  And I do
> > >> understand that strings are not stored, but, rather, array of
> > >> characters.  So, when I created my first NC file, I made a global
> > >> attribute called "history" which I can clearly see when I do ncdump -h
> > >> on the file and it looks like this:
> > >>  string :history = "Created Tue Dec 31 11:08:39 MST 2013" ;
> > >
> > > It's apparently not a 64-bit offset netCDF file, but a netCDF-4 file
> > > instead.  You can verify this using ncdump -k, assuming you have an
> > > ncdump linked against netCDF version 4.0 or later.
> > >
> > > If the file was written using NCL, maybe it's configured for the default
> > > to be writing netCDF-4 files.  NetCDF-4 files use an enhanced data model
> > > and support additional data types, such as "string", that are not
> > > accessible to older netCDF-3 libraries or programs linked against them.
> > >
> > > For more information on the differences between the netCDF classic data
> > > model and the netCDF-4 enhanced data model, see this FAQ section on
> > > "Formats, Data Models, and Software Releases":
> > >
> > >  http://www.unidata.ucar.edu/netcdf/docs/faq.html#formats-and-versions
> > >
> > > Evidently, NCL creates netCDF-4 attributes using the netCDF-4 primitive
> > > type "string" instead of an array of chars.
> > >
> > >> And, since I'm doing additional computations on data, I want to append
> > >> to that history variable.  So I'm trying to read in this global
> > >> attribute to capture that string, append new text to the end and then
> > >> write to my new NC output file.
> > >
> > > To read an attribute of type "string" in netCDF-4, you need to know
> > > about netCDF-4 variable-length types, as the netCDF-4 primitive type
> > > NF90_STRING is handled like the NF90_VLEN user-defined type, because
> > > a read returns a pointer to allocated space that must later be
> > > explicitly freed when you're done with it.
> > >
> > > I see there are no good examples of using NF90_STRING type data in our
> > > on-line examples, and the documentation for use of that type is poor.
> > > The netCDF Fortran-90 users guide was written before the netCDF-4 APIs
> > > were added, and it looks like the needed rewrite of the docs was not
> > > completed before the developer working on that project left.
> > >
> > > I'll try to create a new example that reads and writes NF90_STRING
> > > data using the F90 API, but it may have to wait until after the AMS
> > > meeting.  In the mean time, I recommend seeing if you can make NCL
> > > write either netCDF classic format files or netCDF-4 classic model
> > > files, neither of which use strings for attributes.
> > >
> > >> I've read 10s of web pages trying to find anyone who has applied
> > >> NF_GET_ATT function and there's just something I'm missing.
> > >>
> > >> When I use this:
> > >>               nf_status = nf90_inquire_attribute(ncid, NF90_GLOBAL,
> > >> attname, xtype, leng, j)
> > >> where attname='history', it returns xtype=12 and leng=1.
> > >> The tutorial page (for netcdf) says xtype should be one of NF_BYTE,
> > >> NF_CHAR, ..., which are just consecutively numbered 1 to 6, so I am at a
> > >> loss what 12 indicates.  And, of course, a length of this attribute of 1
> > >> is not helpful.  The string reported by ncdump is clearly longer.
> > >
> > > The netCDF-4 documentation for nf90_def_var mentions using NF90_STRING
> > > as a netCDF variable type, but it's not mentioned in the docs as a valid
> > > return type for any of the other netCDF functions!
> > >
> > > The source file fortran/netcdf_constants.f90 has all the netCDF-4 type
> > > codes, which should have been documented in the Users Guide:
> > >
> > >  ! extra data types:
> > >  integer, parameter, public :: &
> > >       nf90_ubyte = 7, &
> > >       nf90_ushort = 8, &
> > >       nf90_uint = 9, &
> > >       nf90_int64 = 10, &
> > >       nf90_uint64 = 11, &
> > >       nf90_string = 12, &
> > >       nf90_vlen = 13, &
> > >       nf90_opaque = 14, &
> > >       nf90_enum = 15, &
> > >       nf90_compound = 16
> > >
> > >> And, then when I attempt just to read in the attribute anyway, of course
> > >> I encounter the error: "NetCDF: Attempt to convert between text &
> > >> numbers" when using:
> > >>   nf_status = nf90_get_att(ncid, NF90_GLOBAL, attname, str_hist)
> > >> where str_hist is a f90 char(len=128).
> > >>
> > >> I just cannot comprehend how to retrieve textual global attributes of
> > >> netcdf files, nor text attributes of a variable, such as "units" which I
> > >> have clearly set in my variables.  The above inquire_attribute and
> > >> get_att functions do exactly the same thing as my history example.
> > >>
> > >> Any help would really be appreciated!
> > >
> > > I sympathize with your frustration at the incomplete documentation.
> > >
> > > I've added a bugtracking issue about this, so it doesn't continue to get
> > > ignored:
> > >
> > >  https://bugtracking.unidata.ucar.edu/browse/NCFORTRAN-28
> > >
> > > Thanks for reporting the problem!
> > >
> > > --Russ
> > >
> > > Russ Rew                                         UCAR Unidata Program
> > > address@hidden                      http://www.unidata.ucar.edu
> > >
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: XFB-271301
> > > Department: Support netCDF
> > > Priority: Normal
> > > Status: Closed
> >
> >
> 
> Russ Rew                                         UCAR Unidata Program
> address@hidden                      http://www.unidata.ucar.edu
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: XFB-271301
Department: Support netCDF
Priority: Normal
Status: Closed