[netCDF #QAA-889221]: nc_inq_format vs. nc_inq_format_extended

Wei-keng,

> Thanks for these explanations in details.
> 
> I have one more question.
> Say, my program uses NC_64BIT_OFFSET flag to create a new file. Then
> a call to nc_inq_format() will gets me NC_FORMAT_64BIT in the returned 
> formatp.
> Similarly, if not using NC_64BIT_OFFSET, I got NC_FORMAT_CLASSIC.
> This is fine.
> 
> But I found nc_inq_format_extended() could not tell the classic format from
> the 64-bit offset format (i.e. CDF-1 and CDF-2 formats). I tested it with
> netCDF 4.3.2 and got NC_FORMAT_NC3 for both formats. Is this correct?

The nc_inq_format_extended() function is not intended to distinguish
among all the file formats that are already handled by
nc_inq_format().  Instead, it is intended to distinguish among data
sources that cannot be distinguished by their behavior through the
other API calls, but that are handled by different libraries from the
"dispatch layer" of netCDF.

Maybe it will halp to look at this graphic of the netCDF architecture,
and in particular the bottom blue layer:

  http://www.unidata.ucar.edu/netcdf/docs/architecture.html

Those 5 different data sources correspond to 5 different libraries
handled by the dispatch layer, and there are times when a program may
need to know about the actual source of the data, instead of its
behavior through APIs.  That's the purpose of
nc_inq_format_extended().  Note that the 64-bit offset format is not
handled by a different library than netCDF-3 classic, so it's not a
return from the new function.

There was no netCDF API call that could distinguish between a netCDF
file accessed from disk versus data accessed remotely from a DAP
server, until nc_inq_format_extended() provided that capability.
Similarly, there is no other call that can distinguish between an
ordinary netCDF-4 file and an HDF4 or HDF5 file that is not a netCDF-4
file, but that is still readable through the netCDF-4 API.

Maybe a better name for the function would be nc_inq_dispatch(), but
many users would not be familiar with that use of the word "dispatch".

--Russ


> On Oct 3, 2014, at 5:02 PM, Unidata netCDF Support wrote:
> 
> > Hi Wei-keng,
> >
> >> After checking the netCDF docs on line, I still do not fully understand 
> >> the difference
> >> between two APIs: nc_inq_format and nc_inq_format_extended. They both have 
> >> an argument
> >> formatp, but their returned values are not expected to be the same, 
> >> according to the
> >> description of the header file netcdf.h. For the former, the possible 
> >> values are
> >> #define NC_FORMAT_CLASSIC (1)
> >> #define NC_FORMAT_64BIT   (2)
> >> #define NC_FORMAT_NETCDF4 (3)
> >> #define NC_FORMAT_NETCDF4_CLASSIC  (4)
> >
> > Right, those format numbers are associated with limitations and
> > features that depend on the physical format as well as which of the
> > two data models they support:
> >
> > - NC_FORMAT_CLASSIC is restricted to 32-bit file offsets and the size
> >  limits that implies
> >
> > - NC_FORMAT_64BIT allows larger variables while maintaining
> >  compatibility with the classic data model
> >
> > - NC_FORMAT_NETCDF4 supports features of the netCDF-4 enhanced data
> >  model with extended primitive types, groups, and user-defined types
> >
> > - NC_FORMAT_NETCDF4_CLASSIC supports netCDF-4 performance features
> >  such as compression and chunking while remaining compatible with
> >  programs using only the classic data model
> >
> >> and for the latter, the possible values are:
> >> #define NC_FORMAT_NC3     (1)
> >> #define NC_FORMAT_NC_HDF5 (2) /*cdf 4 subset of HDF5 */
> >> #define NC_FORMAT_NC_HDF4 (3) /* netcdf 4 subset of HDF4 */
> >> #define NC_FORMAT_PNETCDF (4)
> >> #define NC_FORMAT_DAP2    (5)
> >> #define NC_FORMAT_DAP4    (6)
> >> #define NC_FORMAT_UNDEFINED (0)
> >>
> >> What the returned values of formatp are expected to be used?
> >
> > Those returned values are intended for use by developers who
> > need to know the true format of the underlying storage that
> > is made to behave like netCDF files by the dispatch layer of
> > the library. This is useful to properly interpret some error
> > conditions or to understand performance behavior. Accessing
> > remote data through DAP protocols that might actually be a
> > virtual time subset of an aggregation from a collection of
> > GRIB files, involving a lot of server-side processing will
> > generally be slower and have different causes for errors than
> > reading from a locally stored HDF4 file made made to look as
> > if it's a netCDF file by the library.
> >
> >> (Other than comparing with the above defined constants.)
> >> Does netCDF expect users to use the returned values in other APIs?
> >
> > No, I don't think so, unless needed for error interpretation
> > or for understanding performance.
> >
> >> Since the first 4 constants between the 2 groups shared the same
> >> integral values, can they cause any trouble?
> >
> > I don't think so. They are used for different purposes, so they
> > shouldn't conflict. Maybe we should have come up with a different
> > name than "format" for the extended file types, but "protocol"
> > didn't exactly fit either.  The two lists may not be completely
> > orthogonal, but serve the needs of the Java and netCDF utilities,
> > such as ncdump, when we need information to help in debugging
> > performance problems or error conditions.
> >
> >> Also, could I make a suggestion on the naming of NC_FORMAT_PNETCDF and 
> >> NC_FORMAT_64BIT.
> >> Since PNETCDF is not a file format and PnetCDF supports two file-create 
> >> flags
> >> NC_64BIT_DATA and NC_64BIT_OFFSET, maybe a better naming for these two in 
> >> netcdf.h
> >> are NC_FORMAT_CDF5 and NC_FORMAT_CDF2, respectively?
> >
> > Thanks for the suggestion.  We'll consider it, or maybe using
> > aliases to clarify the intent.
> >
> > I'm Cc:ing Dennis, the developer of this function, who may have
> > more to add about use cases for these constants.
> >
> > --Russ
> >
> >> Wei-keng Liao
> >>
> >>
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: QAA-889221
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: QAA-889221
Department: Support netCDF
Priority: Normal
Status: Closed