[Support #MJA-592376]: FW: Unable to locat netcdf specification (UNCLASSIFIED)

Jennifer,

> It appears that there is time for me to change my CR submission to use this 
> later standard (NetCDF-4/HDF5) instead of the NetCDF-4 (64-bit offset 
> format), but I need to verify that this later version is compatible with how 
> we are storing our data here at NAVO.  Frank, if you have time, could you 
> look at this?

The 64-bit offset format is different from netCDF-4, but the netCDF-4 library 
supports both
transparently (it detects what variant of netCDF format is being accessed and 
uses the
appropriate code).  A better explanation of the four netCDF format variants and 
two netCDF
data models is here:

  http://www.unidata.ucar.edu/netcdf/docs/faq.html#formats-and-versions

Version 4 of the netCDF library is backward compatible with version 3, both for 
data and
for programs.  Unidata's commitment to backward compatibility is emphasized in a
slide that we've included in numerous presentations and training workshops that 
uses these
bullets:

  Because preserving access to archived data for future generations is very 
important:

  - New netCDF software will provide read and write access to all earlier forms 
of netCDF 
    data.
  - C and Fortran programs using documented netCDF APIs from previous releases 
will be 
    supported by new netCDF software (after recompiling and relinking, if 
needed).
  - Future releases of netCDF software will continue to support data access and 
API 
    compatibility.

  http://www.unidata.ucar.edu/netcdf/workshops/2010/netcdf4/Compatibility.html

> There is another option, I could submit this version as an "emerging" 
> standard and keep the previous document as the submitted "mandated" version.  
> In DISR speak, organizations are allowed to use both the emerging and 
> mandated standards.  Mandated are the versions most users should be working 
> under.  Emerging standards are those that are scheduled to replace the 
> mandated standards, after a certain timeframe, and users should be working 
> towards implementing them.

NASA has two netCDF standards, but both are final.  We are not trying to 
replace netCDF-3
with netCDF-4, because netCDF-3 (both the classic format and the 64-bit offset 
format) are
is very widely used due to its simplicity and the large number of tools that 
support it.
NetCDF-4 (supporting both netCDF-3 variants as well as the netCDF-4/HDF5 format 
and the netCDF-4 
classic model format) is a drop-in replacement for netCDF-3 that offers more 
representational
power in exchange for additional complexity and performance features such as 
compression and
chunking.  You could call it an "emerging standard" in the sense that it is 
less widely used
than the netCDF-3 standard, but it may always be less widely used.  There is no 
schedule for
replacing netCDF-3, either the format(s) or the APIs.  It is supported by 
netCDF version 4
software and we intend that that will always be true.

Sorry to belabor this, but it's a source of confusion, and the FAQ section 
referenced above
attempts to clarify the differences among software versions, format versions, 
and data model 
versions.

> Russ, based on the two options above, do you have a recommendation?  It 
> appears (at first glance) that the file formats are backwards compatible - is 
> that true?  If so, then the first option makes the most sense.  But, I would 
> be happy to take any feedback or advise you can provide.

Yes, the file formats and software versions are compatible in the two senses 
described
above.  I would not mandate that users be required to use only the latest 
version of the
format, because some users may not need it, because lots of third-party 
software has not
yet been adapted to accept it (and some may never be), and because community 
conventions
for using netCDF-4 have not yet crystalized around best practices such as the 
widely
used CF Conventions for netCDF-3.

--Russ

> -----Original Message-----
> From: Unidata User Support [mailto:address@hidden]
> Sent: Wednesday, December 21, 2011 11:46
> To: support-netcdf
> Cc: address@hidden; address@hidden; address@hidden; address@hidden
> Subject: [Support #MJA-592376]: FW: Unable to locat netcdf specification 
> (UNCLASSIFIED)
> 
> Hi Jennifer,
> 
> > I was hoping that one of you might be able to help me.  I work for
> > the Naval Oceanographic Office (NAVOCEANO) and I am trying to submit
> > the NetCDF-4 format as a data standard to the Department of Defense
> > Information Standards Registry (DISR).  The DISR is the approved
> > standards registry for the DoD.  Accordingly, if you are a DoD
> > agency, you can only use standards that are approved and registered
> > in the DISR.  Since NAVOCEANO uses the NetCDF-4 data file standard
> > quite extensively, I am trying to get it registered into the DISR
> > for approval to use.
> >
> > Attached is what I have submitted and below are questions I am
> > receiving from DISR about the standard.  I think the DISR people are
> > looking for a document entitled "NetCDF-4" (not ESDS-RFC-011v1.00)
> > that defines what the format is.  I actually don't care what they
> > call the standard that is submitted, so long as we can continue to
> > store our Ocean Forecast Model data in a NetCDF-4 format.  Can you
> > help me?
> 
> Yes, you might want to start with the recently approved NASA ESDS
> standard for netCDF-4 (ESDS-RFC-022), available by following the link
> to the PDF file from this table of NASA Earth science standards:
> 
> http://www.esdswg.com/spg/docindexfolder
> 
> The document title NASA uses is "NetCDF-4/HDF5 File Format, with
> guidance on how to write netCDF conformant files using the HDF API":
> 
> http://www.esdswg.com/spg/rfc/esds-rfc-022/ESDS-RFC-022v1.pdf
> 
> --Russ
> 
> Russ Rew                                         UCAR Unidata Program
> address@hidden                      http://www.unidata.ucar.edu
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: MJA-592376
> Department: Support
> Priority: Normal
> Status: Closed
> 
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: MJA-592376
Department: Support netCDF
Priority: Normal
Status: Closed