Russ Rew: Re: NetCDF versus HDF

------- Forwarded Message

To: address@hidden
Subject: Re: NetCDF versus HDF 
In-reply-to: Your message of "Tue, 26 Jul 1994 15:05:38 MDT."
Organization: UCAR Unidata Program
Date: Thu, 28 Jul 1994 11:23:21 -0600
From: Russ Rew <russ@buddy>

> Organization: NCAR/HAO
> Keywords: 199407262105.AA02560

Hi Paul,

> I'm trying to tackle a project of making our project's data portable 
> and accessible.  Here are our requirements:
> 
> 1) the data, presently stored in a binary format called UARS, 
>    and stored on repository at Goddard Space Flight Center be
>    transferred and written to a more universal format.  After
>    this, I'm out of the picture ... users will need to write 
>    any additional required routines to manipulate that data.
> 2) the data be in a format machine independent
> 3) the data be compatible with IDL
> 4) HDF readable.
> 
> From compiling and reviewing reference material I have pulled off
> MOSAIC, and a conversation with Russ Rew, I have been able to 
> determine that either HDF or NetCDF may be suitable for our needs.
> At the risk of asking questions you've probably heard all to often, from
> this point on I am unclear how I can determine the advantages or
> disadvantages to either.  
> 
> In particluar:
> 
> ()  HDF has been mandated for EOS and we may pursue this for UARS as
> well, hence requirement #4.  However, am I correct that if a file exists
> in NetCDF, then it can be read via HDF routines and/or reformatted into
> HDF?  easily?  is there reverse (HDF ->>NetCDF) compatiblility?

The HDF library (at least version 3.3 and later) can be built either with or
without the capability to read netCDF files.  As distributed, you can build
just the HDF `base library' (from the hdf/ subdirectory) or you can build
the netCDF/HDF part of the library also (from the mfhdf/ subdirectory before
using the hdf/ subdirectory).  I think most sites that use HDF are using the
mfhdf/ library (which I'll call the `full HDF' library) and thus getting the
capability to read any netCDF file, but I don't know of any way to determine
this for certain.

With the full HDF library, any netCDF file can be read via HDF routines and
written as an HDF-encoded netCDF file.  The HDF interface for reading netCDF
files is identical to the netCDF interface, so it should be equivalently
easy.  In the netCDF software distribution, we supply nothing that reads HDF
files, so there is no reverse compatibility.

With the HDF implementation of the netCDF data access interface, there are
now two kinds of netCDF files:

 - XDR-encoded netCDF files.  These are what is written and read by our
   netCDF software, and they can be read by the full HDF library.

 - HDF-encoded netCDF files.  These are what is written by the netCDF
   that interface in the full HDF library, and they can only be read by the
   that netCDF interface in the full HDF library.

> ()  I have been told that NetCDF is the preferred choice if one needs to
> write customized routines?  is this true?  

It depends.  Some programmers prefer one interface and some prefer the
other.  I have appended excerpts from a reply to a similar question from
Dennis Shea back in June.

> ()  what guidelines can you supply in determining resources required for
>     () writing some routines to read the data into either NetCDF or HDF?  
>     () installation and building libraries?

I'm afraid I can't be of much help with the first question.  I think
installing and building the libraries takes a little longer for HDF because
installing and building the netCDF interface is a subset of the task.  For
netCDF, we have spent a lot of effort in using the GNU "configure" approach
to make the installation relatively painless and independent of platform.

> ()  I have also been told that NetCDF is more straightforward in both
> installation and operation.  Do you concur?

In the sense that the netCDF interface has fewer functions and is thus less
complex, I guess you could say it is simpler and more straightforward.
However, if you limit yourself to using the netCDF interface part of the HDF
library, it is just as straightforward.

> ()  I have been told that the NetCDF documentation is easier to follow
>     and more concise.  Do you concur?

Since I wrote most of the netCDF documentation, I can't give an unbiased
answer.  Again, if you restrict yourself to the netCDF interface of the HDF
library, the documentation is the same.

> ()  any other thoughts on HDF versus NetCDF?

Performance issues may be important.  If you make much use of the unlimited
dimension and record access in netCDF, the XDR-based implementation may have
significant performance advantages.  linked lists in HDF versus direct
access in netCDF made such a large performance difference in some cases.
However, there are some kinds of access for which the HDF-based
implementation is faster.  It sounds like you don't yet have any benchmarks
with which to compare, so this may be difficult.  Several users have
reported that for their purposes, the HDF implementation of the netCDF
interface ran too slowly for them to use.

NASA chose HDF as the interface to use for the first phase of EOSDIS.  I've
never seen the document that gave the criterion used or the evaluation
methodology, but HDF won out over lots of other candidate formats and
interfaces for EOSDIS, including netCDF.  The size of the support staff at
NCSA vs. Unidata as well as the fact that HDF was including the netCDF
interface may have been important factors.

> Bottom line:  I need to determine which will address our needs the best.
> 
> I appreciate your help on this matter.
> 
> Regards,
> 
> Paul Willis
> 
> address@hidden
> (303) 497-1558

Here's the earlier reply sent to Shea:


To: address@hidden (Dennis Shea)
Subject: Re: HDF 
Organization: UCAR Unidata Program
Date: Thu, 09 Jun 1994 11:35:51 -0600
From: Russ Rew <russ@buddy>

> Organization: NCAR/CGD
> Keywords: 199406091630.AA07008

Hi Dennis,

> I have been asked to write a one page
> "position paper" for our group, the
> CLimate Analysis Section, for consideration
> by the people who will decide which format
> to use for future archiving by the climate
> models.
> 
> I will let u see  it before I send it it.
> I will be simple. As I stated in my previous email
> it is between GRIB and netCDF. Howwver, I would like
> to say a few things about HDF. I have asked Dennis
> Joseph about HDF and he says his knowledge of that
> format is rudimentary. Do u know where I can get
> a "non-technical" overview of HDF? In a nutshell,
> what do u think of HDF? pros-ncons?

Yes, below is a recent answer to a very similar question from Ken Schroder
at BB&N.

(Incidentally, you can find this and other information about HDF and netCDF
 by using Mosaic.  Just look at the netCDF home page at
 http://www.unidata.ucar.edu/packages/netcdf/.  Then enter "hdf" in the
 first searchable index of answers to netCDF support questions.  When you
 hit return, you get a menu of all previous support answers to netCDF
 questions that contain the word HDF, sorted by the messages that contain
 the most occurrences first.  Just click on the first of these and you get
 the answer I've appended below.  It doesn't always work this easily, but it
 can be a useful way to get information that doesn't appear in formal
 documents.)

- ------------------------------------------------------------------------------

  First, here's the official answer from the netCDF FAQ at
  http://www.unidata.ucar.edu/packages/netcdf/faq.html

   7. What is the connection between netCDF and HDF?  

      The National Center for Supercomputing Applications (NCSA) developed
      the HDF software and makes it freely available. HDF is an extensible
      data format for self-describing files that was developed independently
      of netCDF. Applications and utilities based on HDF are available that
      support raster-image manipulation and display and browsing through
      multidimensional scientific data. The HDF software includes a package
      of routines for accessing each HDF data type, as well as a lower-level
      interface for building packages to support new types. HDF supports
      both C and Fortran interfaces, and it has been successfully ported to
      a wide variety of machine architectures and operating systems. HDF
      emphasizes a single common format for data, on which many interfaces
      can be built.

      NCSA has implemented software that provides a netCDF interface to
      HDF. With this software, it is possible to use the netCDF calling
      interface to place data into an HDF file. The netCDF calling interface
      has not changed and netCDF files stored in XDR format are readable, so
      existing programs and data will still be usable (although programs
      will need to be relinked to the new library). There is currently no
      support for the mixing of HDF and netCDF structures. For example, a
      raster image can exist in the same file as a netCDF object, but you
      have to use the Raster Image interface to read the image and the
      netCDF interface to read the netCDF object. The other HDF interfaces
      are currently being modified to allow multi-file access, closer
      integration with the netCDF interface will probably be delayed until
      the end of that project.

      Eventually, it may be possible to integrate netCDF objects with the
      rest of the HDF tool suite. Such an integration will then allow tools
      written for netCDF and tools written for HDF to both interact
      intelligently with the new data files.

  We met with the HDF developers a couple of years ago to see if some sort
  of synthesis or combination would be desirable and practical.  What came
  out of that was NCSA's decision to try to add the netCDF interface to the
  set of interfaces layered on top of the HDF format.  HDF originally had a
  simpler interface for scientific data, but the NCSA developers liked the
  netCDF interface better.

  There are significant performance differences between the two
  implementations of the netCDF interface, with each significantly faster
  than the other for some kinds of operations.

  HDF emphasizes a single common format for data, on which many interfaces
  can be built.  NetCDF emphasizes a single common interface to data,
  implemented on top of an architecture-independent representation.

There is a WWW page containing an overview of HDF and links to lots more
information about it.  Just point your Mosaic or other WWW server at:

        http://yahoo.ncsa.uiuc.edu:8001/

(The yahoo host is often overloaded, so you will have to be patient or lucky to
get the above link to work.)

Please let me know if you have additional questions about HDF and netCDF.

- --
Russ Rew                                                   UCAR Unidata Program
address@hidden                                             P.O. Box 3000
http://www.unidata.ucar.edu/                             Boulder, CO 80307-3000

------- End of Forwarded Message