Re: Flavor of netCDF, how does it taste?

Dan Kowal wrote:
> Hi John,
>
> I think the last time I saw you was at the Tea House with Ted Habermann 
> a long while ago.  Hope all is going well by you.  Have a question 
> (really a favor to ask) for you. I was wondering if you had any time to 
> help me evaluate a data format for our archive.  I facilitate the 
> process of new proposals for archiving data at NGDC and one of the 
> things we appraise is the data formats the data provider desires to send 
> us.  I have a new proposal before me that touts a "nearly plug 
> compatible NetCDF format. "  The data is produced by a new insitu 
> observatory system called VIPIR.  It produces ionospheric soundings that 
> characterize the lower ionosphere.  It provides an electron density 
> profile, error bars and other information about the state of the 
> ionosphere in common engineering units of measure.
>
> They have created their own file extension (i.e. 
> stationID_YYYYDOYHHMMSS.NGI) and provided me with an ncdump of the "NGI" 
> data structure - see attached.
>
> I'd appreciate any help, but understand you have a lot on your plate.  
> We're looking at having the appraisal done by the end of the month.  One 
> of the things we're looking for is whether or not data formats are 
> "Independently Understandable" borrowed from the OAIS.  I had Ted run a 
> Unidata Discovery Conformance test on some test data available here 
> ftp://ftp.ngdc.noaa.gov/ionosonde/data/WI937/individual/2010/040/ionogram/ 
> against http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";.  The 
> test only showed 3 out of 42 metadata elements filled with information.  
> Question is whether or not this is a show stopper and what information 
> can we convey to the data provider to improve the data format.
>
> Thanks in advance for any wisdom you can share,
>
> Dan

Hi Dan:

Im not sure what "nearly plug compatible NetCDF format" means ?? Theres no reason not to make it a real netcdf file. 

The main issue is with the metadata. We specialize in "earth science" data. Im 
not sure if thats appropriate for ionispheric data, but if it is, they should use the CF 
conventions. It appears to be a constant lat/lon, with varying height ???

What should be in the "discovery metadata" varies widely. Ted is probably the 
best expert on this around, but youll have to decide whats appropriate for your group.

John