Re: 20030719: hdf-eos to netcdf conversion

>From: Suzana Djurcilov <address@hidden>
>To: Russ Rew <address@hidden>
>Subject: Re: 20030719: hdf-eos to netcdf conversion
>Organization: UC Santa Cruz
>Keywords: 200307190018.h6J0I0Ld012879 HDF to netCDF

Suzana,

> Thanks for asking...the ncdump that comes with the 3.5 netcdf distribution
> simply gives me a "this is not a netcdf file" error message, with or
> without the -c flag. I am attaching the header dump that I managed to get
> using the HDF4.1 distribution. To answer the question about YDim being a
> variable, no it isn't. It is just a fancy dimension definition. The ascii
> file resulting from the ncdump is 470M, but not so large as to run into
> the size problem. I think it would be unwise of me to attach the entire
> file in this message, but perhaps you can get a file yourself off of their
> web-site: http://podaac.jpl.nasa.gov/modis/modis_data.html

 ...

> netcdf modis {
> dimensions:
>         YDim:Grid = 4096 ;
>         XDim:Grid = 8192 ;
> 
> variables:
>         byte sst_mask_1(YDim:Grid, XDim:Grid) ;
>                 sst_mask_1:Long_name = "Sea Surface Temperature mean" ;
>                 sst_mask_1:Name = "Flag" ;
>                 sst_mask_1:Units = "bits" ;
>                 sst_mask_1:Scale_type = "y= Slope * x + Intercept;" ;
>                 sst_mask_1:Slope = 1.f ;
>                 sst_mask_1:Intercept = 0.f ;
 ...

I think I see the problem.  In early versions of the netCDF library,
names of variables, dimensions, and attributes could contain any
characters, including ":" for example.  But since we used ":" as part
of the syntax for assigning attributes in the CDL notation, this would
confuse the CDL parser in interpreting a line like

         YDim:Grid = 4096 ;

which could either mean

 - specifying the size of a dimension named "YDim:Grid" as 4096, or
 - specifying the value of an attribute named "Grid" for a variable
   named "YDim" as 4096.

As a result of problems dealing with such ambiguities, netCDF-2.4.3
and later versions did not permit characters such as ":" in new names
of netCDF dimensions, variables, or attributes, but you could still
access data from files with previously existing components with names
like "YDim:Grid".

A better solution would have been to escape such characters in the
ncdump output, as in 

         YDim\:Grid = 4096 ;

and modify ncgen to interpret such escaped names correctly.  

If YDim:Grid and XDim:Grid are the only problem names in your file,
you might just be able to change them to legal netCDF names in the
ncdump output ("-", "_", "." are all permitted in names so "YDim.Grid"
would be OK, for example), then try running "ncgen -b" on the
resulting CDL file.  This would give you a netCDF file where the names
of these dimensions had changed, but that might be OK for your
application.

The alternative of modifying ncgen to understand escaped special
characters in names will have to wait a while, since it's a
non-trivial modification to the CDL parser ...

--Russ