Re: WDSSII Files

Kevin L. Manross wrote:
> Hi John,
>
> I have multiple issues that I'm struggling with while trying to modify 
> a WDSSII netCDF file so that attempts to meet accepted netCDF 
> standards.  If you can bear with me, I'd like to lay them out and if 
> you might have the time to answer them, I'd love to hear what you have 
> to say.
>
>
> 1) I mentioned on the THREDDS listserv that we have historically used 
> sparse grids.  I'd be happy to try and work around this, but I don't 
> think we can give up the compression we're getting from using the 
> sparse grid to eliminate space for "missing values".
its possible that netcdf-4 compression will give you similar results, so 
lets try that first. It may take a few iterations to really check that out.
> You mentioned trying netCDF-4 to try and get the compression back.  I 
> built netCDF-4.0.1 on my RHEL5 machine and simply tried running ncgen4 
> -k 3 to make a netCDF-4 file out of the CDL file I made by dumping an 
> existing (non sparse grid) data file.  I didn't get any compression 
> out of this, but that's not too surprising, as the little that I've 
> been able to glean suggests the the compression is in a C (or FORTRAN) 
> method for the variable.  Is that the route I need to go?
yes, youll need to set deflate=true on each data variable, see:

http://www.unidata.ucar.edu/netcdf/docs_snapshot/netcdf-c.html#nc_005fdef_005fvar_005fdeflate

im not sure what deflate level to set, try a couple, presumable higher = 
smaller, slower.

youll also want to play with chunking parameters:

http://www.unidata.ucar.edu/software/netcdf/docs_snapshot/netcdf-c.html#nc_005fdef_005fvar_005fchunking

alhtough you could do that on pass 2.

> 2) Along the lines of saving space, can I set up my netCDF file 
> without explicitly declaring my domain grid (i.e., lat/lon)?  Perhaps 
> it won't take up that much space, but we're dealing with a 3000x6200 
> (latxlon) grid and maybe the 9200 gridpoint values for the domain 
> won't add much to the overall size.
is this a uniform grid? if so, one sets 1D lat and 1D lon and its a 
trivial sze.
> 3) Is there a way to write netCDF-4 files from the java libraries yet, 
> or is the only way to write netCDF-4 from the tarball that needs to be 
> compiled?
not yet, were working on a JNI interface, but its not ready; youll need 
the netcdf-4 C library anyway. I assume you already have a C/Fortran 
program that writes the files? If so, tweaking that will be the fastest 
thing to do.
> 4) When I try to open up one of our non sparse grid netCDF files in 
> the toolsUI application, I get an error that says "New variable (cref) 
> must have a datatype attribute".  This only occurs when using toolsUI 
> 4.x.  When I use toolsUI 2.2x, it seems to not have a problem
can you send me a sample file with this problem? which tab in ToolsUI 
are you using?
> 5) To try to set up my existing netCDF file (non sparse grid) to be 
> CDM compliant, it seems that I mainle need to improve on the header 
> information.  (referencing the sample file I submitted to the THREDDS 
> listserv)  Does that seem correct?  If so, should I just try some 
> adjustments using the NcML capabilities in the toolsUI app and write 
> out the file and then test the CDM compliance?
yes, assuming the compression makes the files small enough, adding the 
CF metadata is not too hard.
> I know I'm throwing a bunch at you.  I've got some support to get our 
> data up to standards, and I'd really like to try to achieve this.  If 
> you can help out in any way, I'd be very grateful!  Thanks!
would be great to have your data in THREDDS!