[netCDF #IXU-921318]: help for reducing the number of fill value

Amanda,

> The  Indexed ragged array representation of trajectories sounds like a
> good solution. Could you please send me an example of a program in c?
> Thanks for your help!
> Amanda


The easiest way to get a program that uses the netCDF C API to
generate data corresponding to a specified CDL representation is to
use the "ncgen" tool.

For example, if you store the CDL in the CF Conventions example H.15
in a file named "H15.cdl", then you can get a C program that generates
that data as follows:

  $ ncgen -l c H15.cdl > h15.c

To do this, you have to change a few things in the H.15 example text
to make it legal CDL:

  - surround the CDL by the lines

      netcdf h15 {
        ...
      }

  - Add the dimension definition for the length of trajectory names:

      name_strlen = 10 ;

  - change occurrences of the non-ASCII '``' character around a few of the
    strings to be the ASCII double-quote '"'.

Change the "attributes:" declaration to the comment

   // global attributes

I've done this in the attached file "H15.cdl".  

The resulting h15.c program compiles and links fine with the netCDF C
library.  Running it creates a file named "H15.nc", which has metadata
but no data:

  $ ncgen -l c H15.cdl > h15.c
  $ cc h15.c -o h15 `nc-config --cflags --libs`
  $ ./h15
  $ ncdump h15.nc
     ... shows CDL of the netCDF file ...

If you want to write some sample data as well, you can either add more
CDL for the "data:" section to the H15.cdl file and regenerate h15.c
and h15, or you can manually insert the necessary nc_put_var... calls.

--Russ


> Le 18/06/2013 18:06, Unidata netCDF Support a écrit :
> > Hi Amanda,
> >
> >> I am generating a netcdf file containing multi-dimensional variables:
> >>
> >> dimensions:
> >> time = 1 ;
> >> cell = UNLIMITED ; // (23 currently)
> >> level = 2 ;
> >> contour = 178 ;
> >> pt_traj = 8 ;
> >> slice = 62 ;
> >>
> >> variable:
> >> float temp(cell, pt_traj, slice) ;
> >>
> >> In my case, the number of temp data value can vary from one cell to
> >> another (ex: temp(1,2,*) contains 10 good values while temp(1,4,*)
> >> contains 178 good values). Is there a way to reduce the number of fill
> >> values? (attached is the file i generated)
> > Yes, there are several ways.
> >
> > One is to reorganize your data to follow a convention developed for
> > trajectory data, such as the CF Conventions, which provides two
> > representations for such data:
> >
> >    H.4.3. Contiguous ragged array representation of trajectories
> >    
> > http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8393872
> >
> > or
> >
> >    H.4.4. Indexed ragged array representation of trajectories
> >    
> > http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8399648
> >
> > Another solution is to use compression as supported in netCDF-4.  In this 
> > case, you
> > can preserve the current multi-dimensional representation of your data, but 
> > you need
> > to specify you want the variables compressed when you create them.  The C 
> > function
> > to call would be nc_def_var_deflate to specify the "deflate" compression 
> > level.  There
> > is an analogous Fortran function.
> >
> > Then the data could be written and read using the same code you are using 
> > now, but to
> > read it, the reading application would have to be linked with the netCDF-4 
> > library,
> > which would uncompress the data chunks as the data is read.  For more 
> > information
> > about this, see this section of the netCDF Workshop on compression:
> >
> >    http://www.unidata.ucar.edu/netcdf/workshops/2012/nc4chunking/
> >
> > You can test how well compression works on your data by using the "nccopy" 
> > utility
> > program to copy and compress a typical data file, for example:
> >
> >    nccopy -d1 RDT_output_200905251845.nc RDT_output_200905251845-d1.nc
> >    ls -l RDT_*.nc
> >    -rw-rw-r-- 1 russ ustaff 285251 Jun 18 10:02 
> > RDT_output_200905251845-d1.nc
> >    -rw-rw-r-- 1 russ ustaff 313586 Jun 18 10:01 RDT_output_200905251845.nc
> >
> > shows that level 1 deflation made the file about 90% of the size of the 
> > original.
> >
> > --Russ
> >
> >
> >
> >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: IXU-921318
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
> 
> --
> Amanda Gounou
> Météo France
> DPREVI/PI
> address@hidden
> 
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: IXU-921318
Department: Support netCDF
Priority: Normal
Status: Closed

Attachment: H15.cdl
Description: Binary data