Re: 20050329:Is it possible to archive streaming data using NetCDF?

Unidata Support wrote:

> ------- Forwarded Message
>
>  
>
> > To: address@hidden
> > From: "Peyush Jain" <address@hidden>
> > Subject: netCDF Java - Newbie Question: Is it possible to archive streaming 
> > data using NetCDF?
> > Organization: UCAR/Unidata
> > Keywords: 200503291623.j2TGNUiv002216
> >    
>
> Institution: NASA
> Package Version: Version 2.1
> Operating System: Windows XP Pro
> Hardware Information: P4, 3.0GHz, 2GB RAM
> Inquiry: Hello,
>
> I was wondering if it is possible to archive streaming data using netCDF. If it 
> is, can you please point me to an example? A
no problem with streaming data - its just like any other. the main thing 
is to make sure your outer dimension is the unlimited dimension, so that 
you can continue to expand the file.

example below


> ssuming the data coming in has 5 one dimensional arrays (of different types), I 
> was able to create 5 variables and save them to a netCDF file. Everytime I 
> receive this data, it overwrites the previous data in the file. I am sure that 
> it is not appending the data because the file size never increases.
>
> I am using the following function to write to netCDF file:
> ncfile.write(name, ArrayAbstract.factory(doubleArray));
>  
problem is you are using this call, where origin assumed to be 0

 /** Write data to the named variable, origin assumed to be 0. Must not 
be in define mode.
  * @param varName name of variable. IllegalArgumentException if 
variable name does not exist.
  * @param values write this array; must be same type and rank as Variable
  * @throws IOException
  */
 public void write(String varName, Array values) throws 
java.io.IOException, InvalidRangeException;

you should use this one, and create an origin array; then increment the 
outer dimension each time you write (leave the others zero):

 /** Write data to the named variable. Must not be in define mode.
  * @param varName name of variable. IllegalArgumentException if 
variable name does not exist.
  * @param origin offset within the variable to start writing.
  * @param values write this array; must be same type and rank as Variable
  * @throws IOException
  */
 public void write(String varName, int [] origin, Array values) throws 
java.io.IOException, InvalidRangeException {


 public void testNC3WriteWithRecord() throws IOException {
   NetcdfFileWriteable ncfile = new 
NetcdfFileWriteable("C:/temp/writeRecordExample.nc", false);

   // define dimensions, including unlimited
   Dimension latDim = ncfile.addDimension("lat", 64);
   Dimension lonDim = ncfile.addDimension("lon", 128);
   Dimension timeDim = ncfile.addDimension("time", -1);

   // define Variables
   Dimension[] dim3 = new Dimension[3];
   dim3[0] = timeDim;
   dim3[1] = latDim;
   dim3[2] = lonDim;

   // double T(time, lat, lon) ;
   //   T:long_name="surface temperature" ;
   //   T:units = "degC" ;
   ncfile.addVariable("T", DataType.DOUBLE, dim3);
   ncfile.addVariableAttribute("T", "long_name", "surface temperature");
   ncfile.addVariableAttribute("T", "units", "degC");


   // float lat(lat) ;
   //   lat:units = "degrees_north" ;
   ncfile.addVariable("lat", DataType.FLOAT, new Dimension[] {latDim});
   ncfile.addVariableAttribute("lat", "units", "degrees_north");

   // float lon(lon) ;
   // lon:units = "degrees_east" ;
   ncfile.addVariable("lon", DataType.FLOAT, new Dimension[] {lonDim});
   ncfile.addVariableAttribute("lon", "units", "degrees_east");

   // int time(time) ;
   //   time:units = "hours" ;
   ncfile.addVariable("time", DataType.INT, new Dimension[] {timeDim});
   ncfile.addVariableAttribute("time", "units", "hours");

   //  :title = "Example Data" ;
   ncfile.addGlobalAttribute("title", "Example Data");

   // create the file
   try {
     ncfile.create();
   }  catch (IOException e) {
     System.err.println("ERROR creating file");
     e.printStackTrace();
     return;
   }
   System.out.println( "ncfile = "+ ncfile);

   // now write one record at a time
   Variable v = ncfile.findTopVariable("T");
   ArrayDouble data = new ArrayDouble.D3(1, latDim.getLength(), 
lonDim.getLength());
   ArrayInt timeData = new ArrayInt.D1(1);
   int[] origin = new int[v.getRank()];
   int[] timeOrigin = new int[1];

   for (int time=0; time<100; time++) {
     // fill the data array
     Index ima = data.getIndex();
     for (int j=0; j<latDim.getLength(); j++)
       for (int k=0; k<lonDim.getLength(); k++)
         data.setDouble(ima.set(0,j,k), (double) time*j*k);
     timeData.setInt( timeData.getIndex(), time);

     // write to file
     origin[0] = time;
     timeOrigin[0] = time;
     try {
       ncfile.write("T", origin, data);
       ncfile.write("time", timeOrigin, timeData);
     } catch (IOException e) {
       System.err.println("ERROR writing file");
     } catch (InvalidRangeException e) {
       e.printStackTrace();
     }
   }

   // all done
   try {
     ncfile.close();
   } catch (IOException e) {
     System.err.println("ERROR writing file");
   }

   System.out.println( "**** TestWriteRecord done");
 }