[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [netcdf-java] Writing a 2 Gb NetCDF File

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: [netcdf-java] Writing a 2 Gb NetCDF File
  • Date: Thu, 19 Jun 2008 11:12:48 -0600


Valerio Angelini wrote:

Hi John,

i tried my program with the new release and it works perfectly.
Regarding the performance issue I can't write the data in that order because I cannot choose the order in which I receive the data to be written. 

Just a few questions:
- is it used by the library, or is it possible to enable, at low level the use of the java.nio memory mapped IO? This could maybe speed up my process which needs to write the data not sequentially.
surprisingly, i havent seen any speedup with memory mapped files in Java, at least on Windows under jdk 1.5. so at this point that is not a possibility, though i could add it in the future if it seems worth the extra complexity. 


- which is exactly the meaning of the setLength() function call?
it preallocates the OS file, which may make writing large files faster, though I havent actually timed it. 



Thank you again for your fast and precise support.


you're welcome



Valerio Angelini

Il giorno 17/giu/08, alle ore 20:16, John Caron ha scritto:


Hi Valerio:

1. 2.2 does not support writing large file.
2. 4.0 had various bugs which should now be fixed. I will release 4.0.17 later today. you must add 

   ncFile.setLargeFile(true);

to enable files > 2 Gb.

3. for performance, call

   ncFile.setFill(false);
   ncFile.setLength(approxSize);
before the create() call. with this, it took 267 secs to run on my windows machine.
the most performance is to write the data in physical order. when i did that, the program completed in 72 secs. below is modified version of your program. 

thanks for helping to debug!

-----

 public void testBig() throws IOException, InvalidRangeException {

   long start = System.nanoTime();
   System.out.println("Begin <=");

   String varName = "example";

   int timeSize = 8;
   int latSize = 8022;
   int lonSize = 10627;
System.out.println("File size (B) = " + (long) timeSize * latSize * lonSize * 4); System.out.println("File size~ (MB) = " + Math.round((long) timeSize * latSize * lonSize * 4 / Math.pow(2, 20)));
NetcdfFileWriteable ncFile = NetcdfFileWriteable.createNew("D:/temp/bigFile2.nc"); 
   ncFile.setFill(false);
   ncFile.setLargeFile(true);

   long approxSize = (long) timeSize * latSize * lonSize * 4 + 4000;
   ncFile.setLength(approxSize);

   String timeUnits = "hours since 2008-06-06 12:00:0.0";
   String coordUnits = "degrees";

   Dimension[] dim = new Dimension[3];

   dim[0] = setDimension(ncFile, "time", timeUnits, timeSize);
   dim[1] = setDimension(ncFile, "lat", coordUnits, latSize);
   dim[2] = setDimension(ncFile, "lon", coordUnits, lonSize);

   ncFile.addVariable(varName, DataType.FLOAT, dim);

   ncFile.addVariableAttribute(varName, "_FillValue", -9999);
   ncFile.addVariableAttribute(varName, "missing_value", -9999);

   System.out.println("Creating netcdf <=");
   ncFile.create();
   long stop = System.nanoTime();
   double took = (stop - start) * .001 * .001 * .001;
   System.out.println("That took "+took+" secs");
   start = stop;

   System.out.println("Writing netcdf <=");

   int[] shape = new int[]{1, 1, lonSize};
   float[] floatStorage = new float[lonSize];
   Array floatArray = Array.factory(float.class, shape, floatStorage);
   for (int t = 0; t < timeSize; t++) {
     for (int i = 0; i < latSize; i++) {
       int[] origin = new int[]{t, i, 0};
       ncFile.write(varName, origin, floatArray);
     }
   }

   ncFile.close();

   System.out.println("Done <=");
   stop = System.nanoTime();
   took = (stop - start) * .001 * .001 * .001;
   System.out.println("That took "+took+" secs");
   start = stop;
 }
private static Dimension setDimension(NetcdfFileWriteable ncFile, String name, String units, int length) { 

   Dimension dimension = ncFile.addDimension(name, length);
   ncFile.addVariable(name, DataType.FLOAT, new Dimension[]{dimension});
   ncFile.addVariableAttribute(name, "units", units);

   return dimension;
 }


--
Valerio Angelini
Institute of Methodologies for Environmental Analysis
Italian National Research Council
phone: +39 0574 602535
e-mail: address@hidden