[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Java netCDF 2.2 / Structures
- Subject: Re: Java netCDF 2.2 / Structures
- Date: Mon, 27 Feb 2006 14:47:34 -0700
Norman Fomferra wrote:
Dear John,
Thanks to the very intuitive (read-only) Java netCDF API we have now
been able to quickly provide a reader for netCDF formatted satellite
raster data into the open-source ESA / BEAM software
(http://www.brockmann-consult.de/beam/).
You're welcome.
Unfortunately I have my problems with the writable API. It simply won't
let me create unlimited arrays of Structure records. I would very much
appreciate your comments to the (very, very simple!) test code I have
attached.
Theres a very simple answer. Netcdf (version 3) file format does not support
writing structures, so the API does not have that option.
You can "write structures" using the record dimension, but you are doing it at a lower level, namely writing the variable data for all record variables for record 0, then 1, then 2, etc.
I have made some comments in your file below. Below that is example code.
I am pretty sure that other Java netCDF newbies will run into similar
problems as well, so maybe you can use the code as a demonstration of
how the API shall NOT be used. Especially, because I was not able to
find any unit-test or example code which demonstrates the use of
Structures with the Java API.
I have used netcdf-2.2.14.jar + nlog4j-1.2.22.jar with JDK 1.5.0_06 on a
Windows XP PC.
By the way, I think it was not a good idea to force clients to use the
nlog4j library now. Most professional applications that use the netCDF
API already use some kind of logging. E.g. in BEAM we use the Java
Logging API which is even capable of interacting with logging services
that already exist on the host operating system. We are happy about
every JAR we do not need in our classpath.
The slf4j is a facade that can use various loggers, including the jdk 1.4
logger. nlog4j is the implementation that uses log4j. See the slf4j web site to
download the jdk 1.4 implementation, which you can substitute for nlog4j.
Tell me, if more feedback on the netCDF Java API is welcome and if so,
in which form.
Yes, feedback is good, though we can not always respond quickly. Send to the
netcdf-java email list.
Best regards and thanks in advance for your help
Norman
____________________________________________
Norman Fomferra (Brockmann Consult)
GITZ / GKSS
Max-Planck-Str. 2 21502 Geesthacht
Germany
Tel: +49 (0)4152 889 303
Fax: +49 (0)4152 889 333
Mail: address@hidden
Web: http://www.brockmann-consult.de
___________________________________________
------------------------------------------------------------------------
import ucar.nc2.*;
import ucar.ma2.DataType;
import java.io.IOException;
import java.util.Arrays;
/**
* Provides 2 tests which create an unlimited array of structure records.
* {@link #testAddRecordStructure} has no effect and
* {@link #testAddMyOwnStructure} throws an exception:
* <pre>
* Exception in thread "main" java.lang.IllegalStateException: unknown DataType
== Structure
* at ucar.nc2.N3header.getType(N3header.java:434)
* at ucar.nc2.N3header.writeVars(N3header.java:627)
* at ucar.nc2.N3header.create(N3header.java:487)
* at ucar.nc2.N3iosp.create(N3iosp.java:299)
* at ucar.nc2.NetcdfFileWriteable.create(NetcdfFileWriteable.java:320)
* at
TestUnlimitedStructureArray.testAddMyOwnStructure(TestUnlimitedStructureArray.java:107)
* at TestUnlimitedStructureArray.main(TestUnlimitedStructureArray.java:24)
* </pre>
*/
public class TestUnlimitedStructureArray {
public static final String LOCATION_1 = "test1.nc";
public static final String LOCATION_2 = "test2.nc";
/**
* Performs both tests.
* @param args ignored
*/
public static void main(String[] args) {
try {
testAddRecordStructure(LOCATION_1);
testAddMyOwnStructure(LOCATION_2);
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* Test 1: The method tries to create the unlimited array of records using
the
* NetcdfFileWriteable.addRecordStructure() from all unlimited
* 1-D arrays in the file.
*/
public static void testAddRecordStructure(String location) throws
IOException {
// Obtain file instance in 'define mode'
final NetcdfFileWriteable file =
NetcdfFileWriteable.createNew(location, false);
// Define the unlimited dimension
final Dimension dimN = file.addDimension("n", 0, true, true, false);
// Define four unlimited arrays
file.addVariable("bin_number", DataType.INT, new Dimension[]{dimN});
file.addVariable("counts", DataType.SHORT, new Dimension[]{dimN});
file.addVariable("sum", DataType.FLOAT, new Dimension[]{dimN});
file.addVariable("sum_sqr", DataType.FLOAT, new Dimension[]{dimN});
// create the file which is then no longer in 'define mode'
file.create();
// addRecordStructure() only works after file.create(),
// because otherwise the file has no 'spi'.
// Note that this breaks with the API convention, because 'file'
// is now not in 'define mode' anymore.
file.addRecordStructure();
Remove addRecordStructure, it doesnt help you with writing
dumpVariable(file, "record"); // --> record is correctly dumped out
file.close();
dumpFile(location); // --> 'record' is gone, arrays are the same! ERROR?
}
You dont want to do any of testAddMyOwnStructure(). You can only use the
methods in NetcdfFileWriteable.
/**
* Test 2: The method tries to create the unlimited array of Structure
record from all unlimited
* 1-D arrays in the file by utilizing the
NetcdfFileWriteable.addRecordStructure() .
*/
public static void testAddMyOwnStructure(String location) throws
IOException {
// Obtain file instance in 'define mode'
final NetcdfFileWriteable file =
NetcdfFileWriteable.createNew(LOCATION_2, false);
// Define the unlimited dimension
final Dimension dimN = file.addDimension("n", 0, true, true, false);
// Create the unlimited array of structure type 'bin_list'
final Structure binList = new Structure(file, null, null, "bin_list");
binList.setDimensions(Arrays.asList(new Dimension[]{dimN}));
// Define four structure members
Variable v1 = new Variable(file, null, binList, "bin_number");
v1.setDataType(DataType.INT);
v1.setDimensions((String) null); // = scalar
binList.addMemberVariable(v1);
Variable v2 = new Variable(file, null, binList, "counts");
v2.setDataType(DataType.SHORT);
v2.setDimensions((String) null); // = scalar
binList.addMemberVariable(v2);
Variable v3 = new Variable(file, null, binList, "sum");
v3.setDataType(DataType.FLOAT);
v3.setDimensions((String) null); // = scalar
binList.addMemberVariable(v3);
Variable v4 = new Variable(file, null, binList, "sum_sqr");
v4.setDataType(DataType.FLOAT);
v4.setDimensions((String) null); // = scalar
binList.addMemberVariable(v4);
// add structure array to root group
file.addVariable(null, binList);
// Note: Strange to say, at this point, file.findVariable("bin_list")
// yields 'null'. So I had to call file.finish() first.
file.finish();
// Ok, now we find 'bin_list', but file is still in 'define mode'
dumpVariable(file, "bin_list"); // --> 'bin_list' looks good to me
// create the file which is then no longer in 'define mode'
file.create(); // throws java.lang.IllegalStateException: unknown
DataType == Structure
// Never comes here
file.close();
dumpFile(location);
}
private static void dumpFile(final String location) throws IOException {
final NetcdfFile file = NetcdfFile.open(location);
System.out.println("file '" + location + "' = \n" + file);
file.close();
}
private static void dumpVariable(final NetcdfFile file, final String
fullName) {
final Variable variable = file.findVariable(fullName);
System.out.println("variable '" + fullName + "' = \n" + variable);
}
}
package ucar.nc2;
import junit.framework.*;
import ucar.ma2.*;
import java.io.IOException;
/**
* Simple example to create a new netCDF file with a "record structure"
* @author : John Caron
*/
public class TestWriteRecordStructure extends TestCase {
static String fileName = TestNC2.topDir+"testWriteRecordStructure.nc"; //
default name of file created
static boolean dumpAfterCreate = false;
public TestWriteRecordStructure( String name) {
super(name);
}
public void testWriteRecordStructure() throws IOException,
InvalidRangeException {
NetcdfFileWriteable writeableFile = new NetcdfFileWriteable(fileName, false);
// define dimensions, including unlimited
Dimension latDim = writeableFile.addDimension("lat", 3);
Dimension lonDim = writeableFile.addDimension("lon", 4);
Dimension timeDim = writeableFile.addDimension("time", -1, true, true,
false);
// define Variables
Dimension[] dim3 = new Dimension[3];
dim3[0] = timeDim;
dim3[1] = latDim;
dim3[2] = lonDim;
writeableFile.addVariable("lat", DataType.FLOAT, new Dimension[] {latDim});
writeableFile.addVariableAttribute("lat", "units", "degrees_north");
writeableFile.addVariable("lon", DataType.FLOAT, new Dimension[] {lonDim});
writeableFile.addVariableAttribute("lon", "units", "degrees_east");
writeableFile.addVariable("rh", DataType.INT, dim3);
writeableFile.addVariableAttribute("rh", "long_name", "relative humidity");
writeableFile.addVariableAttribute("rh", "units", "percent");
writeableFile.addVariable("T", DataType.DOUBLE, dim3);
writeableFile.addVariableAttribute("T", "long_name", "surface temperature");
writeableFile.addVariableAttribute("T", "units", "degC");
writeableFile.addVariable("time", DataType.INT, new Dimension[] {timeDim});
writeableFile.addVariableAttribute("time", "units", "hours since
1990-01-01");
// create the file
writeableFile.create();
// write out the non-record variables
writeableFile.write("lat", Array.factory(new float[] {41, 40, 39}));
writeableFile.write("lon", Array.factory(new float[] {-109, -107, -105,
-103}));
//// heres where we write the record variables
// different ways to create the data arrays. Note the outer dimension has
shape 1.
ArrayInt rhData = new ArrayInt.D3(1, latDim.getLength(), lonDim.getLength());
ArrayDouble.D3 tempData = new ArrayDouble.D3(1, latDim.getLength(),
lonDim.getLength());
Array timeData = Array.factory( DataType.INT, new int[] {1});
int[] origin = new int[] {0, 0, 0};
int[] time_origin = new int[] {0};
// loop over each record
for (int time=0; time<10; time++) {
// make up some data for this record, using different ways to fill the
data arrays.
timeData.setInt(timeData.getIndex(), time * 12);
Index ima = rhData.getIndex();
for (int lat=0; lat<latDim.getLength(); lat++)
for (int lon=0; lon<lonDim.getLength(); lon++) {
rhData.setInt(ima.set(0, lat, lon), time * lat * lon);
tempData.set(0, lat, lon, time * lat * lon / 3.14159);
}
// write the data out for this record
time_origin[0] = time;
origin[0] = time;
writeableFile.write("rh", origin, rhData);
writeableFile.write("T", origin, tempData);
writeableFile.write("time", time_origin, timeData);
}
// all done
writeableFile.close();
}
}