Mark Rehbein wrote:
Hi John, The idea of NcML sounds perfect and will give it a go when you announce it in the coming weeks.
Just finished new NcML pages at http://www.unidata.ucar.edu/packages/netcdf/ncml/
We are now experiencing performance issues with reading large portions of data (2.5million pixels) from our netcdf files. It takes about 7 seconds to read the data from the netcdf file. The resulting image is 1900 x 1300 pixels, which is full resolution. However, full resolution is rarely required for requests that cover a large spatial area. So it would be useful for me if there was a read method on VariableDS that could subsample the data to get a lower resolution. Hopefully, this would result in a quicker read time. I am keen to help make the API as fast as possible, so just let me know how I can help.
1.we have prototyped new version using NIO and are seeing speedups from 2.5 to 8 over current java. a few slides are at
http://www.unidata.ucar.edu/staff/caron/presentations/CurrentWork/CurrentWork.htm
that will take a few months to become operational, but let me know if you want to alpha test it.
2. the subsampling is possible, but im not so sure it would speed things up that much when reading on the server. Once you have an array, you can decimate using
public Array section( Range[] ranges); (note Range can have a "step" in the constructor). i will think more about it.
I would like to say, however, that I am finding your API extremely useful so keep up the good work. Thanks for your time, Mark
thanks!
On Mon, 26 May 2003, John Caron wrote:Hi Mark: We are working on a thing called "NcML Dataset" that allows you to describe a dataset that consists of multiple netcdf files using an XML dialect called NcML. Im trying to get this API finalized also, and will announce it in the next few weeks so you can have a look, and try itout.There may be some performance issues with very large datasets like you indicate above.I look forward to hearing your thoughts on what is needed, and perhaps we can work together on making it as fast aspossible.Currently we are only planning on continuing to provide an API that extracts rectangular subsets, not diagonals.Mark Rehbein wrote:Hi John, I have made some excellent progress in our satellite imagery software which is using the netcdf java API you are actively developing. Our application is a web server based where images/maps are generated on the web server (tomcat) using the ESRI MapObjects Java Edition API, JAI and the NetCDF API. The generated maps are then sent to the clients web browser as a JPEG or PNG. The generated maps have GIS data/layers laid over the sattelite imagery retrieved from the netCDF file. I would like to know if you are planning to include in the netcdf API, a set of classes that could manage multiple netcdf files such that it would appear to clients of the classes that they were dealing with a single very large netcdf file. I have a need for this functionality as our imagery is currently taking up 2GB for each 6 months of imagery and we plan to host 15 years worth and as such we would like to split our imagery into netcdf files with 1 months worth of data in each. It would be expected that the system be able to retrieve a time series that would span multiple months/years. I will write my own set of classes to handle the multiple netcdfs, but it might be something that could be included in the netcdf java API in the future. Please let me know your comments on this. Also, I require functionality in the API that could read data from an image in the netcdf file as show in the following example: Data along a "diagonal" line from Point 1 (lat:-10,lon:149) to Point 2 (lat:-15,lon:156). At the moment, I plan to do this by reading all the data bounded by the above coordinates, using JAI to rotate the image (while applying pixel interpolation) such that the line between Point 1 and Point 2 is normal to the Y axis of the rotated image. Then I can read the data along the line easily. I may have missed something in the doco/manual, but I could not see anything in the API that could do this for me. Thanks for all your help so far as I understand that you are busy stabilising the API in preparation for production release. cheers Mark On Fri, 16 May 2003, John Caron wrote:Mark Rehbein wrote:Hi John, I have noticed that you are actively working on the API as I have seen the javadoc change on GridCoordSys in the last few hours. I have since downloaded the library again, as I was using this class for my program.yes, Im trying to get a stable version out. sorry for the changes.I have made some progress but have a question and wish to report a bug/problem. My question is: How do I easily convert a java.util.Date to an array index on the time axis? There is a method GridCoordSys.findXYCoordElement(double, double, int[]) which handles converting latitude and longitude to array indexes and I think it would be useful to have an equivalent for the time axis.how about: /** * Given a Date, find the corresponding time index on the time coordinate axis. * Can only call this is hasDate() is true. * This will return * <ul> * <li> i, if time(i) <= d < time(i+1). * <li> -1, if d < time(0) * <li> n-1, if d > time(n-1), where n is length of time coordinates * </ul> * * @param d date to look for * @param corresponding time index on the time coordinate axis * @throws UnsupportedOperationException is no time axis or isDate() false */ public int findTimeCoordElement(Date d) ;The bug or problem that I have discovered is a negative seek offset error as shown below: <error> java.io.IOException: Negative seek offset at java.io.RandomAccessFile.seek(Native Method) at ucar.netcdf.RandomAccessFile.read_(RandomAccessFile.java:508) at ucar.netcdf.RandomAccessFile.seek(RandomAccessFile.java:350) at ucar.netcdf.NetcdfFile$V1FloatIo.readArray(NetcdfFile.java:1447) at ucar.netcdf.NetcdfFile$V1Io.copyout(NetcdfFile.java:896) at ucar.netcdf.Variable.copyout(Variable.java:276) at ucar.nc2.Variable.read(Variable.java:184) at ucar.nc2.dataset.VariableDS.read(VariableDS.java:334) at aims.app.science.biodiv.sstAtlas.ImageLibrary.readData(ImageLibrary.java:166) at aims.app.science.biodiv.sstAtlas.ImageLibrary.main(ImageLibrary.java:423) done! </error> I have appended the method I have written below for your convenience, if needed. The method works fine when I use: latitude= -7.0 and longitude= 157.99 latitude= -27.0 and longitude= 157.99 latitude= -27.0 and longitude= 142.00 latitude= -7.0 and longitude= 142.00 but it throws the above error when using lat= -19.6 and lon= 150.5 which are coordinates within the bounds of coordinates which work!i assume you found this problemI have noticed that the *bottom* *left* corner of my images (in lat,lon coordinates) represents 0,0 in array index coordinates. I would have expected that the *top* *left* corner of my images (in lat,lon coordinates) would be represented by 0,0 in array index coordinates.latitudes are stored both ways: north to south ans south to north. I am going to add a routine to normalize this in GeoGrid, havent figured out the API yet.