Java, netcdf files and IDV consumption

Richard P Signell wrote:
> Is there a place on the web where we can find examples of 1D, 2D, and 3D 
> netcdf files that *do* work with IDV?

Hi Richard, I saw your posting.  I am currently finishing a converter 
from mm5v3 model output files to CF-1.0 compliant .nc files.  A win is 
direct consumption by IDV, release 1.2 at least.  So far it has been 
working ;)

If you load IDV 1.2 and go to Data Source Chooser|URLs and type

http://metoc.apl.washington.edu/uwme/2005090100-eta-d1.nc

then IDV should download a netcdf file from our server which includes 2D 
and 3D model output fields.

If you wanted to see the .nc format, just type the same URL into a 
regular browser and Save As to your local system and browse with ncdump.

The netcdf functionality of our server is somewhat experimental at the 
moment, since I'm adding handling of the cross and dot staggering of mm5 
grids, but at any time, the 'file' you get should be viewable in IDV.

I learnt most if not all about the CF conventions from the CF web page, 
and from looking at the RUC.cdl example at ucar, and from answers from 
Don Murray!

Also, I have learned that due to the way netcdf files are constructed, 
IDV can actually ask for sections of data without needing all of the 
file contents.  Using the httpclient from Apache commons, IDV hits the 
above url and in the http request says it wants bytes in the range say 
0-20000.  So our web server has to understand this (it's standard http 
1.1, we use Tomcat 5.5.9).  Think of this first data chunk as the table 
of contents of a book, with page numbers to various chapters/sections. 
Then when the IDV user wants to look at some field, IDV works out which 
chapter (byte offset in fact) it's in from the TOC, and asks for a new 
range.  This is neat, since our mm5/nc files are ~50MB which could take 
a while to transfer, while a single field/time combo may only be ~100K.

You can see all these range requests by (a) looking in the web server 
access logs for 206 (partial content) rather than 200 (OK) records, and 
by (b) using tcpmon (google it) as a middle man to view the http 
messages back and forth.  That's an invaluable tool I use it all the 
time for http debugging.  tcpmon becomes the client of the web site and 
the server for IDV, forwarding bytes in both directions while also 
displaying them in a swing panel set.

I will try to build a catalog (adde??) for our data set, at the moment 
you have to know the url format, which is YYYYMMDDHH-model-nest.nc

where HH is 00 or 12, model can be eta, gfs  (plus some others), domain 
is "d1" or "d2".  The data goes back to about 2001.  See our web site at 
http://metoc.apl.washington.edu/uwme/ for more details on what data we 
are processing/offering up.


BTW if anyone is interested from a Java web app view, I use a servlet 
filter technique to launch the mm5 to nc conversion at request time, and 
session expiry to clear up local .nc files.  Our mm5 data set is 3TB and 
we don't have disk space for a local .nc copy as well, so we generate 
just what users want ;)

I hope this rambling is all of use to the community, I had been meaning 
to 'advertise' our .nc data anyways, so your post struck a chord. Feel 
free to ask any questions...

Stuart