Re: netCDF and XML

Hi Kirby,

>  I was very interested to follow the links from your home page, 
>  and to learn more about your plans for netCDF. 
>  
>  PRODUCT GENERATION TRACKING
>  ===========================
>  
>  Here at the Met. Office we are thinking of replacing a legacy 
>  system which tracks the generation and distribution of products. 
>  Typically we:-
>  
>   - start from a run of a global NWP model, 
>   - produce files of output ('fields-files' - sadly not in 
>     netCDF format!),
>   - generate products (GRIB files, T4 images etc.),
>   - send then to a GTS message switch,
>   - route them to Washington, Melbourne etc.
>   - from whence they find their way to end users (who may 
>     put them into netCDF format).
>  
>  Our current system has a few holes in it and we are considering 
>  ways in which we can achieve better trace-ability from source to 
>  destination (recording the processes on the way).
>  
>  As part of netCDF you have defined the CDL language which can be 
>  used to describe most of our datasets (in terms of content). 
>  Have you used any 'language' (other than the processing 
>  programme itself) to describe the high level processes that are 
>  used to transform one netCDF file to another?

No, although this might be useful.  There are several mini-languages
that have been developed by others to express operations in a kind of
netCDF algebra for slicing, transposing, extracting, combining,
concatenating, merging, and differencing netCDF datasets, but we don't
use them much here.  I'm referring in particular to FAN, NCO, and NetCDF
Toolbox for MATLAB-5, all described in

 http://www.unidata.ucar.edu/packages/netcdf/software.html

At one time we wrote up a specification for a fairly complete set of
"netCDF operators" that we thought could be connected with command-line
plumbing to do lots of useful things, but the benefits didn't justify
the expense of implementing them all, so the project was abandoned.

>  For example given a global NWP dataset in netCDF format which 
>  contains various meteorological variables as a function of 
>  latitude, longitude, pressure, time, you can produce a new 
>  netCDF file which contains the surface temperature at T+24, but 
>  you can't produce a plot of buoy observations (because they are 
>  not in the original file). In principle it might be possible to 
>  determine whether a given transformation is possible or not.
>  
>  XML/SGML
>  ========
>  
>  We too are interested in XML and SGML. I submitted a paper to 
>  WMO last year asking that they consider making available all 
>  their technical publications in XML or SGML. Last week I 
>  attended the WMO Region 6 (Europe) meeting and persuaded them to 
>  insert in their recommendations that structured formats (such as 
>  XML/SGML) should be adopted as a format for the distribution of 
>  technical documents.

That's great.

>  My colleague Chris Little has produced a draft Logical Data 
>  Model which I have recast as an SGML document (copy attached in 
>  Word and HTML). I'd be interested to know whether UNIDATA is 
>  involved in any such initiatives?

Is there a URL for an HTML version of this that's kept up-to-date
available from a web server somewhere?  I'd like to be able to just
refer to it with a URL and have it point to the latest version ...

I talked to Chris at a WMO meeting in Washington D.C. last December and
got a copy then, which I've made available to a few other developers
here.  

We're trying to come up with some good Java interfaces for abstract
model output grids, soundings, and satellite images, for the purpose of
developing a new Java application suite for analysis and display of
meteorological data.  We don't know much about SGML and are just
learning about XML.

>  We have also begun storing some of our documentation in SGML. In 
>  the case of one database (MetDB) the documentation can be 
>  'output' in Word and indexed HTML, and we are able to 
>  dynamically generate Web forms which can be used to interrogate 
>  the database.
>  
>  I'd be interested to learn about the areas in which Unidata sees 
>  XML being applied.

I have an interest in RDF for representing metadata, but the fact that
RDF uses XML for syntax seems sort of incidental to the RDF model for
metadata.  We have been conducting some weekly "brainstorming sessions"
among developers here to discuss developments such as XML, RDF, and
various Java APIs.  In case you're interested, there are links to
outlines of our talks at 

  http://www.unidata.ucar.edu/staff/russ/bs/

I'd be interested in keeping up with your and Chris Little's efforts,
since it seems we do have quite a few interests in common.  Thanks for
sending me the update.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu