Re: Paper on OpenDAP

Hi Tennessee,

On Aug 16, 2005, at 2:54 AM, Tennessee Leeuwenburg wrote:

> Peter,
>
> One further question re your 2003 paper. One of the interoperability 
> layers you identify as "Semantic", and the comment is "more 
> consistent" -- by this do you mean such things as conventions within 
> formats, such as COARDS within NetCDF?

Sort of, the structural part of COARDS. I actually identify the layer 
that you are referring to as "Semantic-Structure". The idea here is 
best understood in terms of the example presented in the paper. 
Consider a set of 2 dimensional satellite-derived sea surface 
temperature (SST) fields, each field consisting of SST on a grid in, 
say the western North Atlantic, at one instant in time. All grids in 
the data set are in the same geographic projection. The data set might 
have all of the fields, say one per day, for a 10 years. These data 
could be stored a number of different ways:

1) 3650 files, each file representing one 2-d field,

2) One 3-d (lat, lon, time) file with the time dimension having 3650 
elements,

3) Ten 3-d (lat, lon, time) files with the time dimension of each 
having 365 elements,

4) One 3-d (time, lat, lon) file, again with time having 3650 
elements,...

The "Structure" layer says that all such data sets should be 
represented as one 3-d object. The "Semanti-Structure" layer says that 
they should all be represented as one 4-d object with lat always being 
the first dimension, lon always being the second dimension, depth the 
third and time the fourth. Since this is no depth in this example, the 
3 layer would just have one element. The point here is that the 
structure is constrained by the semantics. As you move from one layer 
to the next, less metadata are required. The "Structure" layer does not 
specify the order of the variables, so metadata telling the client what 
the order is is required. The Semantic-Structure layer does not require 
such metadata. Higher up in the level structure is the metadata that 
describes the data, the metadata that I refer to as semantic and 
syntactic metadata. I don't think that it is worth making a big deal 
out of the Semantic-Structure level, I simply included it to make the 
overall picture clear. The amount of metadata required to specify the 
semantics of the structure is small. Furthermore, there is not a lot of 
agreement on the semantics of the structure which means that data are 
stored in a variety of semantic structures. Converting the data on the 
fly to a consistent semantic structure is a lot more work than 
converting them to a consistent structure, the next level down. I think 
that that is valuable and that's what the Aggregation Server does.

> Do any such conventions exist for OpenDAP?

No. The idea is to let different disciplines impose whatever structure 
they want on top of OPeNDAP. The more appropriate question would be 
"Does the oceanographic (or the Earth Science) community have any such 
convention?" Unfortunately, the answer is again no. That is my fault. I 
should have recommended one a long time ago. I think that COARDS (or 
CF) would be an excellent starting point and COARDS is becoming a de 
facto standard in that many data providers use it. I was reluctant to 
impose COARDS on the oceanographic use of the system since that would 
mean the reordering of some archives which could be very costly. I 
prefer a modified C
COARDS, one that does not adhere to the semantic structure, but does 
adhere to the rest of the standard.

> Here at BOM we more or less have our conventions agreed on for my 
> particular project, but if there were any recommended best practise it 
> could be interesting.

What do you think of adopting COARDS or CF? Would one or the other 
address all of your data sets? If not, what is missing? If it could, 
but your would rather not adopt it, it would really be useful to know 
why and to know what you guys have adopted and why. I am hoping that 
the Marine Metadata effort comes up with a recommendation for the 
oceanographic community. I'm not sure what is being done for the 
meteorological community, but it would be great if there was a similar 
effort (to the Marine Metadata program) and if they came up with a 
recommendation.

Peter

> Cheers,
> -Tennessee
--
 Peter Cornillon
   Graduate School of Oceanography     -  Telephone: (401) 874-6283
      University of Rhode Island                 -               Fax: 
(401) 874-6728
        Narragansett, RI 02882                    -           E-mail: 
address@hidden