Re: announce stable release THREDDS Data Server version 3.4

Steve Hankin wrote:
> John Caron wrote:
>
> > Thank you!
> >
> > BTW, I'll be looking at your case next, in the "new types of 
> > aggregation".
>
> Hi John,
>
> I'll take that as an invitation to elaborate a little on the needs for 
> forecast data set collections. To reiterate, the challenge is to expose 
> three views of a collection of successive (operational) forecasts:
>
>    1. the "integration foreward in time" axes (one virtual data set per
>       model run)
>    2. the "sequence of successive forecasts" axis (one virtual dataset
>       per forecast time increment)
>       and, if possible
>    3. the "successively shorter forecast of the same endpoint" axes (one
>       virtual dataset per end time)

just to make sure i understand, i am going to reword this:

a "ForecastModelRun" is a collection of all the forecasts for one model run. we 
make this into a single netcdf file.

a "ForecastModelRun collection" is a collection of ForecastModelRun files for 
the same model.

so 

#1 = is a virtual dataset consisting of all the time=0 slices in each 
modelForecastRun, except the last, which uses all of the time steps.

#2 = is all of the time steps in a ForecastModelRun

#3 = the set of time steps with constant forecast time

Assuming this is correct, then we already have #2. #1 is probably 
straightforward to do as a special kind of aggregation on the server.

#3 is harder because there is a whole collection of these, one for each 
forecast time. Im not sure whether to make seperate datasets for each, or make 
a more complex object that is parameterized by forecast time (I assume thats 
what you mean by a 5-D dataset).

Any of these aggregatations could efficiently be done on a client, and i might 
be tempted to do it on the client in the nj22 library. But then only java apps 
could use it.

I will take your advice and investigate the vanilla server approach, ie making 
seperate (virtual) datasets on the server that can be accessed by any dods 
client.


> The parentheses indicate where a special challenge is -- namely, that 
> the collection proliferates into a number of discrete virtual datasets.  
> This of course is a reflection of reality; each time the forecast model 
> is run, it  truly does  generate a new dataset.
>
> Here's what I want to add to the discussion:  As a software developer -- 
> striving for simplicity, compactness, elegance -- it may be tempting to 
> solve this by using a more-than-4-dimensional array.  I'd like to lobby 
> that while of course it is fine (and fun) to explore new solution 
> spaces, it is also vital to include in the solution an approach that 
> makes the aggregations look like "standard" (CF) datasets -- i.e. 
> 4-dimensional.  Else, standard CF clients will not be able to read the 
> datasets.
>
> I suspect that this will involve something like the synthesis of a 
> virtual "file system" in the URLs
>
>     http://IP/server/forecast_collection ...
>        /by_start_time...
>           /2006-01-01
>           /2006-01-02
>           /2006-01-03
>           ...
>     /by_forecast_increment ...
>          /z-0
>          /z-4hrs
>          /z-8hrs
>          ...
>     /by_end_time
>          /2006-01-03
>          /2006-01-04
>          /2006-01-05
>          ...
>
> Just initial thoughts.  A unique problem, eh?
>
>     - steve
>
> > Steve Hankin wrote:
> >
> > > Yahoo!  Congratulations.
> > >
> > >    - steve
> > >
> > > John Caron wrote:
> > >
> > > > A new stable release of the THREDDS Data Server 3.4 is now ready at
> > > >
> > > > http://www.unidata.ucar.edu/projects/THREDDS/tech/index.html#TDS
> > > >
> > > > Among other feaures, this version now has all the functionality of 
> > > > the older OpenDAP Aggregation Server (AS), which is no longer 
> > > > supported. The AS allowed aggregation of static sets of files, using 
> > > > "Union", "Join Existing" and "Join New" types. The TDS also adds 
> > > > automatic scans of directories, plus manipulation of the dataset 
> > > > metadata using NcML.
> > > >
> > > > The configuration of the TDS is significantly different than the AS, 
> > > > since the TDS uses NcML Aggregation. See:
> > > >
> > > > http://motherlode.ucar.edu:8080/thredds/docs/NcML.htm
> > > > http://www.unidata.ucar.edu/software/netcdf/ncml/
> > > >
> > > > You are encouraged to upgrade to new stable releases as they become 
> > > > available, since various bugs are fixed. A stable release means that 
> > > > no further features will be added, and only bug fixes will be made.
> > > > New development is now being done on the 3.5 version. This version 
> > > > will be adding more features, especially to deal with dynamically 
> > > > changing datasets, new types of aggregation, and some of the 
> > > > sophisticated features of TDS catalog generation (filter, sort, 
> > > > name) will be pushed into NcML aggregation. Use the development 
> > > > version only if you need the new features and are willing to bleed a 
> > > > little bit.
>
> --
> --
>
> Steve Hankin, NOAA/PMEL -- address@hidden
> 7600 Sand Point Way NE, Seattle, WA 98115-0070
> ph. (206) 526-6080, FAX (206) 526-6744