[Support #JWA-773791]: building best estimate aggregations using ncml

> Hi all,
> 
> I have a few questions about Thredds aggregations.
> 
> My first goal (1) is to make a best estimate view of forecast model runs
> updated every days (in this example because it could be every 6 hours, ..)
> These runs have more than 24 hours forecast so I'd like to aggregate only
> the first 24 time steps of each files.
> 
> Moreover (2), I need to build that kind of dataset with another aggregation
> of type union to merge netcdf files because different parameters (variables)
> are separated in many files (but for sure, share the same time axis values)
> 
> I think (after having discussed about it here quite long time ago..) 

where is "here" ?

it is
> not possible to specify an offset for selecting for example
> the T+24 => T+48 part of a file, however it seems to be possible to select
> the T => T+24 part of files using the ncoords attribute of the netcdf
> elements.
> That was the way I found to build the best estimate views using thredds 3.16
> 
> In these examples, the data files contains 73 time coords, hourly. So
> wrfBRE_d03_temp_2009-01-01T00Z.nc contain data from 2009-01-01T00Z to
> 2009-01-04T00Z
> Taking the 24 first time coords of this file would give a period from
> 2009-01-01T00Z to 2009-01-01T23Z
> 
> 1) JoinExisting aggregation
> ===========================
> 
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> <aggregation dimName="time" type="joinExisting">
> <netcdf location="file:///data/runs/wrfBRE_d03_temp_2009-01-01T00Z.nc"
> ncoords="24"/>
> <netcdf location="file:///data/runs/wrfBRE_d03_temp_2009-01-02T00Z.nc"
> ncoords="24"/>
> <netcdf location="file:///data/runs/wrfBRE_d03_temp_2009-01-03T00Z.nc"
> ncoords="24"/>
> <netcdf location="file:///data/runs/wrfBRE_d03_temp_2009-01-04T00Z.nc"
> ncoords="73"/>
> </aggregation>
> </netcdf>
> 
> Using thredds 3.16 with these configurations, I use the Dataset Query Form,
> select only the time variable with the default proposed slice:  0:1:144, but
> when
> I submit the form, I get the error
> Error {
> code = -1;
> message = "NcSDArray ArrayIndexOutOfBoundsException=null";
> };
> 
> With the slice 0:1:143, I get the time values correctly
> 
> Any idea why I cannot get the last time coord ?
> 
> Using thredds 4.0.26, I get the time values correctly in both cases

bug was fixed in 4.0

> 
> 
> 2) JoinExisting + Union aggregations
> ====================================
> 
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> <aggregation dimName="time" type="joinExisting">
> <netcdf>
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-01T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-01T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf>
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-02T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-02T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf>
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-03T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-03T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf>
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-04T00Z.nc"
> ncoords="73"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-04T00Z.nc"
> ncoords="73"/>
> </aggregation>
> </netcdf>
> </aggregation>
> </netcdf>
> 
> I still have the last time coord problem using thredds 3.16, but the N-1
> available time values are still correct
> 
> Using thredds 4.0.26, I get the 144 time values but these are strangely
> corrupted, I mean it's like the 24 time values of the first
> union aggregation (time values of wrfBRE_d03_temp_2009-01-01T00Z.nc) are
> reproduced four times: I get the same values for the four
> union aggregations !
> This give me the following time evolution
> 2009-01-01T00Z => 2009-01-04T00Z => 2009-01-01T00Z => 2009-01-04T00Z =>
> 2009-01-01T00Z => 2009-01-04T00Z => 2009-01-01T00Z => 2009-01-04T00Z
> 
> 
> Now adding the ncoords attribute to the union's parent netcdf element like
> this:
> 
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> <aggregation dimName="time" type="joinExisting">
> <netcdf ncoords="24">
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-01T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-01T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf ncoords="24">
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-02T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-02T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf ncoords="24">
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-03T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-03T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf ncoords="73">
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-04T00Z.nc"
> ncoords="73"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-04T00Z.nc"
> ncoords="73"/>
> </aggregation>
> </netcdf>
> </aggregation>
> </netcdf>
> 
> Then I get quite the same behavior but the resulting period is varying
> between each call (submit)
> starting at 2009-01-01T00Z or 2009-01-02T00Z, or 2009-01-03T00Z or
> 2009-01-04T00Z
> and getting for example when starting at 2009-01-01T00Z, the following
> evolution:
> 2009-01-01T00Z => 2009-01-02T00Z => 2009-01-01T00Z => 2009-01-02T00Z =>
> 2009-01-01T00Z => 2009-01-02T00Z => 2009-01-01T00Z => 2009-01-04T00Z
> 
> (when the result starts from 2009-01-02T00Z:
> 2009-01-02T00Z => 2009-01-03T00Z => 2009-01-02T00Z => 2009-01-03T00Z =>
> 2009-01-02T00Z => 2009-01-03T00Z => 2009-01-02T00Z => 2009-01-05T00Z)
> 
> This is very strange... am I missing something ?
> 
> I also tried to redefine the time variable in combination with the ncoords
> attribute at different levels of the aggregations but the behavior were
> differents and not those expected, I can expose these if needed.
> 
> Now a last question, maybe the most important .. does the way I do to
> achieve this is the right one and if not how could I do ?


currently i dont think you can "cut off" the aggregation using the ncoords 
element. perhaps you should get things working without that first? also, i 
would use the latest 4.1 release. Its not yet stable, but has many bugs fixed. 
If you do see a bug, we will likely only fix it on 4.1.




> 
> Best regards
> 
> Jonathan
> 
> 


Ticket Details
===================
Ticket ID: JWA-773791
Department: Support THREDDS
Priority: Normal
Status: Open