Re: aggregating NAM model output

Rich Signell wrote:
> Netcdf-java folk:
>
> I tried a simple NcML aggregation along time for a few 12-km NAM
> files, downloaded from motherlode via the NetCDF subset server.  The
> forecasts are updated every 6 hours, and each forecast has a different
> time origin.  Does NetCDF-Java only look at the first time origin when
> concatenating?
>
> I was trying to create a "best wind" time series by taking only
> forecast hours 3 and 6 from each 72 hour forecast.   If I load the
> files using NetCDF-Java, the times are correct, with a uniform time
> interval of 3 hours.   But when I look at the aggregation, the first
> two time values get repeated for the duration of the time series.
>
> The .nc files and .ncml file are in this zip:
>
> http://stellwagen.er.usgs.gov/rps/share/nam_test.zip  (1.1 Mb)
>
> Is this a bug, and if so, is there a workaround?
>
> Thanks,
> Rich

Hi Rich:

The problem is that a joinExisting aggregation simply works on the file by 
manipulating array indexes, it doesnt know anything about coordinates. If you 
look at the underlying files, you see that the time coordinates are indeed 
repeated (3, 6), but the units attribute is changing:

    :units = "hours since 2006-09-25T06:00:00Z";

however, theres no way currently to make NcML adjust the coordinate values for 
a changing unit.

To fix this, you have to override the coordinate values "by hand" in the NcML 
file, for example:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
<aggregation dimName="time" type="joinExisting"> 
 <netcdf location="20060925_0600.nc" coordValue="3,6"/>
 <netcdf location="20060925_1200.nc" coordValue="9,12"/>
 <netcdf location="20060925_1800.nc" coordValue="15,18"/>
 <netcdf location="20060926_0000.nc" coordValue="21,24"/>
</aggregation>
</netcdf>



We are working on a new kind of aggregation for model run files, that does know 
how to adjust dates automatically. This would be the equivilent:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
<aggregation dimName="runtime" type="forecastModelRunCollection" 
timeUnitsChange="true">
 <netcdf location="20060925_0600.nc" enhance="true"/>
 <netcdf location="20060925_1200.nc" enhance="true"/>
 <netcdf location="20060925_1800.nc" enhance="true"/>
 <netcdf location="20060926_0000.nc" enhance="true"/>
</aggregation>
</netcdf>

The power of this is most apparent by using a scan element:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
<aggregation dimName="runtime" type="forecastModelRunCollection" 
timeUnitsChange="true">
  <scan location="C:/data/test/NAMagg/" suffix=".nc" enhance="true" />
</aggregation>
</netcdf>

However, you dont directly get a time series here, you get a dataset with two time 
coordinates, which can then be processed further, for example to get the "best" 
time series. This is experimental, and you will need the latest development version of 
2.2.17. Some draft docs are here if you want to play with it:

 http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/FmrcAggregation.html


It does seem possible that joinExisting could try to adjust the coordinate 
values of a joined dimension, which might be a more direct solution for your 
use case. I would welcome your thoughts on this.