[netCDFJava #HIP-291542]: Question on NcML aggregation

Li,

After looking at the files you are serving up, I think joinExisting is the way 
to go.  You'll need to set up different NcML aggregations for the different 
buoys (DUO and SEF, it looks like).  Is there some other information you 
thought users would be selecting?  I'm not entirely sure what you mean in your 
comments below on joinExisting.

-Lansing

> Thanks Lansing and Matt!
> 
> I put the three ways of aggregated files under the test THREDDS server.
> Here are few comments.
> http://data.nodc.noaa.gov/thredds/catalog/testdata/20130328/catalog.html
> 
> joinNew: There is "timeagg" in the OPeNDAP selection, but the actual
> temperature values are extracted from the first timeagg, no matter which
> timeagg we choose.
> 
> joinExisting: this one picks up all the values for all the joined files.
> Though there is no list of station_name, plfatform1, instrument, for the
> users to select.
> 
> Union: this one picks up only the first file in the aggregation, and there
> is no way to select the files after.
> 
> The joinExisting is the way to go.
> 
> Thanks,
> Li
> 
> 
> 
> 
> 
> address@hidden> wrote:
> 
> > Matt,
> >
> > I set up an aggregation using four of the files.  At it's simplest, this
> > works to join the datasets along the existing time dimension:
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >   <aggregation dimName="time" type="joinExisting">
> >     <netcdf id="1" location="DOU001_000_20070530_20070906.nc"/>
> >     <netcdf id="2" location="DOU001_000_20070906_20071107.nc"/>
> >     <netcdf id="3" location="DOU001_000_20080807_20081204.nc"/>
> >     <netcdf id="4" location="DOU001_000_20100803_20101117.nc"/>
> >   </aggregation>
> > </netcdf>
> >
> > Since you already have a good dimension for aggregating (time), then you
> > probably don't need to worry about doing a joinNew.  You would use that if
> > there were no time data in the files, and you wanted to add it into the
> > dataset you were serving, for instance.
> >
> > Let me know if there is something else you are trying to do, I'm happy to
> > help.
> >
> > -Lansing
> >
> > > Lansing,
> > >
> > > When I set it up using the "joinExisting" on the time dimension it seems
> > to
> > > aggregate them. Below is the ncml:
> > > <?xml version="1.0" encoding="UTF-8"?>
> > > <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> > > <variable name="lat">
> > > <attribute name="standard_name" type="String" value="latitude"/>
> > > <attribute name="axis" type="string" value="Y"/>
> > > <attribute name="units" type="string" value="degrees_north"/>
> > > </variable>
> > > <variable name="lon">
> > > <attribute name="standard_name" type="string" value="longitude"/>
> > > <attribute name="axis" type="string" value="X"/>
> > > <attribute name="units" type="string" value="degrees_east"/>
> > > </variable>
> > > <variable name="time">
> > > <attribute name="standard_name" type="string" value="time"/>
> > > <attribute name="axis" type="string" value="T"/>
> > > <attribute name="units" type="string" value="seconds since 1970-01-01
> > > 00:00:00"/>
> > > </variable>
> > > <aggregation dimName="time" type="joinExisting">
> > > <netcdf id="1" location="
> > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070530_20070906.nc
> > > "/>
> > > <netcdf id="2" location="
> > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070906_20071107.nc
> > > "/>
> > > <netcdf id="3" location="
> > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20080807_20081204.nc
> > > "/>
> > > <netcdf id="4" location="
> > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20100803_20101117.nc
> > > "/>
> > > <netcdf id="5" location="
> > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20101117_20110318.nc
> > > "/>
> > > </aggregation>
> > > </netcdf>
> > >
> > > Is there a reason the joinExisting would work, but not the joinNew? I
> > > thought joinNew would be able to aggregate anything, since your creating
> > a
> > > new variable to aggregate them with.
> > >
> > > I apologize if this is something simple that I am just missing, I am just
> > > starting to get my bearings with this.
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> > >
> > > address@hidden> wrote:
> > >
> > > > Lansing,
> > > >
> > > > Below are a few of the files I am trying to aggregate.
> > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070530_20070906.nc.html
> > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070906_20071107.nc.html
> > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20080807_20081204.nc.html
> > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20100803_20101117.nc.html
> > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20101117_20110318.nc.html
> > > >
> > > > Here is the ncml for the joinNew I've been fiddling around with:
> > > >
> > > > <?xml version="1.0" encoding="UTF-8"?>
> > > > <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2
> > ">
> > > >  <!-- <variable name="time" type="int">
> > > > <attribute name="units" type="string" value="seconds since 1970-01-01
> > > > 00:00:00"/>
> > > >  <attribute name="_CoordinateAxisType" value="Time" />
> > > >   </variable>-->
> > > >   <aggregation dimName="timeagg" type="joinNew">
> > > >   <variableAgg name="A"/>
> > > > <netcdf id="1" location="
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070530_20070906.nc
> > > > "/>
> > > > <netcdf id="2" location="
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070906_20071107.nc
> > > > "/>
> > > > <netcdf id="3" location="
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20080807_20081204.nc
> > > > "/>
> > > > <netcdf id="4" location="
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20100803_20101117.nc
> > > > "/>
> > > > <netcdf id="5" location="
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20101117_20110318.nc
> > > > "/>
> > > >   </aggregation>
> > > >  </netcdf>
> > > >
> > > > For some reason the Aggregation Variables are not being listed... Am I
> > > > completely missing something?
> > > > I truly appreciate your assistance in this matter. :)
> > > >
> > > > Thanks,
> > > >
> > > > Matt
> > > >
> > > >
> > > > address@hidden> wrote:
> > > >
> > > >> Matt,
> > > >>
> > > >> If you send me a few of the files, I can set up the aggregation and
> > see
> > > >> if there are any pitfalls in the data.  It's not always
> > straightforward.
> > > >>
> > > >> -Lansing
> > > >>
> > > >> > Lansing,
> > > >> >
> > > >> > The times "should be" unified, but I should look into that.
> > Probably, to
> > > >> > avoid complications a JoinNew would suffice in this case.
> > > >> >
> > > >> > Matt
> > > >> >
> > > >> >
> > > >> > address@hidden> wrote:
> > > >> >
> > > >> > > Li,
> > > >> > >
> > > >> > > Are the times in the data sets unified (i.e., all the same), or
> > are
> > > >> they
> > > >> > > interlaced?  In other words, if you were to set up a single time
> > axis
> > > >> for
> > > >> > > all of the buoys, would the data line up on the axis, or would it
> > be
> > > >> > > scattered around?  If the times are all the same, then you should
> > be
> > > >> able
> > > >> > > to do a simple Union as described here:
> > > >> > >
> > > >> > >
> > > >>
> > http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html
> > > >> > >
> > > >> > > Otherwise, you will need to to a JoinNew, wherein you declare a
> > new
> > > >> > > dimension as the aggregation dimension.  This is described on the
> > > >> same page.
> > > >> > >
> > > >> > > -Lansing
> > > >> > >
> > > >> > > > Hi Lansing,
> > > >> > > >
> > > >> > > > I think there are group of buoys that are reporting data
> > > >> simultaneously.
> > > >> > > > The data sets should be overlapping in space.
> > > >> > > > I included Matt, who is the data officer for this data set, in
> > our
> > > >> > > > conversation.
> > > >> > > >
> > > >> > > > Hi Matt,
> > > >> > > > Please correct me if the description about the data is not
> > right.
> > > >> > > >
> > > >> > > > Thanks a lot,
> > > >> > > > Li
> > > >> > > >
> > > >> > > >
> > > >> > > > address@hidden> wrote:
> > > >> > > >
> > > >> > > > > Li,
> > > >> > > > >
> > > >> > > > > Are the data sets from the buoys overlapping in time, space,
> > or
> > > >> both?
> > > >> > > > >  That is, do you have many buoys that are reporting data
> > > >> > > simultaneously, or
> > > >> > > > > are the individual data set series generated by each buoy
> > > >> temporally
> > > >> > > > > distinct?
> > > >> > > > >
> > > >> > > > > Also, are the nc files generated by the buoys, or have they
> > been
> > > >> > > generated
> > > >> > > > > through some post-processing from a raw data set?
> > > >> > > > >
> > > >> > > > > If the files are not too large, feel free to send me a few
> > > >> > > representative
> > > >> > > > > files to work with locally, and I will try to set up an
> > > >> aggregation.
> > > >> > > > >
> > > >> > > > > Regards,
> > > >> > > > >   Lansing Madry
> > > >> > > > >   Unidata
> > > >> > > > >   Boulder, Colorado
> > > >> > > > >
> > > >> > > > > > Dear Sir,
> > > >> > > > > >
> > > >> > > > > > I am trying to reach the experts about NcML aggregation.
> > > >> > > > > >
> > > >> > > > > > I have a group of time series buoy nc files. I tried to
> > > >> aggregate
> > > >> > > them
> > > >> > > > > > by time, but failed since each of them holds a time series
> > > >> itself.
> > > >> > > > > >
> > > >> > > > > > It would be great if anyone could advise me on how should I
> > > >> aggregate
> > > >> > > > > > these data.
> > > >> > > > > >
> > > >> > > > > > Thanks and Regards,
> > > >> > > > > > Li
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Ticket Details
> > > >> > > > > ===================
> > > >> > > > > Ticket ID: HIP-291542
> > > >> > > > > Department: Support netCDF Java
> > > >> > > > > Priority: Normal
> > > >> > > > > Status: Open
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > > Ticket Details
> > > >> > > ===================
> > > >> > > Ticket ID: HIP-291542
> > > >> > > Department: Support netCDF Java
> > > >> > > Priority: Normal
> > > >> > > Status: Open
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Mathew Biddle, Oceanographer
> > > >> > NOAA/NODC UMD/ESSIC/CICS E/OC1
> > > >> > 1315 East-West Hwy
> > > >> > Silver Spring, MD 20910-3282
> > > >> > Phone: (301) 713-3272 X163
> > > >> > Email: address@hidden
> > > >> > http://www.nodc.noaa.gov/
> > > >> > http://www.facebook.com/noaa.nodc
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >> Ticket Details
> > > >> ===================
> > > >> Ticket ID: HIP-291542
> > > >> Department: Support netCDF Java
> > > >> Priority: Normal
> > > >> Status: Open
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > Mathew Biddle, Oceanographer
> > > > NOAA/NODC UMD/ESSIC/CICS E/OC1
> > > > 1315 East-West Hwy
> > > > Silver Spring, MD 20910-3282
> > > > Phone: (301) 713-3272 X163
> > > > Email: address@hidden
> > > > http://www.nodc.noaa.gov/
> > > > http://www.facebook.com/noaa.nodc
> > > >
> > >
> > >
> > >
> > > --
> > > Mathew Biddle, Oceanographer
> > > NOAA/NODC UMD/ESSIC/CICS E/OC1
> > > 1315 East-West Hwy
> > > Silver Spring, MD 20910-3282
> > > Phone: (301) 713-3272 X163
> > > Email: address@hidden
> > > http://www.nodc.noaa.gov/
> > > http://www.facebook.com/noaa.nodc
> > >
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: HIP-291542
> > Department: Support netCDF Java
> > Priority: Normal
> > Status: Open
> >
> >
> 
> 


Ticket Details
===================
Ticket ID: HIP-291542
Department: Support netCDF Java
Priority: Normal
Status: Open