[netCDFJava #HIP-291542]: Question on NcML aggregation

Li,

Unfortunately, you probably won't be able to aggregate across buoys at this 
point in time.  A feature collection such as what you are describing is on our 
wish list, but isn't yet implemented in a reliable fashion.  The NcML 
aggregations would fail because of holes in the data, as aggregations really 
need homogenous data sets.

One thing I am doing, though, is pulling down a bunch of your buoy data files 
to use as a test set.  I'd guess this is a "sometime in the next year" horizon 
project.

Regards,
  Lansing

> Hi Lansing,
> 
> It would be great if we could set up different buoys in one NcML
> aggregation.
> 
> Hi Matt,
> Do you have idea about how many buoys and station names are there from all
> the files?
> 
> 
> Thanks,
> Li
> 
> 
> 
> 
> 
> 
> *
> 
> 
> *
> address@hidden> wrote:
> 
> > Li,
> >
> > After looking at the files you are serving up, I think joinExisting is the
> > way to go.  You'll need to set up different NcML aggregations for the
> > different buoys (DUO and SEF, it looks like).  Is there some other
> > information you thought users would be selecting?  I'm not entirely sure
> > what you mean in your comments below on joinExisting.
> >
> > -Lansing
> >
> > > Thanks Lansing and Matt!
> > >
> > > I put the three ways of aggregated files under the test THREDDS server.
> > > Here are few comments.
> > > http://data.nodc.noaa.gov/thredds/catalog/testdata/20130328/catalog.html
> > >
> > > joinNew: There is "timeagg" in the OPeNDAP selection, but the actual
> > > temperature values are extracted from the first timeagg, no matter which
> > > timeagg we choose.
> > >
> > > joinExisting: this one picks up all the values for all the joined files.
> > > Though there is no list of station_name, plfatform1, instrument, for the
> > > users to select.
> > >
> > > Union: this one picks up only the first file in the aggregation, and
> > there
> > > is no way to select the files after.
> > >
> > > The joinExisting is the way to go.
> > >
> > > Thanks,
> > > Li
> > >
> > >
> > >
> > >
> > >
> > > address@hidden> wrote:
> > >
> > > > Matt,
> > > >
> > > > I set up an aggregation using four of the files.  At it's simplest,
> > this
> > > > works to join the datasets along the existing time dimension:
> > > >
> > > > <?xml version="1.0" encoding="UTF-8"?>
> > > > <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2
> > ">
> > > >   <aggregation dimName="time" type="joinExisting">
> > > >     <netcdf id="1" location="DOU001_000_20070530_20070906.nc"/>
> > > >     <netcdf id="2" location="DOU001_000_20070906_20071107.nc"/>
> > > >     <netcdf id="3" location="DOU001_000_20080807_20081204.nc"/>
> > > >     <netcdf id="4" location="DOU001_000_20100803_20101117.nc"/>
> > > >   </aggregation>
> > > > </netcdf>
> > > >
> > > > Since you already have a good dimension for aggregating (time), then
> > you
> > > > probably don't need to worry about doing a joinNew.  You would use
> > that if
> > > > there were no time data in the files, and you wanted to add it into the
> > > > dataset you were serving, for instance.
> > > >
> > > > Let me know if there is something else you are trying to do, I'm happy
> > to
> > > > help.
> > > >
> > > > -Lansing
> > > >
> > > > > Lansing,
> > > > >
> > > > > When I set it up using the "joinExisting" on the time dimension it
> > seems
> > > > to
> > > > > aggregate them. Below is the ncml:
> > > > > <?xml version="1.0" encoding="UTF-8"?>
> > > > > <netcdf xmlns="
> > http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> > > > > <variable name="lat">
> > > > > <attribute name="standard_name" type="String" value="latitude"/>
> > > > > <attribute name="axis" type="string" value="Y"/>
> > > > > <attribute name="units" type="string" value="degrees_north"/>
> > > > > </variable>
> > > > > <variable name="lon">
> > > > > <attribute name="standard_name" type="string" value="longitude"/>
> > > > > <attribute name="axis" type="string" value="X"/>
> > > > > <attribute name="units" type="string" value="degrees_east"/>
> > > > > </variable>
> > > > > <variable name="time">
> > > > > <attribute name="standard_name" type="string" value="time"/>
> > > > > <attribute name="axis" type="string" value="T"/>
> > > > > <attribute name="units" type="string" value="seconds since 1970-01-01
> > > > > 00:00:00"/>
> > > > > </variable>
> > > > > <aggregation dimName="time" type="joinExisting">
> > > > > <netcdf id="1" location="
> > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070530_20070906.nc
> > > > > "/>
> > > > > <netcdf id="2" location="
> > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070906_20071107.nc
> > > > > "/>
> > > > > <netcdf id="3" location="
> > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20080807_20081204.nc
> > > > > "/>
> > > > > <netcdf id="4" location="
> > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20100803_20101117.nc
> > > > > "/>
> > > > > <netcdf id="5" location="
> > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20101117_20110318.nc
> > > > > "/>
> > > > > </aggregation>
> > > > > </netcdf>
> > > > >
> > > > > Is there a reason the joinExisting would work, but not the joinNew? I
> > > > > thought joinNew would be able to aggregate anything, since your
> > creating
> > > > a
> > > > > new variable to aggregate them with.
> > > > >
> > > > > I apologize if this is something simple that I am just missing, I am
> > just
> > > > > starting to get my bearings with this.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Matt
> > > > >
> > > > >
> > > > > address@hidden> wrote:
> > > > >
> > > > > > Lansing,
> > > > > >
> > > > > > Below are a few of the files I am trying to aggregate.
> > > > > >
> > > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070530_20070906.nc.html
> > > > > >
> > > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070906_20071107.nc.html
> > > > > >
> > > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20080807_20081204.nc.html
> > > > > >
> > > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20100803_20101117.nc.html
> > > > > >
> > > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20101117_20110318.nc.html
> > > > > >
> > > > > > Here is the ncml for the joinNew I've been fiddling around with:
> > > > > >
> > > > > > <?xml version="1.0" encoding="UTF-8"?>
> > > > > > <netcdf xmlns="
> > http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2
> > > > ">
> > > > > >  <!-- <variable name="time" type="int">
> > > > > > <attribute name="units" type="string" value="seconds since
> > 1970-01-01
> > > > > > 00:00:00"/>
> > > > > >  <attribute name="_CoordinateAxisType" value="Time" />
> > > > > >   </variable>-->
> > > > > >   <aggregation dimName="timeagg" type="joinNew">
> > > > > >   <variableAgg name="A"/>
> > > > > > <netcdf id="1" location="
> > > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070530_20070906.nc
> > > > > > "/>
> > > > > > <netcdf id="2" location="
> > > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070906_20071107.nc
> > > > > > "/>
> > > > > > <netcdf id="3" location="
> > > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20080807_20081204.nc
> > > > > > "/>
> > > > > > <netcdf id="4" location="
> > > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20100803_20101117.nc
> > > > > > "/>
> > > > > > <netcdf id="5" location="
> > > > > >
> > > >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20101117_20110318.nc
> > > > > > "/>
> > > > > >   </aggregation>
> > > > > >  </netcdf>
> > > > > >
> > > > > > For some reason the Aggregation Variables are not being listed...
> > Am I
> > > > > > completely missing something?
> > > > > > I truly appreciate your assistance in this matter. :)
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Matt
> > > > > >
> > > > > >
> > > > > > address@hidden> wrote:
> > > > > >
> > > > > >> Matt,
> > > > > >>
> > > > > >> If you send me a few of the files, I can set up the aggregation
> > and
> > > > see
> > > > > >> if there are any pitfalls in the data.  It's not always
> > > > straightforward.
> > > > > >>
> > > > > >> -Lansing
> > > > > >>
> > > > > >> > Lansing,
> > > > > >> >
> > > > > >> > The times "should be" unified, but I should look into that.
> > > > Probably, to
> > > > > >> > avoid complications a JoinNew would suffice in this case.
> > > > > >> >
> > > > > >> > Matt
> > > > > >> >
> > > > > >> >
> > > > > >> > address@hidden> wrote:
> > > > > >> >
> > > > > >> > > Li,
> > > > > >> > >
> > > > > >> > > Are the times in the data sets unified (i.e., all the same),
> > or
> > > > are
> > > > > >> they
> > > > > >> > > interlaced?  In other words, if you were to set up a single
> > time
> > > > axis
> > > > > >> for
> > > > > >> > > all of the buoys, would the data line up on the axis, or
> > would it
> > > > be
> > > > > >> > > scattered around?  If the times are all the same, then you
> > should
> > > > be
> > > > > >> able
> > > > > >> > > to do a simple Union as described here:
> > > > > >> > >
> > > > > >> > >
> > > > > >>
> > > > http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html
> > > > > >> > >
> > > > > >> > > Otherwise, you will need to to a JoinNew, wherein you declare
> > a
> > > > new
> > > > > >> > > dimension as the aggregation dimension.  This is described on
> > the
> > > > > >> same page.
> > > > > >> > >
> > > > > >> > > -Lansing
> > > > > >> > >
> > > > > >> > > > Hi Lansing,
> > > > > >> > > >
> > > > > >> > > > I think there are group of buoys that are reporting data
> > > > > >> simultaneously.
> > > > > >> > > > The data sets should be overlapping in space.
> > > > > >> > > > I included Matt, who is the data officer for this data set,
> > in
> > > > our
> > > > > >> > > > conversation.
> > > > > >> > > >
> > > > > >> > > > Hi Matt,
> > > > > >> > > > Please correct me if the description about the data is not
> > > > right.
> > > > > >> > > >
> > > > > >> > > > Thanks a lot,
> > > > > >> > > > Li
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Fri, Mar 29, 2013 at 10:40 AM, Unidata netCDF Java
> > > > > >> > > > address@hidden> wrote:
> > > > > >> > > >
> > > > > >> > > > > Li,
> > > > > >> > > > >
> > > > > >> > > > > Are the data sets from the buoys overlapping in time,
> > space,
> > > > or
> > > > > >> both?
> > > > > >> > > > >  That is, do you have many buoys that are reporting data
> > > > > >> > > simultaneously, or
> > > > > >> > > > > are the individual data set series generated by each buoy
> > > > > >> temporally
> > > > > >> > > > > distinct?
> > > > > >> > > > >
> > > > > >> > > > > Also, are the nc files generated by the buoys, or have
> > they
> > > > been
> > > > > >> > > generated
> > > > > >> > > > > through some post-processing from a raw data set?
> > > > > >> > > > >
> > > > > >> > > > > If the files are not too large, feel free to send me a few
> > > > > >> > > representative
> > > > > >> > > > > files to work with locally, and I will try to set up an
> > > > > >> aggregation.
> > > > > >> > > > >
> > > > > >> > > > > Regards,
> > > > > >> > > > >   Lansing Madry
> > > > > >> > > > >   Unidata
> > > > > >> > > > >   Boulder, Colorado
> > > > > >> > > > >
> > > > > >> > > > > > Dear Sir,
> > > > > >> > > > > >
> > > > > >> > > > > > I am trying to reach the experts about NcML aggregation.
> > > > > >> > > > > >
> > > > > >> > > > > > I have a group of time series buoy nc files. I tried to
> > > > > >> aggregate
> > > > > >> > > them
> > > > > >> > > > > > by time, but failed since each of them holds a time
> > series
> > > > > >> itself.
> > > > > >> > > > > >
> > > > > >> > > > > > It would be great if anyone could advise me on how
> > should I
> > > > > >> aggregate
> > > > > >> > > > > > these data.
> > > > > >> > > > > >
> > > > > >> > > > > > Thanks and Regards,
> > > > > >> > > > > > Li
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > Ticket Details
> > > > > >> > > > > ===================
> > > > > >> > > > > Ticket ID: HIP-291542
> > > > > >> > > > > Department: Support netCDF Java
> > > > > >> > > > > Priority: Normal
> > > > > >> > > > > Status: Open
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Ticket Details
> > > > > >> > > ===================
> > > > > >> > > Ticket ID: HIP-291542
> > > > > >> > > Department: Support netCDF Java
> > > > > >> > > Priority: Normal
> > > > > >> > > Status: Open
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Mathew Biddle, Oceanographer
> > > > > >> > NOAA/NODC UMD/ESSIC/CICS E/OC1
> > > > > >> > 1315 East-West Hwy
> > > > > >> > Silver Spring, MD 20910-3282
> > > > > >> > Phone: (301) 713-3272 X163
> > > > > >> > Email: address@hidden
> > > > > >> > http://www.nodc.noaa.gov/
> > > > > >> > http://www.facebook.com/noaa.nodc
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >> Ticket Details
> > > > > >> ===================
> > > > > >> Ticket ID: HIP-291542
> > > > > >> Department: Support netCDF Java
> > > > > >> Priority: Normal
> > > > > >> Status: Open
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Mathew Biddle, Oceanographer
> > > > > > NOAA/NODC UMD/ESSIC/CICS E/OC1
> > > > > > 1315 East-West Hwy
> > > > > > Silver Spring, MD 20910-3282
> > > > > > Phone: (301) 713-3272 X163
> > > > > > Email: address@hidden
> > > > > > http://www.nodc.noaa.gov/
> > > > > > http://www.facebook.com/noaa.nodc
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Mathew Biddle, Oceanographer
> > > > > NOAA/NODC UMD/ESSIC/CICS E/OC1
> > > > > 1315 East-West Hwy
> > > > > Silver Spring, MD 20910-3282
> > > > > Phone: (301) 713-3272 X163
> > > > > Email: address@hidden
> > > > > http://www.nodc.noaa.gov/
> > > > > http://www.facebook.com/noaa.nodc
> > > > >
> > > > >
> > > >
> > > >
> > > > Ticket Details
> > > > ===================
> > > > Ticket ID: HIP-291542
> > > > Department: Support netCDF Java
> > > > Priority: Normal
> > > > Status: Open
> > > >
> > > >
> > >
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: HIP-291542
> > Department: Support netCDF Java
> > Priority: Normal
> > Status: Open
> >
> >
> 
> 


Ticket Details
===================
Ticket ID: HIP-291542
Department: Support netCDF Java
Priority: Normal
Status: Open