[IDV #VFL-607764]: Problem getting time dimension specified for loading data using 'Directory' option and looping over time

Hi Dave,

That's good news!

So, NcML is a feature of netCDF-Java, and can be very helpful in getting date
into the IDV without the need of re-writing grids. NcML allows users to make
minor modifications (like renaming variables, adding attributes, etc.) in a
relatively easy way. Check out the following page for more details:

http://www.unidata.ucar.edu/software/netcdf/ncml/

The IDV uses NcML behinds the scenes to aggregate grids by time,
but in a limited way as you've found out. The NcML I sent you looks
at all of the files ending with .nc in the directory you pointed the
scan element at, renames the Time dimension to time, and then
aggregates over the newly renamed dimension "time".

I'm glad things worked out - let me know if you run into anything
else!

Cheers,

Sean

> OK, problem solved. Re-launching IDV seems to have done the trick and I
> can now load all files at once. I am unfamiliar with the use of the
> .ncml file so I guess I need to get up to speed on that. I assume its in
> the documentation somewhere?
> 
> Otherwise, it seems that this problem is solved and I really appreciate
> your help.
> 
> -dave
> 
> On 2013-06-27 11:00, Unidata IDV Support wrote:
> > Hi Dave,
> >
> > The NcML file can be loaded through the feeling lucky option, so
> > that's ok.
> >
> > I'm not sure why it's only loading one grid - I am able to see all
> > three
> > on my machine. Where are you located? Maybe I could drop by and see
> > what's up.
> >
> > One quick test would be to try loading the ncml file from a fresh
> > start
> > of the IDV, in case there are any cache issues at play.
> >
> > Cheers,
> >
> > Sean
> >
> >> Hi Sean,
> >>
> >> Thanks for the quick feedback.  I used the 'feeling lucky' option as
> >> I'm not sure which option the .ncml file is supposed to be used
> >> with.
> >> My directory structure is as follows:
> >>
> >>
> >> saudi-c1-int1.rap.ucar.edu:/var/autofs/mnt/qnap-25/NDHMS/cases/Col_Headwat/tmp_data/test/test%
> >> ls -la
> >> total 3936
> >> drwxrwxr-x 2 gochis rap    4096 Jun 27  2013 .
> >> drwxrwxr-x 3 gochis rap    4096 Jun 27  2013 ..
> >> -rw-rw-r-- 1 gochis rap 1338516 Jun 26 17:11
> >> 2008052000.LDASOUT_DOMAIN1.nc.nc
> >> -rw-rw-r-- 1 gochis rap 1338344 Jun 26 17:11
> >> 2008052100.LDASOUT_DOMAIN1.nc.nc
> >> -rw-rw-r-- 1 gochis rap 1338520 Jun 26 17:11
> >> 2008052200.LDASOUT_DOMAIN1.nc.nc
> >> -rw-r--r-- 1 gochis rap     322 Jun 27  2013 LDASOUT_DOMAIN1.ncml
> >>
> >> saudi-c1-int1.rap.ucar.edu:/var/autofs/mnt/qnap-25/NDHMS/cases/Col_Headwat/tmp_data/test/test%
> >>
> >>
> >> So, I edited the .ncml file to be the following to point to this
> >> directory:
> >>
> >>
> >> saudi-c1-int1.rap.ucar.edu:/var/autofs/mnt/qnap-25/NDHMS/cases/Col_Headwat/tmp_data/test/test%
> >> cat LDASOUT_DOMAIN1.ncml
> >> <?xml version='1.0' encoding='UTF-8'?>
> >> <netcdf
> >> xmlns='http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2'>
> >> <aggregation dimName='time' type='joinExisting'>
> >> <dimension name="time" orgName="Time" />
> >> <scan
> >> location='/qnap-25/NDHMS/cases/Col_Headwat/tmp_data/test/test'
> >> suffix='.nc' />
> >> </aggregation>
> >> </netcdf>
> >>
> >> saudi-c1-int1.rap.ucar.edu:/var/autofs/mnt/qnap-25/NDHMS/cases/Col_Headwat/tmp_data/test/test%
> >>
> >>
> >>
> >> Lastly, if it helps I have put the 3 input files on my ftp site at:
> >> ftp.rap.ucar.edu/incoming/irap/gochis/
> >>
> >>
> >> Thanks again for the help!
> >> -Dave
> >>
> >>
> >>
> >>
> >>
> >> On 2013-06-27 10:38, Unidata IDV Support wrote:
> >> > Hi Dave,
> >> >
> >> > There should't be anything else if all of the files live in the
> >> top
> >> > level
> >> > of the directory identified in the scan tag. How is your directory
> >> > structure
> >> > laid out?
> >> >
> >> > Also, when you load the ncml file, what Data Source Type are you
> >> > using
> >> > (the drop-down box on the Data Choosers Tab)?
> >> >
> >> > Sean
> >> >
> >> >> Hi Sean,
> >> >>
> >> >> OK, I did what you suggested and one file loaded. Is there an
> >> >> additional wildcard or something we need to add to the .ncml file
> >> to
> >> >> get
> >> >> all files to load?
> >> >>
> >> >> -Dave
> >> >>
> >> >> On 2013-06-27 10:24, Unidata IDV Support wrote:
> >> >> > Greetings David,
> >> >> >
> >> >> > The IDV aggregation code is a little...shall we say, picky.
> >> >> > Currently, the time dimension must
> >> >> > be named "time" for this feature to work (case sensitive).
> >> >> >
> >> >> > In order to aggregate your files, try using the attached ncml
> >> >> file.
> >> >> >
> >> >> > If you open the NcML file, you will see on the 5th line a tag
> >> like
> >> >> > this:
> >> >> >
> >> >> > scan location='LDASOUT' suffix='.nc'
> >> >> >
> >> >> > Change the location to the directory on disk where your data
> >> are
> >> >> > stored. If you
> >> >> > then load the ncml file into the idv through the Data Choosers
> >> ->
> >> >> > Files, then
> >> >> > the aggregation should work.
> >> >> >
> >> >> > Please let me know how it turns out.
> >> >> >
> >> >> > Cheers,
> >> >> >
> >> >> > Sean
> >> >> >
> >> >> >> Full Name: David Gochis
> >> >> >> Email Address: address@hidden
> >> >> >> Organization: NCAR/RAL
> >> >> >> Package Version: 4.0u1 build date:2013-03-29 18:10 UTC
> >> >> >> Operating System: Linux
> >> >> >> Hardware: Java: home:
> >> >> >> /var/autofs/mnt/linux-amd64/debian/squeeze/local/IDV_4.0u1/jre
> >> >> >> version: 1.6.0_41 j3d:1.5.2 fcs (build4)
> >> >> >> Description of problem: There was an error loading the data:
> >> >> >> Error creating data source:file.grid with:
> >> >> >>
> >> >>
> >> /qnap-25/NDHMS/cases/Col_Headwat/tmp_data/test/2008052100.LDASOUT_DOMAIN1.nc.nc
> >> >> >> Grid data source failed making data set:
> >> >> >>
> >> >>
> >> /home/gochis/.unidata/idv/DefaultIdv/tmp/multigrid_3a11d392-abc3-4ef0-ac8c-25c0394b1b910.ncml
> >> >> >> Dimension not found= time
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> Hi there,
> >> >> >> I am attempting to load several files at once which are
> >> sequenced
> >> >> in
> >> >> >> time. I've been trying to add the appropriate time variables
> >> to
> >> >> these
> >> >> >> files using nco commands. I have been mimicking other file
> >> >> formats
> >> >> >> that seem to work ok but these files are not working. I get
> >> the
> >> >> error
> >> >> >> message above upon clicking the 'Add Source' button in  Data
> >> >> Chooser
> >> >> >> -> Directory option.  Can you advise on what is missing/wrong
> >> >> with the
> >> >> >> attached file?  The attached file is a single time (the first
> >> >> time) of
> >> >> >> output. The other files simply increment the 'time' variable
> >> by
> >> >> the
> >> >> >> seconds since the start of the run (e.g. 3600, 7200, etc).
> >> >> >>
> >> >> >> Thank you,
> >> >> >> Dave Gochis
> >> >> >>
> >> >> >> ******************
> >> >> >> Stack trace:
> >> >> >> java.lang.IllegalArgumentException: Dimension not found= time
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.nc2.ncml.AggregationOuterDimension$DatasetOuterDimension.getNcoords(AggregationOuterDimension.java:656)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.nc2.ncml.AggregationOuterDimension$DatasetOuterDimension.setStartEnd(AggregationOuterDimension.java:675)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.nc2.ncml.AggregationOuterDimension.buildCoords(AggregationOuterDimension.java:161)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.nc2.ncml.AggregationExisting.buildNetcdfDataset(AggregationExisting.java:97)
> >> >> >> at ucar.nc2.ncml.Aggregation.finish(Aggregation.java:404)
> >> >> >> at ucar.nc2.ncml.NcMLReader.readNetcdf(NcMLReader.java:492)
> >> >> >> at ucar.nc2.ncml.NcMLReader.readNcML(NcMLReader.java:437)
> >> >> >> at ucar.nc2.ncml.NcMLReader.readNcML(NcMLReader.java:252)
> >> >> >> at ucar.nc2.ncml.NcMLReader.readNcML(NcMLReader.java:202)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.nc2.dataset.NetcdfDataset.acquireNcml(NetcdfDataset.java:841)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:680)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:418)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.nc2.dataset.NetcdfDataset.acquireDataset(NetcdfDataset.java:513)
> >> >> >> at ucar.nc2.dt.grid.GridDataset.open(GridDataset.java:104)
> >> >> >> at ucar.nc2.dt.grid.GridDataset.open(GridDataset.java:90)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.data.grid.GeoGridDataSource.doMakeDataSet(GeoGridDataSource.java:1113)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.data.grid.GeoGridDataSource.getDataset(GeoGridDataSource.java:1135)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.data.grid.GeoGridDataSource.doMakeDataChoices(GeoGridDataSource.java:1162)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.data.DataSourceImpl.getDataChoices(DataSourceImpl.java:1852)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.data.FilesDataSource.sourcesChanged(FilesDataSource.java:418)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.data.grid.GeoGridDataSource.sourcesChanged(GeoGridDataSource.java:591)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.data.FilesDataSource.initWithPollingInfo(FilesDataSource.java:372)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.data.FilesDataSource.initAfterCreation(FilesDataSource.java:283)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.data.grid.GeoGridDataSource.initAfterCreation(GeoGridDataSource.java:496)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.data.DataManager.createDataSource(DataManager.java:1597)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.idv.IntegratedDataViewer.createDataSource(IntegratedDataViewer.java:1980)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1897)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1831)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1804)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.idv.chooser.IdvChooser.makeDataSource(IdvChooser.java:943)
> >> >> >> at
> >> >> >>
> >> >>
> >> ucar.unidata.idv.chooser.PollingFileChooser.doLoadInThread(PollingFileChooser.java:192)
> >> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> >> >> >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> >> >> Source)
> >> >> >> at java.lang.reflect.Method.invoke(Unknown Source)
> >> >> >> at ucar.unidata.util.Misc$2.run(Misc.java:1221)
> >> >> >> at ucar.unidata.util.Misc$3.run(Misc.java:1249)
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >
> >> >> > Ticket Details
> >> >> > ===================
> >> >> > Ticket ID: VFL-607764
> >> >> > Department: Support IDV
> >> >> > Priority: Normal
> >> >> > Status: Open
> >> >>
> >> >>
> >> >
> >> >
> >> > Ticket Details
> >> > ===================
> >> > Ticket ID: VFL-607764
> >> > Department: Support IDV
> >> > Priority: Normal
> >> > Status: Open
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: VFL-607764
> > Department: Support IDV
> > Priority: Normal
> > Status: Open
> 
> 

Ticket Details
===================
Ticket ID: VFL-607764
Department: Support IDV
Priority: Normal
Status: Open