[IDV #SJQ-840722]: .idx files

Hi Rob,

The ctl and idx files are metadata files that describe what data are available 
and how its stored in the actual grib data file. The .ctl file is a GrADS 
Control File (or Data Descriptor File) and describes what data are available in 
the grib file, where the actual datafile (GRIB -> .grb) is located on your 
system, the 'shape' or dimensions of the variables in the file, and a bit of 
GrADS specific information. The .idx file is a GRIB index, or Gridmap, file. 
The .idx file relates the variables described in the .ctl file the actual GRIB 
messages ('chunks' of data associated with the same variable - they don't 
necessarily live right next to each other inside the GRIB file). I'm not sure 
what information you would want to visualize from these metadata files.

In a simplified sense, the .ctl files says "this is what we have" and the .idx 
file says "this is how it's stored on the disk." GrADS uses this information to 
read the actual data from the GRIB file.

The IDV is capable of reading GRIB files straight up, without the need for .ctl 
or .idx files, although if you analyze data in GrADS first and save the 
results, then I think you need the .ctl file to tell the IDV what is in the new 
GRIB file. I was able to download the GRIB files from the link you sent and 
open them in the IDV without issue. Have you tried just using the .grb files? I 
see that for the example you sent (RUC data from 2006-05-03, there aren't any 
actual data files there, which would mean NOMADS does not have the data you 
need. You may want to contact them to see why the data are missing.

Cheers,

Sean

> Yes, that is the method I have been using but there is a big gap in missing 
> data between 2002 and 2009 which is why I have been ordering my files from 
> the nomads site. I'm just confused of what an actual .ctl or idx file 
> actually plot and what they mean. Are they the same thing as the .grb file? I 
> really need to plot the .ctl file somehow.
> 
> Thanks,
> Rob
> On Apr 19, 2012, at 11:54 AM, Unidata IDV Support wrote:
> 
> > Greetings Rob,
> >
> > It looks like you can directly load in the grib data without using the ctl 
> > or idx files. That said, I would recommend that you access the data via the 
> > NOMADS THREDDS server. Simply use this URL in the "Data Choosers -> 
> > Catalogs" screen in the IDV Dashboard:
> >
> > http://nomads.ncdc.noaa.gov/thredds/catalog.xml
> >
> > In particular, I think you are looking for the following area in their 
> > THREDDS archive:
> >
> > http://nomads.ncdc.noaa.gov/thredds/catalog/ruc/catalog.xml
> >
> > Let me know if this gets you what you need :-)
> >
> > Cheers!
> >
> > Sean
> >
> >> Not sure if the URL is public or just confined to me. I know its offline 
> >> data so I'm not sure. The URL is:
> >>
> >> http://ftp3.ncdc.noaa.gov/pub/download/nomads/model-4299/ruc2_252_20060503_1300_fff.ctl
> >>
> >> Any suggestions?
> >>
> >> Thanks,
> >> Rob
> >> On Apr 18, 2012, at 5:28 PM, Unidata IDV Support wrote:
> >>
> >>> Hi Rob,
> >>>
> >>> What is the URL? If it's publicly accessible data, I can take a look at 
> >>> the file here.
> >>>
> >>> Cheers!
> >>>
> >>> Sean
> >>>
> >>>> Just an update. I updated the IDV to version 3 and now see the option 
> >>>> under the data type for the grADS binary files but I'm still unsure how 
> >>>> to open it. I tired entering the URL but it didnt work either.
> >>>> On Apr 18, 2012, at 10:16 AM, Unidata IDV Support wrote:
> >>>>
> >>>>> Greetings Rob,
> >>>>>
> >>>>> Are you trying to read GrADS data? If so, the .idx file isn't needed. 
> >>>>> You should be able to load the .ctl file into the IDV using the "GrADS 
> >>>>> Binary Grid Files" Data Source Type in the data chooser.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Sean
> >>>>>
> >>>>>> Full Name: Robert D'Arienzo
> >>>>>> Email Address: address@hidden
> >>>>>> Organization: Rutgers University
> >>>>>> Package Version: 2.9u3 build date:2011-01-14 00:05 UTC
> >>>>>> Operating System: Mac OS X
> >>>>>> Hardware: Java: home: 
> >>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home 
> >>>>>> version: 1.6.0_29 j3d:1.5.2 fcs (build4)
> >>>>>> Description of problem: There was an error loading the data:
> >>>>>> Error creating data source:file.profiler2 with: 
> >>>>>> /Users/hockeymaster475/Downloads/ruc2_252_20060916_1600_fff.idx
> >>>>>> Cant read 
> >>>>>> /Users/hockeymaster475/Downloads/ruc2_252_20060916_1600_fff.idx: not a 
> >>>>>> valid CDM file.
> >>>>>>
> >>>>>> How would I open an idx file in the IDV? Thanks!
> >>>>>>
> >>>>>>
> >>>>>> -Rob
> >>>>>>
> >>>>>> ******************
> >>>>>> Stack trace:
> >>>>>> java.io.IOException: Cant read 
> >>>>>> /Users/hockeymaster475/Downloads/ruc2_252_20060916_1600_fff.idx: not a 
> >>>>>> valid CDM file.
> >>>>>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:737)
> >>>>>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:387)
> >>>>>> at 
> >>>>>> ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:686)
> >>>>>> at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:425)
> >>>>>> at 
> >>>>>> ucar.nc2.dataset.NetcdfDataset.acquireDataset(NetcdfDataset.java:521)
> >>>>>> at 
> >>>>>> ucar.nc2.dataset.NetcdfDataset.acquireDataset(NetcdfDataset.java:494)
> >>>>>> at 
> >>>>>> ucar.nc2.ft.FeatureDatasetFactoryManager.open(FeatureDatasetFactoryManager.java:216)
> >>>>>> at 
> >>>>>> ucar.unidata.data.profiler.CDMProfilerDataSource.initProfiler(CDMProfilerDataSource.java:385)
> >>>>>> at 
> >>>>>> ucar.unidata.data.profiler.CDMProfilerDataSource.initProfilerAll(CDMProfilerDataSource.java:367)
> >>>>>> at 
> >>>>>> ucar.unidata.data.profiler.CDMProfilerDataSource.<init>(CDMProfilerDataSource.java:180)
> >>>>>> at 
> >>>>>> ucar.unidata.data.profiler.CDMProfilerDataSource.<init>(CDMProfilerDataSource.java:158)
> >>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> >>>>>> Method)
> >>>>>> at 
> >>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >>>>>> at 
> >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >>>>>> at 
> >>>>>> ucar.unidata.data.DataManager.createDataSource(DataManager.java:1552)
> >>>>>> at 
> >>>>>> ucar.unidata.idv.IntegratedDataViewer.createDataSource(IntegratedDataViewer.java:1998)
> >>>>>> at 
> >>>>>> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1915)
> >>>>>> at 
> >>>>>> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1849)
> >>>>>> at 
> >>>>>> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1822)
> >>>>>> at 
> >>>>>> ucar.unidata.idv.chooser.IdvChooser.makeDataSource(IdvChooser.java:943)
> >>>>>> at 
> >>>>>> ucar.unidata.idv.chooser.FileChooser.selectFilesInner(FileChooser.java:656)
> >>>>>> at 
> >>>>>> ucar.unidata.idv.chooser.FileChooser.selectFiles(FileChooser.java:525)
> >>>>>> at 
> >>>>>> ucar.unidata.idv.chooser.FileChooser.doLoadInThread(FileChooser.java:547)
> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>>> at 
> >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>>>> at 
> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>> at ucar.unidata.util.Misc$2.run(Misc.java:1063)
> >>>>>> at ucar.unidata.util.Misc$3.run(Misc.java:1091)
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> Ticket Details
> >>>>> ===================
> >>>>> Ticket ID: SJQ-840722
> >>>>> Department: Support IDV
> >>>>> Priority: Normal
> >>>>> Status: Open
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: SJQ-840722
> >>> Department: Support IDV
> >>> Priority: Normal
> >>> Status: Open
> >>>
> >>
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: SJQ-840722
> > Department: Support IDV
> > Priority: Normal
> > Status: Open
> >
> 
> 


Ticket Details
===================
Ticket ID: SJQ-840722
Department: Support IDV
Priority: Normal
Status: Open