Re: NPP files

Tom Rink wrote:
> All,
>
> john caron wrote:
> > Dave Santek wrote:
> > > John,
> > >
> > > Any update to the 4 issues you list below? Tommy is beginning to 
> > > look at the NPP VIIRS data files. My first goal is to be able to 
> > > deal with the union aggregation of these files so we can associate 
> > > the data with the geolocation [since they are in separate 
> > > files]....issue #2 below. Even being able to use NcML would be a 
> > > great start!
> > >
> > > Do you have any thoughts on how we can get from Point A to Point B? 
> > > We have time & people to throw at this problem.
> > >
> > > Thanks,
> > > dave
> > >
> > > John Caron wrote:
> > > > Hi Dave:
> > > >
> > > > My apologies for losing your email down the cracks. 4.1 is really 
> > > > now the recommended version, we just havent had a chance to 
> > > > stabilize the code, but all the changes are in the point data stuff.
> > > >
> > > > Im reviewing our emails to make sure i understand what you need, 
> > > > which i think is access to NPOESS/NPP swath data through the CDM, 
> > > > to be visualized by McIDAS-V. I assume McIDAS-V == IDV for these 
> > > > purposes?
> > > >
> > > > The issues that I know about:
> > > >
> > > > 1. The NPP files ive seen use HDF5 "region references". Currently 
> > > > our HDF5 IOSP doesnt handle these (one of the few things we dont 
> > > > yet handle in HDF5 format). Region references appear to be a way to 
> > > > specify logical subsets of other data arrays which are in the file. 
> > > > Those "regular" data arrays are accessible through the CDM, so im 
> > > > unclear if we are missing any functionality by not having the 
> > > > region references working. So one task is to understand what the 
> > > > point of the NPOES region references is, and if we really need them. .
> > this one is in your court.
> > > > 2. The CDM needs geolocation. Another task is to charactorize which 
> > > > NPOESS/NPP files have this in the data files, and which have them 
> > > > seperate, and so need a union aggregation.
> > sounds like the coordinates are in a seperate file. can you count on 
> > that? do you know the name?
> Do you mean the geolocation coordinates?  If so, I think we can count 
> on them
> being located in a different file.
is the file name derivable from the original name? where does that file 
come from?
> > > > 3. The swath data type is different from grid, particularly in how 
> > > > time dimesnion works. Sometimes the current geogrid implementation 
> > > > works and sometimes not. A task is to upgrade CDM to handle swaths 
> > > > correctly and generally.
>
> This seems a question of whether an Eulerian or Lagrangian view of 
> measurement
> is more appropriate, with regard the time dimension.
>
> > > > 4. Aggregation of granules. Current aggregation techniques probably 
> > > > wont work. Solutions depends on the context: a few files or large 
> > > > collections? There may have to be some external indexing for fast 
> > > > searching. First task is to characterize the collections, and the 
> > > > use cases. to
> > im currrently at NCDC where theyve just agreed to have someone here 
> > work on a swath feature type. also they will propose a CF convention. 
> > im not sure how fast those will happen. do you want to be involved?
>
> We certainly should be involved in that, and should have some input.  
> In fact, we already
> have an effective swath feature type in McIDAS-V which handles multi 
> and hyper-spectral
> data like AIRS, MODIS, AVHRR, IASI wherein instrument-specific 
> peculiarities and
> requirements are handled.
this would be swath feature type within the CDM, analogous to 
GridDatatype. Perhaps you should extract your public API as a starting 
point?
> > aggregation may be a more difficult matter if you are trying to 
> > answer queries like "give me all data in lat/lon and time bounding 
> > box", but simpler if you just want to allow index space subsetting. 
> > what do you need?
>
> A big step forward would be able to union files together so we can
> treat them as one, for example, of the same time range.  We should
> be able to handle the relationship between lon/lat/time and index
> space.

that should already work. have you tried it?