Re: NPP files

john caron wrote:
> Tom Rink wrote:
> > All,
> >
> > john caron wrote:
> > > Dave Santek wrote:
> > > > John,
> > > >
> > > > Any update to the 4 issues you list below? Tommy is beginning to 
> > > > look at the NPP VIIRS data files. My first goal is to be able to 
> > > > deal with the union aggregation of these files so we can associate 
> > > > the data with the geolocation [since they are in separate 
> > > > files]....issue #2 below. Even being able to use NcML would be a 
> > > > great start!
> > > >
> > > > Do you have any thoughts on how we can get from Point A to Point B? 
> > > > We have time & people to throw at this problem.
> > > >
> > > > Thanks,
> > > > dave
> > > >
> > > > John Caron wrote:
> > > > > Hi Dave:
> > > > >
> > > > > My apologies for losing your email down the cracks. 4.1 is really 
> > > > > now the recommended version, we just havent had a chance to 
> > > > > stabilize the code, but all the changes are in the point data stuff.
> > > > >
> > > > > Im reviewing our emails to make sure i understand what you need, 
> > > > > which i think is access to NPOESS/NPP swath data through the CDM, 
> > > > > to be visualized by McIDAS-V. I assume McIDAS-V == IDV for these 
> > > > > purposes?
> > > > >
> > > > > The issues that I know about:
> > > > >
> > > > > 1. The NPP files ive seen use HDF5 "region references". Currently 
> > > > > our HDF5 IOSP doesnt handle these (one of the few things we dont 
> > > > > yet handle in HDF5 format). Region references appear to be a way 
> > > > > to specify logical subsets of other data arrays which are in the 
> > > > > file. Those "regular" data arrays are accessible through the CDM, 
> > > > > so im unclear if we are missing any functionality by not having 
> > > > > the region references working. So one task is to understand what 
> > > > > the point of the NPOES region references is, and if we really need 
> > > > > them. .
> > > this one is in your court.
> > > > > 2. The CDM needs geolocation. Another task is to charactorize 
> > > > > which NPOESS/NPP files have this in the data files, and which have 
> > > > > them seperate, and so need a union aggregation.
> > > sounds like the coordinates are in a seperate file. can you count on 
> > > that? do you know the name?
> > Do you mean the geolocation coordinates?  If so, I think we can count 
> > on them
> > being located in a different file.
> is the file name derivable from the original name? where does that 
> file come from?

It's my understanding, at this time, that there's a strict file naming 
convention, which
we could consider taking advantage of.

> > > > > 3. The swath data type is different from grid, particularly in how 
> > > > > time dimesnion works. Sometimes the current geogrid implementation 
> > > > > works and sometimes not. A task is to upgrade CDM to handle swaths 
> > > > > correctly and generally.
> >
> > This seems a question of whether an Eulerian or Lagrangian view of 
> > measurement
> > is more appropriate, with regard the time dimension.
> >
> > > > > 4. Aggregation of granules. Current aggregation techniques 
> > > > > probably wont work. Solutions depends on the context: a few files 
> > > > > or large collections? There may have to be some external indexing 
> > > > > for fast searching. First task is to characterize the collections, 
> > > > > and the use cases. to
> > > im currrently at NCDC where theyve just agreed to have someone here 
> > > work on a swath feature type. also they will propose a CF 
> > > convention. im not sure how fast those will happen. do you want to 
> > > be involved?
> >
> > We certainly should be involved in that, and should have some input.  
> > In fact, we already
> > have an effective swath feature type in McIDAS-V which handles multi 
> > and hyper-spectral
> > data like AIRS, MODIS, AVHRR, IASI wherein instrument-specific 
> > peculiarities and
> > requirements are handled.
> this would be swath feature type within the CDM, analogous to 
> GridDatatype. Perhaps you should extract your public API as a starting 
> point?

Sure, let's talk about how to do this.  I'd like to be able to take 
advantage
of the mark-up features, with the swath feature functionality that I've 
already
developed.

> > > aggregation may be a more difficult matter if you are trying to 
> > > answer queries like "give me all data in lat/lon and time bounding 
> > > box", but simpler if you just want to allow index space subsetting. 
> > > what do you need?
> >
> > A big step forward would be able to union files together so we can
> > treat them as one, for example, of the same time range.  We should
> > be able to handle the relationship between lon/lat/time and index
> > space.
>
> that should already work. have you tried it?

Yes, I created a union via ncml of two of these files.  When I open the
ncml in toolsUI, it gives an error.  We can provide the files if necessary.

Tom