Re: NPP files

On 3/11/2010 8:04 AM, Tom Rink wrote:
> john caron wrote:
> > Tom Rink wrote:
> > > All,
> > >
> > > john caron wrote:
> > > > Dave Santek wrote:
> > > > > John,
> > > > >
> > > > > Any update to the 4 issues you list below? Tommy is beginning to 
> > > > > look at the NPP VIIRS data files. My first goal is to be able to 
> > > > > deal with the union aggregation of these files so we can associate 
> > > > > the data with the geolocation [since they are in separate 
> > > > > files]....issue #2 below. Even being able to use NcML would be a 
> > > > > great start!
> > > > >
> > > > > Do you have any thoughts on how we can get from Point A to Point 
> > > > > B? We have time & people to throw at this problem.
> > > > >
> > > > > Thanks,
> > > > > dave
> > > > >
> > > > > John Caron wrote:
> > > > > > Hi Dave:
> > > > > >
> > > > > > My apologies for losing your email down the cracks. 4.1 is really 
> > > > > > now the recommended version, we just havent had a chance to 
> > > > > > stabilize the code, but all the changes are in the point data stuff.
> > > > > >
> > > > > > Im reviewing our emails to make sure i understand what you need, 
> > > > > > which i think is access to NPOESS/NPP swath data through the CDM, 
> > > > > > to be visualized by McIDAS-V. I assume McIDAS-V == IDV for these 
> > > > > > purposes?
> > > > > >
> > > > > > The issues that I know about:
> > > > > >
> > > > > > 1. The NPP files ive seen use HDF5 "region references". Currently 
> > > > > > our HDF5 IOSP doesnt handle these (one of the few things we dont 
> > > > > > yet handle in HDF5 format). Region references appear to be a way 
> > > > > > to specify logical subsets of other data arrays which are in the 
> > > > > > file. Those "regular" data arrays are accessible through the CDM, 
> > > > > > so im unclear if we are missing any functionality by not having 
> > > > > > the region references working. So one task is to understand what 
> > > > > > the point of the NPOES region references is, and if we really 
> > > > > > need them. .
> > > > this one is in your court.
> > > > > > 2. The CDM needs geolocation. Another task is to charactorize 
> > > > > > which NPOESS/NPP files have this in the data files, and which 
> > > > > > have them seperate, and so need a union aggregation.
> > > > sounds like the coordinates are in a seperate file. can you count 
> > > > on that? do you know the name?
> > > Do you mean the geolocation coordinates?  If so, I think we can 
> > > count on them
> > > being located in a different file.
> > is the file name derivable from the original name? where does that 
> > file come from?
>
> It's my understanding, at this time, that there's a strict file naming 
> convention, which
> we could consider taking advantage of.

this is a core issue to solve. without geolocation, not much can be done.

> > > > > > 3. The swath data type is different from grid, particularly in 
> > > > > > how time dimesnion works. Sometimes the current geogrid 
> > > > > > implementation works and sometimes not. A task is to upgrade CDM 
> > > > > > to handle swaths correctly and generally.
> > >
> > > This seems a question of whether an Eulerian or Lagrangian view of 
> > > measurement
> > > is more appropriate, with regard the time dimension.
> > >
> > > > > > 4. Aggregation of granules. Current aggregation techniques 
> > > > > > probably wont work. Solutions depends on the context: a few files 
> > > > > > or large collections? There may have to be some external indexing 
> > > > > > for fast searching. First task is to characterize the 
> > > > > > collections, and the use cases. to
> > > > im currrently at NCDC where theyve just agreed to have someone here 
> > > > work on a swath feature type. also they will propose a CF 
> > > > convention. im not sure how fast those will happen. do you want to 
> > > > be involved?
> > >
> > > We certainly should be involved in that, and should have some 
> > > input.  In fact, we already
> > > have an effective swath feature type in McIDAS-V which handles multi 
> > > and hyper-spectral
> > > data like AIRS, MODIS, AVHRR, IASI wherein instrument-specific 
> > > peculiarities and
> > > requirements are handled.
> > this would be swath feature type within the CDM, analogous to 
> > GridDatatype. Perhaps you should extract your public API as a 
> > starting point?
>
> Sure, let's talk about how to do this.  I'd like to be able to take 
> advantage
> of the mark-up features, with the swath feature functionality that 
> I've already
> developed.

not sure what "mark-up features" mean

> > > > aggregation may be a more difficult matter if you are trying to 
> > > > answer queries like "give me all data in lat/lon and time bounding 
> > > > box", but simpler if you just want to allow index space subsetting. 
> > > > what do you need?
> > >
> > > A big step forward would be able to union files together so we can
> > > treat them as one, for example, of the same time range.  We should
> > > be able to handle the relationship between lon/lat/time and index
> > > space.
> >
> > that should already work. have you tried it?
>
> Yes, I created a union via ncml of two of these files.  When I open the
> ncml in toolsUI, it gives an error.  We can provide the files if 
> necessary.

yes, and send the error also