20040419: LIDAR data

Hi JoAnn-

JoAnn Parikh wrote:

> Enclosed are two jpg files created from MATLAB using the lidar
> data that was stored in the netcdf file.  The first file displays
> all the lidar data from the first to the last altitude (up to 60 km)
> using the jet colortable in MATLAB.  Time is on the horizontal axis
> with the values for 59 minutes and height is on the vertical axis.
> The MATLAB command used for the display was imagesc with a scale
> between 0 and 2. The data needs to be scaled so the user
> can visualize the aerosol structure.  The second file is
> a more realistic file of the display that the user might want
> to see.  It goes up to only about 12 km and deletes the points
> at the levels where the data is probably garbage.  (The scale
> is close to 0 to 2, perhaps it was 0 to 2.3 if I recall.) It was
> created by displaying only the data in rows 1597 to 1996 for
> each time frame.  MATLAB displays the last row in the lower
> left-hand corner of the image, so you are seeing the data
> near the ground at the bottom of the image.  MATLAB also
> runs all indices from 1 up (not from zero).  I am also
> enclosing a MATLAB log file with the actual numbers of the data
> for column 1 and rows 1597 to 1996 (recalling that these
> indices start at 1).  This data discards the points in
> rows 1997, 1998, 1999, 2000 and 2001.  It would be nice
> if IDV could probe into the actual data values for the
> lidar data just as it does on other files.

The latter is a longstanding request, but we're not there yet.
Read on...

> Your program is probably displaying the data correctly.  You
> said the data looked strange, but that may just be because
> of scaling.  I am looking forward to seeing how you created
> the netCDF file so that it displays the lidar data.

The problem I was encountering was a problem with my graphics
card, but I still have some other issues.  Using the file that
you sent Jeff, I was able (with some mods to the IDV) display
your data as a Time/Height cross section.  See the attached
JPG.  A couple of notes:

1) The data you sent seem to be "upside down".  I'm thinking
this is a problem in the way the netCDF file was written, not
in my interpretation.  For example, if you just look at the
first few values for rawdata:

rawdata
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,
 0.1668,.....

These should be at the lowest elevations.  In your images, it looked
like the highest values were at the lowest elevations.  As you see from
my image, it's upside down from yours.

2) This did require some mods to the IDV to handle the single point
grid.  I think this is the best approach and I should have something
finished by the end of the week.  I'm tied up in a meeting all day
today and tomorrow.

> I was wondering how IDV handles file transfers.  Does IDV
> transfer the whole netCDF file to the user and, if so, where
> does it put the file?  Are there any security issues involved?
> Should I try to prepare small files or does IDV keep on going
> back and forth to the server if, for example, color tables
> are changed?  Also, can I call IDV from a web page that I created?

In general, we read only the data as it is requested for data like
yours.  It does everything in memory, so there is no file "transfer".
You can read netCDF files from a web page, or through and OPeNDAP
server.  The IDV is a "thick" client, so most of the manipulation
is done in the IDV.  So once you read in data, changing color tables,
units, etc is done locally and does not require a trip back to the
server.  Only when you load in a new data choice does it go back
to the file (in general).

You could start the IDV from a web page through Web Start.  Whoever
accesses the JNLP file would have to have Web Start installed (comes
with Java) on their machine.  The IDV would be downloaded if it wasn't
already and then launched.  You can customize the JNLP file to load
in certain datasets and data views through the IDV bundling
mechanism.

> Please feel free to call me at (203) 281-4589 or send e-mail with
> any questions about the enclosed files.
>
> Thanks very much for working on this data.

Sorry it's taking so long.

Don
*************************************************************
Don Murray                               UCAR Unidata Program
address@hidden                        P.O. Box 3000
(303) 497-8628                              Boulder, CO 80307
http://www.unidata.ucar.edu/staff/donm
"There's someone in my head, but it's not me"    Roger Waters
*************************************************************