[IDV #LPW-471935]: Plotting .hdf files in IDV

Hi Alexa,

Thanks for asking. Actually, I would be most interested in seeing the final NCML
where you had to tweak the date/time stuff.

Thanks again.

-Julien

> Are you interested in the rest of the HDF files (18 in total I think), and/or 
> the lightning data (as a csv)? -Alexa
> 
> 
> > On Apr 5, 2016, at 8:16 AM, Unidata IDV Support <address@hidden> wrote:
> >
> > Excellent. Thank you.
> >
> >> Feel free to use this example.
> >>
> >> Best Regards,
> >> Mike
> >>
> >>
> >> Michael J. Pavolonis, PhD
> >> Physical Scientist (NOAA/NESDIS)
> >> Phone: +1.608.263.9597
> >> Fax: +1.608.262.5974
> >> address@hidden
> >>
> >> Duty Station and Mailing Address:
> >> NOAA Cooperative Institute for Meteorological Satellite Studies
> >> University of Wisconsin - Madison
> >> 1225 W. Dayton St., Room 207
> >> Madison, WI 53706 USA
> >>
> >>
> >>
> >>> On Apr 4, 2016, at 1:57 PM, Alexa Van Eaton <address@hidden> wrote:
> >>>
> >>> Hi there,
> >>>
> >>> The person to ask about this would be Mike Pavolonis (Ccd), who created 
> >>> these brightness temperature files. Mike, could you read the message 
> >>> below and provide your thoughts? I’m attaching a screencast of what I’ve 
> >>> done with these files so far. The crosshairs are volcanic lighting 
> >>> detected by WWLLN, brought in through a point text data file.
> >>>
> >>> All the best
> >>>
> >>> Alexa
> >>>
> >>> <Kelut_MTSAT_Lightning.mov>
> >>>
> >>>
> >>>> On Apr 4, 2016, at 11:41 AM, Unidata IDV Support <address@hidden> wrote:
> >>>>
> >>>> Alexa,
> >>>>
> >>>> Great! I am glad it worked.
> >>>>
> >>>> I was contemplating writing a blog entry for our Unidata Developer Blog 
> >>>> on this
> >>>> experience as it makes a good case study. Would it be OK if I used your
> >>>> data/ncml for this purpose? It is no problem if you would prefer not to.
> >>>>
> >>>> Best,
> >>>>
> >>>> Unidata IDV Support
> >>>>
> >>>>> Dear IDV Support (and Sean),
> >>>>>
> >>>>> This works perfectly. I realized the file naming scheme was actually 
> >>>>> actually giving year + day of the year, so I adjusted the 
> >>>>> dateFormatMark using the syntax in this example 
> >>>>> http://docs.opendap.org/index.php/Dynamic_Aggregation_Tutorial and now 
> >>>>> the dates/times are being read accurately.
> >>>>>
> >>>>> As a side note, I noticed that Projections—>Enable Adaptive Resolution 
> >>>>> is on by default in the IDV, which automatically subsamples the data. 
> >>>>> But once that function was unchecked and the data reloaded, everything 
> >>>>> looks good. Big thanks to the both of you.
> >>>>>
> >>>>> All the best
> >>>>>
> >>>>> Alexa
> >>>>>
> >>>>>
> >>>>>> On Apr 1, 2016, at 12:06 PM, Unidata IDV Support <address@hidden> 
> >>>>>> wrote:
> >>>>>>
> >>>>>> Hi Alexa,
> >>>>>>
> >>>>>> What I was describing below had nothing to do with the problem. Please 
> >>>>>> see the
> >>>>>> attached NCML. This file should correctly aggregate the variables over
> >>>>>> the time dimension, and also it will extract the times from the file 
> >>>>>> name, so
> >>>>>> you don't have to manually describe them.
> >>>>>>
> >>>>>> Let us know how it goes.
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> Unidata IDV Support
> >>>>>>
> >>>>>> PS Thanks to my colleague Sean Arms for the NCML lessons :-)
> >>>>>>
> >>>>>>> Sure thing, I’ll have a play and keep you posted. All the best, Alexa
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Mar 31, 2016, at 4:21 PM, Unidata IDV Support <address@hidden> 
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Alexa,
> >>>>>>>>
> >>>>>>>> I think I know what the problem is, but don't have time to try 
> >>>>>>>> before I go home.
> >>>>>>>>
> >>>>>>>> I think the problem is that we have only changed one variable when 
> >>>>>>>> we should
> >>>>>>>> have changed them all, at least all those that are listed under the 
> >>>>>>>> Field
> >>>>>>>> Selector.  The IDV may be getting confused on this.
> >>>>>>>>
> >>>>>>>> Do you want to give that a shot and let me know what happens?
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>>
> >>>>>>>> Unidata IDV Support
> >>>>>>>>
> >>>>>>>>> Alexa,
> >>>>>>>>>
> >>>>>>>>> Yes, thanks for the clarification. (I think the ordering of our 
> >>>>>>>>> messages got
> >>>>>>>>> jumbled there.)
> >>>>>>>>>
> >>>>>>>>> I see what you are saying, and when I load each individual file 
> >>>>>>>>> separately I can
> >>>>>>>>> see the data are definitely not the same so something is amiss. 
> >>>>>>>>> We'll resume
> >>>>>>>>> tomorrow.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>>
> >>>>>>>>> Unidata IDV Support
> >>>>>>>>>
> >>>>>>>>>> Hi there — thank you for for your reply. Using your syntax below, 
> >>>>>>>>>> I tried aggregating three files (edited script attached) — 
> >>>>>>>>>> different files this time. Again, the files are read into IDV with 
> >>>>>>>>>> the correct time stamps, and I can toggle between the times, but 
> >>>>>>>>>> the strange thing is the plot does not change with time. It’s like 
> >>>>>>>>>> it’s only plotting one time. -Alexa
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Mar 31, 2016, at 3:46 PM, Unidata IDV Support <address@hidden> 
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Alexa,
> >>>>>>>>>>>
> >>>>>>>>>>> Great! Glad we are making progress and also that you are hanging 
> >>>>>>>>>>> in there. This
> >>>>>>>>>>> stuff is not so easy.
> >>>>>>>>>>>
> >>>>>>>>>>> Replace the current values element with something like this:
> >>>>>>>>>>>
> >>>>>>>>>>> <values>0 3 11</values>
> >>>>>>>>>>>
> >>>>>>>>>>> But if that is too onerous because you have many file, we will 
> >>>>>>>>>>> have to go via
> >>>>>>>>>>> the time extraction route. Let me know...
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>>
> >>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>
> >>>>>>>>>>>> Hello — this looks great, and I appreciate your help with this. 
> >>>>>>>>>>>> Your ncml script worked for the two test files and the times 
> >>>>>>>>>>>> look right. However, since we have some missing satellite 
> >>>>>>>>>>>> images, not all the files are spaced exactly 10 mins apart. Is 
> >>>>>>>>>>>> there a way to specify time gaps between individual files begin 
> >>>>>>>>>>>> aggregated? Otherwise, perhaps extracting the time from the file 
> >>>>>>>>>>>> name would be good route. -Alexa
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Mar 31, 2016, at 3:29 PM, Unidata IDV Support 
> >>>>>>>>>>>>> <address@hidden> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi again,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The embedded NcML is a bit hard to read so attached, you will 
> >>>>>>>>>>>>> find
> >>>>>>>>>>>>> that same NcML file.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Alexa,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> It is a little tricky but you can fix these problems with a 
> >>>>>>>>>>>>>> bit of
> >>>>>>>>>>>>>> NcML. To get us going, let's look at the following NcML which 
> >>>>>>>>>>>>>> can be
> >>>>>>>>>>>>>> read into the IDV and will allow you to view the
> >>>>>>>>>>>>>> `channel_14_brightness_temperature' over time. This file 
> >>>>>>>>>>>>>> should be
> >>>>>>>>>>>>>> located next to your `hdf' files. Please read the comments in 
> >>>>>>>>>>>>>> the NcML
> >>>>>>>>>>>>>> for more information about what is going on here.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ,----
> >>>>>>>>>>>>>> | <?xml version="1.0" encoding="UTF-8"?>
> >>>>>>>>>>>>>> | <netcdf 
> >>>>>>>>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >>>>>>>>>>>>>> |   <!-- Create a time coordinate variable that will be used 
> >>>>>>>>>>>>>> for the aggregation. -->
> >>>>>>>>>>>>>> |   <variable name="time" type="int" shape="time" >
> >>>>>>>>>>>>>> |     <!-- Let's add some units. Alexa: please make sure these 
> >>>>>>>>>>>>>> units are correct. -->
> >>>>>>>>>>>>>> |     <attribute name="units" value="minutes since 2014-04-04 
> >>>>>>>>>>>>>> 19:09"/>
> >>>>>>>>>>>>>> |     <!-- Specify this variable as a time coordinate -->
> >>>>>>>>>>>>>> |     <attribute name="_CoordinateAxisType" value="Time" />
> >>>>>>>>>>>>>> |     <!-- Increment each file listed below by 10 minutes. 
> >>>>>>>>>>>>>> Alexa, please make sure this -->
> >>>>>>>>>>>>>> |     <!-- assumption about the times is correct. -->
> >>>>>>>>>>>>>> |     <values start="0" increment="10" />
> >>>>>>>>>>>>>> |   </variable>
> >>>>>>>>>>>>>> |   <!--   Specify the coordinates of time lat lon for this 
> >>>>>>>>>>>>>> one variable. Please use -->
> >>>>>>>>>>>>>> |   <!-- this as an example for other variables you are 
> >>>>>>>>>>>>>> interersted in. In other -->
> >>>>>>>>>>>>>> |   <!-- words, you can copy and paste this variable NcML 
> >>>>>>>>>>>>>> element for your other -->
> >>>>>>>>>>>>>> |   <!-- variables. -->
> >>>>>>>>>>>>>> |   <variable name="channel_14_brightness_temperature" 
> >>>>>>>>>>>>>> shape="lines elements" type="short">
> >>>>>>>>>>>>>> |     <attribute name="coordinates" value="time pixel_latitude 
> >>>>>>>>>>>>>> pixel_longitude" />
> >>>>>>>>>>>>>> |   </variable>
> >>>>>>>>>>>>>> |   <!-- Now you are ready to do your aggregation, which is 
> >>>>>>>>>>>>>> the easy part at this point -->
> >>>>>>>>>>>>>> |   <aggregation dimName="time" type="joinNew">
> >>>>>>>>>>>>>> |     <variableAgg name="T"/>
> >>>>>>>>>>>>>> |     <netcdf location="geocatL1.MTSAT-1R.2014044.190900.hdf"/>
> >>>>>>>>>>>>>> |     <netcdf location="geocatL1.MTSAT-1R.2014044.191900.hdf"/>
> >>>>>>>>>>>>>> |   </aggregation>
> >>>>>>>>>>>>>> | </netcdf>
> >>>>>>>>>>>>>> `----
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In summary, this NcML should aggregate over time the two of 
> >>>>>>>>>>>>>> HDF files
> >>>>>>>>>>>>>> you provided. You will see only one variable that is 
> >>>>>>>>>>>>>> aggregated over
> >>>>>>>>>>>>>> time; `channel_14_brightness_temperature'.  I'll leave it up 
> >>>>>>>>>>>>>> to you to
> >>>>>>>>>>>>>> do the others though do not hesitate to ask for help.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Also, if interested, there are also fancier tricks we can do 
> >>>>>>>>>>>>>> here such
> >>>>>>>>>>>>>> as NcML can be smart enough to extract the times from the file 
> >>>>>>>>>>>>>> names.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Ok sounds promising. Here is the link to another file, which 
> >>>>>>>>>>>>>>> should differ only in time — 
> >>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRDVyczU3N2NMbFU/view?usp=sharing
> >>>>>>>>>>>>>>>    -Alexa
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mar 30, 2016, at 4:49 PM, Unidata IDV Support 
> >>>>>>>>>>>>>>>> <address@hidden> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Alexa,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Indeed we may be able to patch things up with a bit of ncml. 
> >>>>>>>>>>>>>>>> I only found one
> >>>>>>>>>>>>>>>> file at the drive.google location. Would it be possible to 
> >>>>>>>>>>>>>>>> see more so that we
> >>>>>>>>>>>>>>>> have something to aggregate?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello IDV Support,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I am working on reading in a number of HDF files using 
> >>>>>>>>>>>>>>>>> ‘aggregate grids by time’ or similar. I am able to read the 
> >>>>>>>>>>>>>>>>> files in individually without a problem, but it looks like 
> >>>>>>>>>>>>>>>>> the time variable is not recognized. The error message is 
> >>>>>>>>>>>>>>>>> "Grid data source failed making data set….Dimension time 
> >>>>>>>>>>>>>>>>> does not exist”. Any ideas for how to specify the time 
> >>>>>>>>>>>>>>>>> variable? Could this be achieved with an .ncml script?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Example file is located here: 
> >>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRFhkRVM2aVVRVGc/view?usp=sharing
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> All the best
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Alexa
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Ticket Details
> >>>>>>>>>>>>>>>> ===================
> >>>>>>>>>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>>>>>>>>> Department: Support IDV
> >>>>>>>>>>>>>>>> Priority: Normal
> >>>>>>>>>>>>>>>> Status: Closed
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Ticket Details
> >>>>>>>>>>>>> ===================
> >>>>>>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>>>>>> Department: Support IDV
> >>>>>>>>>>>>> Priority: Normal
> >>>>>>>>>>>>> Status: Closed<alexa.ncml>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Ticket Details
> >>>>>>>>>>> ===================
> >>>>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>>>> Department: Support IDV
> >>>>>>>>>>> Priority: Normal
> >>>>>>>>>>> Status: Closed
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Ticket Details
> >>>>>>>> ===================
> >>>>>>>> Ticket ID: LPW-471935
> >>>>>>>> Department: Support IDV
> >>>>>>>> Priority: Normal
> >>>>>>>> Status: Open
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Ticket Details
> >>>>>> ===================
> >>>>>> Ticket ID: LPW-471935
> >>>>>> Department: Support IDV
> >>>>>> Priority: Normal
> >>>>>> Status: Closed<alexa.ncml>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> Ticket Details
> >>>> ===================
> >>>> Ticket ID: LPW-471935
> >>>> Department: Support IDV
> >>>> Priority: Normal
> >>>> Status: Closed
> >>>>
> >>>
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: LPW-471935
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed
> >
> 
> 


Ticket Details
===================
Ticket ID: LPW-471935
Department: Support IDV
Priority: Normal
Status: Closed