[IDV #LPW-471935]: Plotting .hdf files in IDV

Excellent. Thank you.

> Feel free to use this example.
> 
> Best Regards,
> Mike
> 
> 
> Michael J. Pavolonis, PhD
> Physical Scientist (NOAA/NESDIS)
> Phone: +1.608.263.9597
> Fax: +1.608.262.5974
> address@hidden
> 
> Duty Station and Mailing Address:
> NOAA Cooperative Institute for Meteorological Satellite Studies
> University of Wisconsin - Madison
> 1225 W. Dayton St., Room 207
> Madison, WI 53706 USA
> 
> 
> 
> > On Apr 4, 2016, at 1:57 PM, Alexa Van Eaton <address@hidden> wrote:
> >
> > Hi there,
> >
> > The person to ask about this would be Mike Pavolonis (Ccd), who created 
> > these brightness temperature files. Mike, could you read the message below 
> > and provide your thoughts? I’m attaching a screencast of what I’ve done 
> > with these files so far. The crosshairs are volcanic lighting detected by 
> > WWLLN, brought in through a point text data file.
> >
> > All the best
> >
> > Alexa
> >
> > <Kelut_MTSAT_Lightning.mov>
> >
> >
> >> On Apr 4, 2016, at 11:41 AM, Unidata IDV Support <address@hidden> wrote:
> >>
> >> Alexa,
> >>
> >> Great! I am glad it worked.
> >>
> >> I was contemplating writing a blog entry for our Unidata Developer Blog on 
> >> this
> >> experience as it makes a good case study. Would it be OK if I used your
> >> data/ncml for this purpose? It is no problem if you would prefer not to.
> >>
> >> Best,
> >>
> >> Unidata IDV Support
> >>
> >>> Dear IDV Support (and Sean),
> >>>
> >>> This works perfectly. I realized the file naming scheme was actually 
> >>> actually giving year + day of the year, so I adjusted the dateFormatMark 
> >>> using the syntax in this example 
> >>> http://docs.opendap.org/index.php/Dynamic_Aggregation_Tutorial and now 
> >>> the dates/times are being read accurately.
> >>>
> >>> As a side note, I noticed that Projections—>Enable Adaptive Resolution is 
> >>> on by default in the IDV, which automatically subsamples the data. But 
> >>> once that function was unchecked and the data reloaded, everything looks 
> >>> good. Big thanks to the both of you.
> >>>
> >>> All the best
> >>>
> >>> Alexa
> >>>
> >>>
> >>>> On Apr 1, 2016, at 12:06 PM, Unidata IDV Support <address@hidden> wrote:
> >>>>
> >>>> Hi Alexa,
> >>>>
> >>>> What I was describing below had nothing to do with the problem. Please 
> >>>> see the
> >>>> attached NCML. This file should correctly aggregate the variables over
> >>>> the time dimension, and also it will extract the times from the file 
> >>>> name, so
> >>>> you don't have to manually describe them.
> >>>>
> >>>> Let us know how it goes.
> >>>>
> >>>> Best,
> >>>>
> >>>> Unidata IDV Support
> >>>>
> >>>> PS Thanks to my colleague Sean Arms for the NCML lessons :-)
> >>>>
> >>>>> Sure thing, I’ll have a play and keep you posted. All the best, Alexa
> >>>>>
> >>>>>
> >>>>>> On Mar 31, 2016, at 4:21 PM, Unidata IDV Support <address@hidden> 
> >>>>>> wrote:
> >>>>>>
> >>>>>> Alexa,
> >>>>>>
> >>>>>> I think I know what the problem is, but don't have time to try before 
> >>>>>> I go home.
> >>>>>>
> >>>>>> I think the problem is that we have only changed one variable when we 
> >>>>>> should
> >>>>>> have changed them all, at least all those that are listed under the 
> >>>>>> Field
> >>>>>> Selector.  The IDV may be getting confused on this.
> >>>>>>
> >>>>>> Do you want to give that a shot and let me know what happens?
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> Unidata IDV Support
> >>>>>>
> >>>>>>> Alexa,
> >>>>>>>
> >>>>>>> Yes, thanks for the clarification. (I think the ordering of our 
> >>>>>>> messages got
> >>>>>>> jumbled there.)
> >>>>>>>
> >>>>>>> I see what you are saying, and when I load each individual file 
> >>>>>>> separately I can
> >>>>>>> see the data are definitely not the same so something is amiss. We'll 
> >>>>>>> resume
> >>>>>>> tomorrow.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>>
> >>>>>>> Unidata IDV Support
> >>>>>>>
> >>>>>>>> Hi there — thank you for for your reply. Using your syntax below, I 
> >>>>>>>> tried aggregating three files (edited script attached) — different 
> >>>>>>>> files this time. Again, the files are read into IDV with the correct 
> >>>>>>>> time stamps, and I can toggle between the times, but the strange 
> >>>>>>>> thing is the plot does not change with time. It’s like it’s only 
> >>>>>>>> plotting one time. -Alexa
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Mar 31, 2016, at 3:46 PM, Unidata IDV Support <address@hidden> 
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Alexa,
> >>>>>>>>>
> >>>>>>>>> Great! Glad we are making progress and also that you are hanging in 
> >>>>>>>>> there. This
> >>>>>>>>> stuff is not so easy.
> >>>>>>>>>
> >>>>>>>>> Replace the current values element with something like this:
> >>>>>>>>>
> >>>>>>>>> <values>0 3 11</values>
> >>>>>>>>>
> >>>>>>>>> But if that is too onerous because you have many file, we will have 
> >>>>>>>>> to go via
> >>>>>>>>> the time extraction route. Let me know...
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>>
> >>>>>>>>> Unidata IDV Support
> >>>>>>>>>
> >>>>>>>>>> Hello — this looks great, and I appreciate your help with this. 
> >>>>>>>>>> Your ncml script worked for the two test files and the times look 
> >>>>>>>>>> right. However, since we have some missing satellite images, not 
> >>>>>>>>>> all the files are spaced exactly 10 mins apart. Is there a way to 
> >>>>>>>>>> specify time gaps between individual files begin aggregated? 
> >>>>>>>>>> Otherwise, perhaps extracting the time from the file name would be 
> >>>>>>>>>> good route. -Alexa
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Mar 31, 2016, at 3:29 PM, Unidata IDV Support <address@hidden> 
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi again,
> >>>>>>>>>>>
> >>>>>>>>>>> The embedded NcML is a bit hard to read so attached, you will find
> >>>>>>>>>>> that same NcML file.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>>
> >>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Alexa,
> >>>>>>>>>>>>
> >>>>>>>>>>>> It is a little tricky but you can fix these problems with a bit 
> >>>>>>>>>>>> of
> >>>>>>>>>>>> NcML. To get us going, let's look at the following NcML which 
> >>>>>>>>>>>> can be
> >>>>>>>>>>>> read into the IDV and will allow you to view the
> >>>>>>>>>>>> `channel_14_brightness_temperature' over time. This file should 
> >>>>>>>>>>>> be
> >>>>>>>>>>>> located next to your `hdf' files. Please read the comments in 
> >>>>>>>>>>>> the NcML
> >>>>>>>>>>>> for more information about what is going on here.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ,----
> >>>>>>>>>>>> | <?xml version="1.0" encoding="UTF-8"?>
> >>>>>>>>>>>> | <netcdf 
> >>>>>>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >>>>>>>>>>>> |   <!-- Create a time coordinate variable that will be used for 
> >>>>>>>>>>>> the aggregation. -->
> >>>>>>>>>>>> |   <variable name="time" type="int" shape="time" >
> >>>>>>>>>>>> |     <!-- Let's add some units. Alexa: please make sure these 
> >>>>>>>>>>>> units are correct. -->
> >>>>>>>>>>>> |     <attribute name="units" value="minutes since 2014-04-04 
> >>>>>>>>>>>> 19:09"/>
> >>>>>>>>>>>> |     <!-- Specify this variable as a time coordinate -->
> >>>>>>>>>>>> |     <attribute name="_CoordinateAxisType" value="Time" />
> >>>>>>>>>>>> |     <!-- Increment each file listed below by 10 minutes. 
> >>>>>>>>>>>> Alexa, please make sure this -->
> >>>>>>>>>>>> |     <!-- assumption about the times is correct. -->
> >>>>>>>>>>>> |     <values start="0" increment="10" />
> >>>>>>>>>>>> |   </variable>
> >>>>>>>>>>>> |   <!--   Specify the coordinates of time lat lon for this one 
> >>>>>>>>>>>> variable. Please use -->
> >>>>>>>>>>>> |   <!-- this as an example for other variables you are 
> >>>>>>>>>>>> interersted in. In other -->
> >>>>>>>>>>>> |   <!-- words, you can copy and paste this variable NcML 
> >>>>>>>>>>>> element for your other -->
> >>>>>>>>>>>> |   <!-- variables. -->
> >>>>>>>>>>>> |   <variable name="channel_14_brightness_temperature" 
> >>>>>>>>>>>> shape="lines elements" type="short">
> >>>>>>>>>>>> |     <attribute name="coordinates" value="time pixel_latitude 
> >>>>>>>>>>>> pixel_longitude" />
> >>>>>>>>>>>> |   </variable>
> >>>>>>>>>>>> |   <!-- Now you are ready to do your aggregation, which is the 
> >>>>>>>>>>>> easy part at this point -->
> >>>>>>>>>>>> |   <aggregation dimName="time" type="joinNew">
> >>>>>>>>>>>> |     <variableAgg name="T"/>
> >>>>>>>>>>>> |     <netcdf location="geocatL1.MTSAT-1R.2014044.190900.hdf"/>
> >>>>>>>>>>>> |     <netcdf location="geocatL1.MTSAT-1R.2014044.191900.hdf"/>
> >>>>>>>>>>>> |   </aggregation>
> >>>>>>>>>>>> | </netcdf>
> >>>>>>>>>>>> `----
> >>>>>>>>>>>>
> >>>>>>>>>>>> In summary, this NcML should aggregate over time the two of HDF 
> >>>>>>>>>>>> files
> >>>>>>>>>>>> you provided. You will see only one variable that is aggregated 
> >>>>>>>>>>>> over
> >>>>>>>>>>>> time; `channel_14_brightness_temperature'.  I'll leave it up to 
> >>>>>>>>>>>> you to
> >>>>>>>>>>>> do the others though do not hesitate to ask for help.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Also, if interested, there are also fancier tricks we can do 
> >>>>>>>>>>>> here such
> >>>>>>>>>>>> as NcML can be smart enough to extract the times from the file 
> >>>>>>>>>>>> names.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Ok sounds promising. Here is the link to another file, which 
> >>>>>>>>>>>>> should differ only in time — 
> >>>>>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRDVyczU3N2NMbFU/view?usp=sharing
> >>>>>>>>>>>>>    -Alexa
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mar 30, 2016, at 4:49 PM, Unidata IDV Support 
> >>>>>>>>>>>>>> <address@hidden> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Alexa,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Indeed we may be able to patch things up with a bit of ncml. I 
> >>>>>>>>>>>>>> only found one
> >>>>>>>>>>>>>> file at the drive.google location. Would it be possible to see 
> >>>>>>>>>>>>>> more so that we
> >>>>>>>>>>>>>> have something to aggregate?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello IDV Support,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I am working on reading in a number of HDF files using 
> >>>>>>>>>>>>>>> ‘aggregate grids by time’ or similar. I am able to read the 
> >>>>>>>>>>>>>>> files in individually without a problem, but it looks like 
> >>>>>>>>>>>>>>> the time variable is not recognized. The error message is 
> >>>>>>>>>>>>>>> "Grid data source failed making data set….Dimension time does 
> >>>>>>>>>>>>>>> not exist”. Any ideas for how to specify the time variable? 
> >>>>>>>>>>>>>>> Could this be achieved with an .ncml script?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Example file is located here: 
> >>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRFhkRVM2aVVRVGc/view?usp=sharing
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> All the best
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Alexa
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ticket Details
> >>>>>>>>>>>>>> ===================
> >>>>>>>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>>>>>>> Department: Support IDV
> >>>>>>>>>>>>>> Priority: Normal
> >>>>>>>>>>>>>> Status: Closed
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Ticket Details
> >>>>>>>>>>> ===================
> >>>>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>>>> Department: Support IDV
> >>>>>>>>>>> Priority: Normal
> >>>>>>>>>>> Status: Closed<alexa.ncml>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Ticket Details
> >>>>>>>>> ===================
> >>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>> Department: Support IDV
> >>>>>>>>> Priority: Normal
> >>>>>>>>> Status: Closed
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Ticket Details
> >>>>>> ===================
> >>>>>> Ticket ID: LPW-471935
> >>>>>> Department: Support IDV
> >>>>>> Priority: Normal
> >>>>>> Status: Open
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> Ticket Details
> >>>> ===================
> >>>> Ticket ID: LPW-471935
> >>>> Department: Support IDV
> >>>> Priority: Normal
> >>>> Status: Closed<alexa.ncml>
> >>>
> >>>
> >>
> >>
> >> Ticket Details
> >> ===================
> >> Ticket ID: LPW-471935
> >> Department: Support IDV
> >> Priority: Normal
> >> Status: Closed
> >>
> >
> 
> 


Ticket Details
===================
Ticket ID: LPW-471935
Department: Support IDV
Priority: Normal
Status: Closed