[IDV #LPW-471935]: Plotting .hdf files in IDV

Alexa,

Great! I am glad it worked.

I was contemplating writing a blog entry for our Unidata Developer Blog on this
experience as it makes a good case study. Would it be OK if I used your
data/ncml for this purpose? It is no problem if you would prefer not to.

Best,

Unidata IDV Support

> Dear IDV Support (and Sean),
> 
> This works perfectly. I realized the file naming scheme was actually actually 
> giving year + day of the year, so I adjusted the dateFormatMark using the 
> syntax in this example 
> http://docs.opendap.org/index.php/Dynamic_Aggregation_Tutorial and now the 
> dates/times are being read accurately.
> 
> As a side note, I noticed that Projections—>Enable Adaptive Resolution is on 
> by default in the IDV, which automatically subsamples the data. But once that 
> function was unchecked and the data reloaded, everything looks good. Big 
> thanks to the both of you.
> 
> All the best
> 
> Alexa
> 
> 
> > On Apr 1, 2016, at 12:06 PM, Unidata IDV Support <address@hidden> wrote:
> >
> > Hi Alexa,
> >
> > What I was describing below had nothing to do with the problem. Please see 
> > the
> > attached NCML. This file should correctly aggregate the variables over
> > the time dimension, and also it will extract the times from the file name, 
> > so
> > you don't have to manually describe them.
> >
> > Let us know how it goes.
> >
> > Best,
> >
> > Unidata IDV Support
> >
> > PS Thanks to my colleague Sean Arms for the NCML lessons :-)
> >
> >> Sure thing, I’ll have a play and keep you posted. All the best, Alexa
> >>
> >>
> >>> On Mar 31, 2016, at 4:21 PM, Unidata IDV Support <address@hidden> wrote:
> >>>
> >>> Alexa,
> >>>
> >>> I think I know what the problem is, but don't have time to try before I 
> >>> go home.
> >>>
> >>> I think the problem is that we have only changed one variable when we 
> >>> should
> >>> have changed them all, at least all those that are listed under the Field
> >>> Selector.  The IDV may be getting confused on this.
> >>>
> >>> Do you want to give that a shot and let me know what happens?
> >>>
> >>> Best,
> >>>
> >>> Unidata IDV Support
> >>>
> >>>> Alexa,
> >>>>
> >>>> Yes, thanks for the clarification. (I think the ordering of our messages 
> >>>> got
> >>>> jumbled there.)
> >>>>
> >>>> I see what you are saying, and when I load each individual file 
> >>>> separately I can
> >>>> see the data are definitely not the same so something is amiss. We'll 
> >>>> resume
> >>>> tomorrow.
> >>>>
> >>>> Best,
> >>>>
> >>>> Unidata IDV Support
> >>>>
> >>>>> Hi there — thank you for for your reply. Using your syntax below, I 
> >>>>> tried aggregating three files (edited script attached) — different 
> >>>>> files this time. Again, the files are read into IDV with the correct 
> >>>>> time stamps, and I can toggle between the times, but the strange thing 
> >>>>> is the plot does not change with time. It’s like it’s only plotting one 
> >>>>> time. -Alexa
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Mar 31, 2016, at 3:46 PM, Unidata IDV Support <address@hidden> 
> >>>>>> wrote:
> >>>>>>
> >>>>>> Alexa,
> >>>>>>
> >>>>>> Great! Glad we are making progress and also that you are hanging in 
> >>>>>> there. This
> >>>>>> stuff is not so easy.
> >>>>>>
> >>>>>> Replace the current values element with something like this:
> >>>>>>
> >>>>>> <values>0 3 11</values>
> >>>>>>
> >>>>>> But if that is too onerous because you have many file, we will have to 
> >>>>>> go via
> >>>>>> the time extraction route. Let me know...
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> Unidata IDV Support
> >>>>>>
> >>>>>>> Hello — this looks great, and I appreciate your help with this. Your 
> >>>>>>> ncml script worked for the two test files and the times look right. 
> >>>>>>> However, since we have some missing satellite images, not all the 
> >>>>>>> files are spaced exactly 10 mins apart. Is there a way to specify 
> >>>>>>> time gaps between individual files begin aggregated? Otherwise, 
> >>>>>>> perhaps extracting the time from the file name would be good route. 
> >>>>>>> -Alexa
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Mar 31, 2016, at 3:29 PM, Unidata IDV Support <address@hidden> 
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi again,
> >>>>>>>>
> >>>>>>>> The embedded NcML is a bit hard to read so attached, you will find
> >>>>>>>> that same NcML file.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>>
> >>>>>>>> Unidata IDV Support
> >>>>>>>>
> >>>>>>>>> Hi Alexa,
> >>>>>>>>>
> >>>>>>>>> It is a little tricky but you can fix these problems with a bit of
> >>>>>>>>> NcML. To get us going, let's look at the following NcML which can be
> >>>>>>>>> read into the IDV and will allow you to view the
> >>>>>>>>> `channel_14_brightness_temperature' over time. This file should be
> >>>>>>>>> located next to your `hdf' files. Please read the comments in the 
> >>>>>>>>> NcML
> >>>>>>>>> for more information about what is going on here.
> >>>>>>>>>
> >>>>>>>>> ,----
> >>>>>>>>> | <?xml version="1.0" encoding="UTF-8"?>
> >>>>>>>>> | <netcdf 
> >>>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >>>>>>>>> |   <!-- Create a time coordinate variable that will be used for 
> >>>>>>>>> the aggregation. -->
> >>>>>>>>> |   <variable name="time" type="int" shape="time" >
> >>>>>>>>> |     <!-- Let's add some units. Alexa: please make sure these 
> >>>>>>>>> units are correct. -->
> >>>>>>>>> |     <attribute name="units" value="minutes since 2014-04-04 
> >>>>>>>>> 19:09"/>
> >>>>>>>>> |     <!-- Specify this variable as a time coordinate -->
> >>>>>>>>> |     <attribute name="_CoordinateAxisType" value="Time" />
> >>>>>>>>> |     <!-- Increment each file listed below by 10 minutes. Alexa, 
> >>>>>>>>> please make sure this -->
> >>>>>>>>> |     <!-- assumption about the times is correct. -->
> >>>>>>>>> |     <values start="0" increment="10" />
> >>>>>>>>> |   </variable>
> >>>>>>>>> |   <!--   Specify the coordinates of time lat lon for this one 
> >>>>>>>>> variable. Please use -->
> >>>>>>>>> |   <!-- this as an example for other variables you are interersted 
> >>>>>>>>> in. In other -->
> >>>>>>>>> |   <!-- words, you can copy and paste this variable NcML element 
> >>>>>>>>> for your other -->
> >>>>>>>>> |   <!-- variables. -->
> >>>>>>>>> |   <variable name="channel_14_brightness_temperature" shape="lines 
> >>>>>>>>> elements" type="short">
> >>>>>>>>> |     <attribute name="coordinates" value="time pixel_latitude 
> >>>>>>>>> pixel_longitude" />
> >>>>>>>>> |   </variable>
> >>>>>>>>> |   <!-- Now you are ready to do your aggregation, which is the 
> >>>>>>>>> easy part at this point -->
> >>>>>>>>> |   <aggregation dimName="time" type="joinNew">
> >>>>>>>>> |     <variableAgg name="T"/>
> >>>>>>>>> |     <netcdf location="geocatL1.MTSAT-1R.2014044.190900.hdf"/>
> >>>>>>>>> |     <netcdf location="geocatL1.MTSAT-1R.2014044.191900.hdf"/>
> >>>>>>>>> |   </aggregation>
> >>>>>>>>> | </netcdf>
> >>>>>>>>> `----
> >>>>>>>>>
> >>>>>>>>> In summary, this NcML should aggregate over time the two of HDF 
> >>>>>>>>> files
> >>>>>>>>> you provided. You will see only one variable that is aggregated over
> >>>>>>>>> time; `channel_14_brightness_temperature'.  I'll leave it up to you 
> >>>>>>>>> to
> >>>>>>>>> do the others though do not hesitate to ask for help.
> >>>>>>>>>
> >>>>>>>>> Also, if interested, there are also fancier tricks we can do here 
> >>>>>>>>> such
> >>>>>>>>> as NcML can be smart enough to extract the times from the file 
> >>>>>>>>> names.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>>
> >>>>>>>>> Unidata IDV Support
> >>>>>>>>>
> >>>>>>>>>> Ok sounds promising. Here is the link to another file, which 
> >>>>>>>>>> should differ only in time — 
> >>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRDVyczU3N2NMbFU/view?usp=sharing
> >>>>>>>>>>    -Alexa
> >>>>>>>>>>
> >>>>>>>>>>> On Mar 30, 2016, at 4:49 PM, Unidata IDV Support <address@hidden> 
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Alexa,
> >>>>>>>>>>>
> >>>>>>>>>>> Indeed we may be able to patch things up with a bit of ncml. I 
> >>>>>>>>>>> only found one
> >>>>>>>>>>> file at the drive.google location. Would it be possible to see 
> >>>>>>>>>>> more so that we
> >>>>>>>>>>> have something to aggregate?
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>>
> >>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>
> >>>>>>>>>>>> Hello IDV Support,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am working on reading in a number of HDF files using 
> >>>>>>>>>>>> ‘aggregate grids by time’ or similar. I am able to read the 
> >>>>>>>>>>>> files in individually without a problem, but it looks like the 
> >>>>>>>>>>>> time variable is not recognized. The error message is "Grid data 
> >>>>>>>>>>>> source failed making data set….Dimension time does not exist”. 
> >>>>>>>>>>>> Any ideas for how to specify the time variable? Could this be 
> >>>>>>>>>>>> achieved with an .ncml script?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Example file is located here: 
> >>>>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRFhkRVM2aVVRVGc/view?usp=sharing
> >>>>>>>>>>>>
> >>>>>>>>>>>> All the best
> >>>>>>>>>>>>
> >>>>>>>>>>>> Alexa
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Ticket Details
> >>>>>>>>>>> ===================
> >>>>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>>>> Department: Support IDV
> >>>>>>>>>>> Priority: Normal
> >>>>>>>>>>> Status: Closed
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Ticket Details
> >>>>>>>> ===================
> >>>>>>>> Ticket ID: LPW-471935
> >>>>>>>> Department: Support IDV
> >>>>>>>> Priority: Normal
> >>>>>>>> Status: Closed<alexa.ncml>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Ticket Details
> >>>>>> ===================
> >>>>>> Ticket ID: LPW-471935
> >>>>>> Department: Support IDV
> >>>>>> Priority: Normal
> >>>>>> Status: Closed
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: LPW-471935
> >>> Department: Support IDV
> >>> Priority: Normal
> >>> Status: Open
> >>>
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: LPW-471935
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed<alexa.ncml>
> 
> 


Ticket Details
===================
Ticket ID: LPW-471935
Department: Support IDV
Priority: Normal
Status: Closed