[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[IDV #LPW-471935]: Plotting .hdf files in IDV
- Subject: [IDV #LPW-471935]: Plotting .hdf files in IDV
- Date: Tue, 05 Apr 2016 09:16:47 -0600
Excellent. Thank you.
> Feel free to use this example.
>
> Best Regards,
> Mike
>
>
> Michael J. Pavolonis, PhD
> Physical Scientist (NOAA/NESDIS)
> Phone: +1.608.263.9597
> Fax: +1.608.262.5974
> address@hidden
>
> Duty Station and Mailing Address:
> NOAA Cooperative Institute for Meteorological Satellite Studies
> University of Wisconsin - Madison
> 1225 W. Dayton St., Room 207
> Madison, WI 53706 USA
>
>
>
> > On Apr 4, 2016, at 1:57 PM, Alexa Van Eaton <address@hidden> wrote:
> >
> > Hi there,
> >
> > The person to ask about this would be Mike Pavolonis (Ccd), who created
> > these brightness temperature files. Mike, could you read the message below
> > and provide your thoughts? I’m attaching a screencast of what I’ve done
> > with these files so far. The crosshairs are volcanic lighting detected by
> > WWLLN, brought in through a point text data file.
> >
> > All the best
> >
> > Alexa
> >
> > <Kelut_MTSAT_Lightning.mov>
> >
> >
> >> On Apr 4, 2016, at 11:41 AM, Unidata IDV Support <address@hidden> wrote:
> >>
> >> Alexa,
> >>
> >> Great! I am glad it worked.
> >>
> >> I was contemplating writing a blog entry for our Unidata Developer Blog on
> >> this
> >> experience as it makes a good case study. Would it be OK if I used your
> >> data/ncml for this purpose? It is no problem if you would prefer not to.
> >>
> >> Best,
> >>
> >> Unidata IDV Support
> >>
> >>> Dear IDV Support (and Sean),
> >>>
> >>> This works perfectly. I realized the file naming scheme was actually
> >>> actually giving year + day of the year, so I adjusted the dateFormatMark
> >>> using the syntax in this example
> >>> http://docs.opendap.org/index.php/Dynamic_Aggregation_Tutorial and now
> >>> the dates/times are being read accurately.
> >>>
> >>> As a side note, I noticed that Projections—>Enable Adaptive Resolution is
> >>> on by default in the IDV, which automatically subsamples the data. But
> >>> once that function was unchecked and the data reloaded, everything looks
> >>> good. Big thanks to the both of you.
> >>>
> >>> All the best
> >>>
> >>> Alexa
> >>>
> >>>
> >>>> On Apr 1, 2016, at 12:06 PM, Unidata IDV Support <address@hidden> wrote:
> >>>>
> >>>> Hi Alexa,
> >>>>
> >>>> What I was describing below had nothing to do with the problem. Please
> >>>> see the
> >>>> attached NCML. This file should correctly aggregate the variables over
> >>>> the time dimension, and also it will extract the times from the file
> >>>> name, so
> >>>> you don't have to manually describe them.
> >>>>
> >>>> Let us know how it goes.
> >>>>
> >>>> Best,
> >>>>
> >>>> Unidata IDV Support
> >>>>
> >>>> PS Thanks to my colleague Sean Arms for the NCML lessons :-)
> >>>>
> >>>>> Sure thing, I’ll have a play and keep you posted. All the best, Alexa
> >>>>>
> >>>>>
> >>>>>> On Mar 31, 2016, at 4:21 PM, Unidata IDV Support <address@hidden>
> >>>>>> wrote:
> >>>>>>
> >>>>>> Alexa,
> >>>>>>
> >>>>>> I think I know what the problem is, but don't have time to try before
> >>>>>> I go home.
> >>>>>>
> >>>>>> I think the problem is that we have only changed one variable when we
> >>>>>> should
> >>>>>> have changed them all, at least all those that are listed under the
> >>>>>> Field
> >>>>>> Selector. The IDV may be getting confused on this.
> >>>>>>
> >>>>>> Do you want to give that a shot and let me know what happens?
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> Unidata IDV Support
> >>>>>>
> >>>>>>> Alexa,
> >>>>>>>
> >>>>>>> Yes, thanks for the clarification. (I think the ordering of our
> >>>>>>> messages got
> >>>>>>> jumbled there.)
> >>>>>>>
> >>>>>>> I see what you are saying, and when I load each individual file
> >>>>>>> separately I can
> >>>>>>> see the data are definitely not the same so something is amiss. We'll
> >>>>>>> resume
> >>>>>>> tomorrow.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>>
> >>>>>>> Unidata IDV Support
> >>>>>>>
> >>>>>>>> Hi there — thank you for for your reply. Using your syntax below, I
> >>>>>>>> tried aggregating three files (edited script attached) — different
> >>>>>>>> files this time. Again, the files are read into IDV with the correct
> >>>>>>>> time stamps, and I can toggle between the times, but the strange
> >>>>>>>> thing is the plot does not change with time. It’s like it’s only
> >>>>>>>> plotting one time. -Alexa
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Mar 31, 2016, at 3:46 PM, Unidata IDV Support <address@hidden>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Alexa,
> >>>>>>>>>
> >>>>>>>>> Great! Glad we are making progress and also that you are hanging in
> >>>>>>>>> there. This
> >>>>>>>>> stuff is not so easy.
> >>>>>>>>>
> >>>>>>>>> Replace the current values element with something like this:
> >>>>>>>>>
> >>>>>>>>> <values>0 3 11</values>
> >>>>>>>>>
> >>>>>>>>> But if that is too onerous because you have many file, we will have
> >>>>>>>>> to go via
> >>>>>>>>> the time extraction route. Let me know...
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>>
> >>>>>>>>> Unidata IDV Support
> >>>>>>>>>
> >>>>>>>>>> Hello — this looks great, and I appreciate your help with this.
> >>>>>>>>>> Your ncml script worked for the two test files and the times look
> >>>>>>>>>> right. However, since we have some missing satellite images, not
> >>>>>>>>>> all the files are spaced exactly 10 mins apart. Is there a way to
> >>>>>>>>>> specify time gaps between individual files begin aggregated?
> >>>>>>>>>> Otherwise, perhaps extracting the time from the file name would be
> >>>>>>>>>> good route. -Alexa
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Mar 31, 2016, at 3:29 PM, Unidata IDV Support <address@hidden>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi again,
> >>>>>>>>>>>
> >>>>>>>>>>> The embedded NcML is a bit hard to read so attached, you will find
> >>>>>>>>>>> that same NcML file.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>>
> >>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Alexa,
> >>>>>>>>>>>>
> >>>>>>>>>>>> It is a little tricky but you can fix these problems with a bit
> >>>>>>>>>>>> of
> >>>>>>>>>>>> NcML. To get us going, let's look at the following NcML which
> >>>>>>>>>>>> can be
> >>>>>>>>>>>> read into the IDV and will allow you to view the
> >>>>>>>>>>>> `channel_14_brightness_temperature' over time. This file should
> >>>>>>>>>>>> be
> >>>>>>>>>>>> located next to your `hdf' files. Please read the comments in
> >>>>>>>>>>>> the NcML
> >>>>>>>>>>>> for more information about what is going on here.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ,----
> >>>>>>>>>>>> | <?xml version="1.0" encoding="UTF-8"?>
> >>>>>>>>>>>> | <netcdf
> >>>>>>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
> >>>>>>>>>>>> | <!-- Create a time coordinate variable that will be used for
> >>>>>>>>>>>> the aggregation. -->
> >>>>>>>>>>>> | <variable name="time" type="int" shape="time" >
> >>>>>>>>>>>> | <!-- Let's add some units. Alexa: please make sure these
> >>>>>>>>>>>> units are correct. -->
> >>>>>>>>>>>> | <attribute name="units" value="minutes since 2014-04-04
> >>>>>>>>>>>> 19:09"/>
> >>>>>>>>>>>> | <!-- Specify this variable as a time coordinate -->
> >>>>>>>>>>>> | <attribute name="_CoordinateAxisType" value="Time" />
> >>>>>>>>>>>> | <!-- Increment each file listed below by 10 minutes.
> >>>>>>>>>>>> Alexa, please make sure this -->
> >>>>>>>>>>>> | <!-- assumption about the times is correct. -->
> >>>>>>>>>>>> | <values start="0" increment="10" />
> >>>>>>>>>>>> | </variable>
> >>>>>>>>>>>> | <!-- Specify the coordinates of time lat lon for this one
> >>>>>>>>>>>> variable. Please use -->
> >>>>>>>>>>>> | <!-- this as an example for other variables you are
> >>>>>>>>>>>> interersted in. In other -->
> >>>>>>>>>>>> | <!-- words, you can copy and paste this variable NcML
> >>>>>>>>>>>> element for your other -->
> >>>>>>>>>>>> | <!-- variables. -->
> >>>>>>>>>>>> | <variable name="channel_14_brightness_temperature"
> >>>>>>>>>>>> shape="lines elements" type="short">
> >>>>>>>>>>>> | <attribute name="coordinates" value="time pixel_latitude
> >>>>>>>>>>>> pixel_longitude" />
> >>>>>>>>>>>> | </variable>
> >>>>>>>>>>>> | <!-- Now you are ready to do your aggregation, which is the
> >>>>>>>>>>>> easy part at this point -->
> >>>>>>>>>>>> | <aggregation dimName="time" type="joinNew">
> >>>>>>>>>>>> | <variableAgg name="T"/>
> >>>>>>>>>>>> | <netcdf location="geocatL1.MTSAT-1R.2014044.190900.hdf"/>
> >>>>>>>>>>>> | <netcdf location="geocatL1.MTSAT-1R.2014044.191900.hdf"/>
> >>>>>>>>>>>> | </aggregation>
> >>>>>>>>>>>> | </netcdf>
> >>>>>>>>>>>> `----
> >>>>>>>>>>>>
> >>>>>>>>>>>> In summary, this NcML should aggregate over time the two of HDF
> >>>>>>>>>>>> files
> >>>>>>>>>>>> you provided. You will see only one variable that is aggregated
> >>>>>>>>>>>> over
> >>>>>>>>>>>> time; `channel_14_brightness_temperature'. I'll leave it up to
> >>>>>>>>>>>> you to
> >>>>>>>>>>>> do the others though do not hesitate to ask for help.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Also, if interested, there are also fancier tricks we can do
> >>>>>>>>>>>> here such
> >>>>>>>>>>>> as NcML can be smart enough to extract the times from the file
> >>>>>>>>>>>> names.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Ok sounds promising. Here is the link to another file, which
> >>>>>>>>>>>>> should differ only in time —
> >>>>>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRDVyczU3N2NMbFU/view?usp=sharing
> >>>>>>>>>>>>> -Alexa
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mar 30, 2016, at 4:49 PM, Unidata IDV Support
> >>>>>>>>>>>>>> <address@hidden> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Alexa,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Indeed we may be able to patch things up with a bit of ncml. I
> >>>>>>>>>>>>>> only found one
> >>>>>>>>>>>>>> file at the drive.google location. Would it be possible to see
> >>>>>>>>>>>>>> more so that we
> >>>>>>>>>>>>>> have something to aggregate?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello IDV Support,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I am working on reading in a number of HDF files using
> >>>>>>>>>>>>>>> ‘aggregate grids by time’ or similar. I am able to read the
> >>>>>>>>>>>>>>> files in individually without a problem, but it looks like
> >>>>>>>>>>>>>>> the time variable is not recognized. The error message is
> >>>>>>>>>>>>>>> "Grid data source failed making data set….Dimension time does
> >>>>>>>>>>>>>>> not exist”. Any ideas for how to specify the time variable?
> >>>>>>>>>>>>>>> Could this be achieved with an .ncml script?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Example file is located here:
> >>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRFhkRVM2aVVRVGc/view?usp=sharing
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> All the best
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Alexa
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ticket Details
> >>>>>>>>>>>>>> ===================
> >>>>>>>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>>>>>>> Department: Support IDV
> >>>>>>>>>>>>>> Priority: Normal
> >>>>>>>>>>>>>> Status: Closed
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Ticket Details
> >>>>>>>>>>> ===================
> >>>>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>>>> Department: Support IDV
> >>>>>>>>>>> Priority: Normal
> >>>>>>>>>>> Status: Closed<alexa.ncml>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Ticket Details
> >>>>>>>>> ===================
> >>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>> Department: Support IDV
> >>>>>>>>> Priority: Normal
> >>>>>>>>> Status: Closed
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Ticket Details
> >>>>>> ===================
> >>>>>> Ticket ID: LPW-471935
> >>>>>> Department: Support IDV
> >>>>>> Priority: Normal
> >>>>>> Status: Open
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> Ticket Details
> >>>> ===================
> >>>> Ticket ID: LPW-471935
> >>>> Department: Support IDV
> >>>> Priority: Normal
> >>>> Status: Closed<alexa.ncml>
> >>>
> >>>
> >>
> >>
> >> Ticket Details
> >> ===================
> >> Ticket ID: LPW-471935
> >> Department: Support IDV
> >> Priority: Normal
> >> Status: Closed
> >>
> >
>
>
Ticket Details
===================
Ticket ID: LPW-471935
Department: Support IDV
Priority: Normal
Status: Closed