[IDV #DGO-203144]: Point data questions

Hi Hans-

> Have done a quick test on builds from May 8 and 9.
> 
> First issue: The Point Data windows is still not being re-drawn (filled in) 
> after having applied the saved preferences. I have to resize the window to 
> get that to happen.

Okay, I'm having issues recreating this on my machine.  I'll test on
on others.  I did fix another bug where the skip lines weren't being
done.  Before I had to apply the preference twice.

> Second issue: The original csv file still generates the same error.

Still having trouble getting this to fail consistently, but will  keep
plugging away on it.

> But the good news is that other csv files, that didn't work before, now works!

At least there's some progress!

> However, a file downloaded today ("metars.current090509.csv") comes up with a 
> new error (see below, "samples values may not be missing", is that a bad 
> thing i.e. something NOT missing?).

It's coming from the low level VisAD code where you can't have a missing value
(NaN) for data when trying to create the set of times.  I meant to fix that in
the last round of changes.

The problem is that one ob:

VOTV 090740Z 31006KT 6000 SCT018 FEW025CB(W,N,E) BKN100 31/24 Q1008 NOSIG,VOTV,2
009-05-09T07:40:00Z,8.47,76.95,31.0,24.0,310,6,,3.73,29.763779,,,,,,TRUE,,,,,SCT
,1800,BKN,10000,,,,,MVFR,,,,,,,,,,,,METAR,8.0

has commas in the raw observation (W,N,E) which throws off the parser.  We don't
have any way around that, so I just skip the ob now.

> And I sometimes still have the problem with different behavior (different 
> errors) the first and the second time I read a specific csv file. The file 
> that caused the error below ("samples values may not be missing") actually 
> generated another error the first time I read it (first time error was the 
> one I reported before, i.e. "An error occurred reading column number:6" (as 
> per my previous email)

Again, I'll keep trying to reproduce this, but it's been tough to replicate.  I 
know
it is an issue and I think it's a timing issue way down in the low level code 
where it
tries to create the variables.

The error below will be fixed in the next release:

> visad.SetException: Gridded1DDoubleSet: samples values may not be missing
> at visad.Gridded1DDoubleSet.<init>(Gridded1DDoubleSet.java:120)
> at visad.Gridded1DDoubleSet.<init>(Gridded1DDoubleSet.java:89)
> at visad.DateTime.makeTimeSet(DateTime.java:537)
> at 
> ucar.unidata.data.point.PointObFactory.makeTimeSequenceOfPointObs(PointObFactory.java:252)
> at 
> ucar.unidata.data.point.PointObFactory.makeTimeSequenceOfPointObs(PointObFactory.java:178)
> at 
> ucar.unidata.data.point.PointObFactory.makeTimeSequenceOfPointObs(PointObFactory.java:155)
> at 
> ucar.unidata.data.point.PointDataSource.getDataInner(PointDataSource.java:1121)
> at 
> ucar.unidata.data.point.TextPointDataSource.getDataInner(TextPointDataSource.java:1667)
> at ucar.unidata.data.DataSourceImpl.getData(DataSourceImpl.java:2122)
> at ucar.unidata.data.DirectDataChoice.getData(DirectDataChoice.java:332)
> at ucar.unidata.data.DataChoice.getData(DataChoice.java:634)
> at ucar.unidata.data.DataInstance.getData(DataInstance.java:239)
> at ucar.unidata.data.DataInstance.getData(DataInstance.java:203)
> at ucar.unidata.data.grid.GridDataInstance.init(GridDataInstance.java:204)
> at ucar.unidata.data.grid.GridDataInstance.<init>(GridDataInstance.java:161)
> at ucar.unidata.data.grid.GridDataInstance.<init>(GridDataInstance.java:142)
> at 
> ucar.unidata.idv.control.PlanViewControl.doMakeDataInstance(PlanViewControl.java:1274)
> at 
> ucar.unidata.idv.control.DisplayControlImpl.initializeDataInstance(DisplayControlImpl.java:2968)
> at 
> ucar.unidata.idv.control.DisplayControlImpl.setData(DisplayControlImpl.java:2950)
> at ucar.unidata.idv.control.PlanViewControl.setData(PlanViewControl.java:559)
> at 
> ucar.unidata.idv.control.ContourPlanViewControl.setData(ContourPlanViewControl.java:112)
> at ucar.unidata.idv.control.PlanViewControl.init(PlanViewControl.java:425)
> at 
> ucar.unidata.idv.control.DisplayControlImpl.init(DisplayControlImpl.java:1269)
> at 
> ucar.unidata.idv.control.DisplayControlImpl.init(DisplayControlImpl.java:978)
> at ucar.unidata.idv.ControlDescriptor.initControl(ControlDescriptor.java:979)
> at ucar.unidata.idv.ControlDescriptor$1.run(ControlDescriptor.java:904)
> at ucar.unidata.util.Misc$3.run(Misc.java:1060)
> 
> 
> I realize I was supposed to provide you with sample files and headers before, 
> but haven't. Sorry about that. Doing it now. The attached csv file is the one 
> I described in the previous paragraph. The simple header file is also 
> attached, as is the original file I tested ("metars.current.csv", i.e. the 
> generating only the "An error occurred reading column number:6").

Thanks.  Of course, there is no way to import the header, so we're going to add 
that
capability today.

> So, maybe all this is just bad data symptoms? Is there a way to find out 
> without having to read the files manually, checking every value?

Some yes, some no.  There will be an error printed out when we can't parse a 
observation
correctly but we'll just ignore the observation.

> And, yes, GRIB issue seems OK.

At least we got one thing right. ;-)  Thanks for your patience and reporting 
these
issues.

Don

> -----Original Message-----
> From: Unidata IDV Support [mailto:address@hidden]
> Sent: Thursday, May 07, 2009 9:06 PM
> To: address@hidden
> Cc: address@hidden
> Subject: [IDV #DGO-203144]: Point data questions
> 
> Hi Hans-
> 
> I thought I had replied last week, but just found my reply session sitting in
> one of my browser windows.
> 
> > > Just tried today's (Apr 28) build. The data looks much better now!
> > > However, I do get some errors trying to read, and plot, the data.
> >
> > Sorry about this.  Fix one thing, break another. ;-)
> 
> Hopefully we've made more fixes this time.  Please try the latest nightly
> build.
> 
> > > Firstly, in the previous version I defined names and units for the fields 
> > > in the csv file and saved it under its own name as a preference. In the 
> > > previous version (I think), when selecting the saved preference the field 
> > > names popped up in the point data form. That does not happen with the new 
> > > version.
> >
> > If you resize the window slightly, you should see the fields displayed. 
> > We're
> > working on a fix for this.  At this time, you'll need to scroll down to the
> > first observation before applying the old preference.
> 
> Please try this with the latest build and see if the problem still
> exists.
> 
> > > Secondly, the first time I try to read the file (i.e. clicking OK after 
> > > having applied the saved preference def that does not show), I get an 
> > > error message. Trying a second time, however, the data is usually read 
> > > without error messages. Could perhaps have something to do with some data 
> > > missing in the file which results in an uninitialized field in IDV the 
> > > first time the file is read (just my wild guess)?
> >
> > There was a change made to the underlying VisAD units package that is 
> > causing
> > this.  We are trying to track down the problem and hope to have a fix soon.
> 
> We put in some fixes for the units, but I'm not sure the issue below
> is completely resolved.  I haven't been able to reproduce it today with
> the latest build.  However, I was in that same situation yesterday and
> then it popped up again.
> 
> > > The error message shown is as follows:
> > > " There was an error loading the data:
> > > Error creating data source:file.pointtext with: 
> > > E:\WorkingData\Data\Observations\metars.current.csv
> > > Making grid parameters
> > > An error occurred reading column number:6"
> > > Detailed:
> > > " java.lang.IllegalArgumentException: An error occurred reading column 
> > > number:6
> > > at visad.data.text.TextAdapter.readit(TextAdapter.java:990)
> > > at visad.data.text.TextAdapter.<init>(TextAdapter.java:210)
> > > at 
> > > ucar.unidata.data.point.TextPointDataSource.makeObs(TextPointDataSource.java:386)
> > > at 
> > > ucar.unidata.data.point.TextPointDataSource.makeObs(TextPointDataSource.java:316)
> > > at 
> > > ucar.unidata.data.point.TextPointDataSource.getSample(TextPointDataSource.java:1578)
> > > at 
> > > ucar.unidata.data.point.PointDataSource.doMakeDataChoices(PointDataSource.java:900)
> > > at 
> > > ucar.unidata.data.point.TextPointDataSource.doMakeDataChoices(TextPointDataSource.java:1586)
> > > at 
> > > ucar.unidata.data.DataSourceImpl.getDataChoices(DataSourceImpl.java:1729)
> > > at ucar.unidata.data.DataManager.createDataSource(DataManager.java:1484)
> > > at 
> > > ucar.unidata.idv.IntegratedDataViewer.createDataSource(IntegratedDataViewer.java:1954)
> > > at 
> > > ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1871)
> > > at 
> > > ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1805)
> > > at 
> > > ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1778)
> > > at ucar.unidata.idv.chooser.IdvChooser.makeDataSource(IdvChooser.java:839)
> > > at 
> > > ucar.unidata.idv.chooser.FileChooser.selectFilesInner(FileChooser.java:646)
> > > at ucar.unidata.idv.chooser.FileChooser.selectFiles(FileChooser.java:515)
> > > at 
> > > ucar.unidata.idv.chooser.FileChooser.doLoadInThread(FileChooser.java:537)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at 
> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > at 
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at ucar.unidata.util.Misc$2.run(Misc.java:1032)
> > > at ucar.unidata.util.Misc$3.run(Misc.java:1060)
> > > "
> > >
> > > Thirdly:
> > > Plotting the data usually generates an error with no plot being shown. 
> > > Trying with a much smaller subset of the file works though. The original 
> > > file contains about 3800 lines of METAR data, each consisting of about 44 
> > > data fields. The file size of that file is about 700 kB. Is there a size 
> > > restriction in IDV that might be the cause of this error perhaps?
> >
> > There shouldn't be.  I wonder if there is bad data causing a problem.  Could
> > you send me two things:
> >
> > - the file you are trying to use
> > - your preference for reading the file.  You can export this by reading in
> > the data and under the Preferences button, select Export Header.
> >
> > Also, which field are you trying to contour?
> 
> The issue here is that the objective analysis was breaking when there was only
> one station.  There are two fixes to this:
> 
> - I put in a check to have at least 3 stations before trying to do the 
> analysis
> - Since the times in the observations are not grouped, you will get lots of 
> times,
> each with as few as one or more, it would be better to group the observations
> into manageable chunks of time.  Bring up the data source properties and set 
> the
> bin size and the round to times.  I tried a bin size of 1 hour and a round to 
> of
> on the hour.
> 
> The latter avoids trying to do an objective analysis on too few stations, but
> my fix will prevent that as well.
> 
> The latest nightly build should also have the fix for your grib problem.
> 
> Don Murray
> 
> Ticket Details
> ===================
> Ticket ID: DGO-203144
> Department: Support IDV
> Priority: Normal
> Status: Open
> 
> 
> 


Ticket Details
===================
Ticket ID: DGO-203144
Department: Support IDV
Priority: Normal
Status: Open