[IDV #ILT-566886]: vertical profiles with depth, not altitude

> Hi Murray,
> 
> I agree with you 100% that creating TGZ files on the fly is the best way
> to go and will bring it to the NODC leadership's attention.
> 
> Charles
> On 03/18/2012 10:43 AM, Brown, Murray wrote:
> > Gang (sounds right),
> >
> > Charles' other product line is covered by a different exercise at
> > http://marinedataliteracy.org/odv/gtspp_bestnc.htm.  These are TGZ
> > compressed nc profiles all loaded at once into the display program
> > OceanDataView.  The TGZ files are available per-ocean basin, and
> > per-month.
> >
> > But with that approach and (if we had it) with multi-profile NC files,
> > the more important question is how do we reduce the file load to the
> > spatial and temporal envelope of interest?  I know beans about the
> > technology, but it seems like creating TGZ compilations or NC
> > compilations on the fly would be a good bet.  I say this mainly
> > because I'm thinking of our IOC students in Pago Pago who have just
> > slightly better technology than those old phones we used to stick into
> > rubber cradles (yeah, I know you remember Omnet).  The global
> > operational game isn't won until we can offer everybody a reasonable
> > shot at any kind of data.  I'm very very grateful about just this past
> > week's revelations.
> >
> > Murray
> >
> > ----- Original Message ----- From: "Rich Signell" <address@hidden>
> > To: "Charles Sun" <address@hidden>
> > Cc: "Brown, Murray" <address@hidden>;
> > <address@hidden>; <address@hidden>; "Reed, Greg"
> > <address@hidden>
> > Sent: Sunday, March 18, 2012 10:02 AM
> > Subject: Re: PC issues
> >
> >
> > Gang,
> >
> > With the new CF Discrete Sampling Geometry featureTypes implemented in
> > NetCDF-Java Common Data Model, you can save many profiles in a single
> > file, rather than separate files.   For example:
> >
> > http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8372832
> >
> >
> > But I don't know if IDV can read these files and do anything useful
> > with them yet.
> >
> > -Rich

This is the best solution for this kind of dataset, and the IDV can certainly 
handle it better in this approach.


Yuan
> >
> >
> > On Sun, Mar 18, 2012 at 9:29 AM, Charles Sun <address@hidden>
> > wrote:
> >> Hi Murray,
> >>
> >> It is a typical issue for using IDV or IDV-like applications for reading
> >> large number of small size files, either from local hard drive or over
> >> Internet.
> >>
> >> One other maybe unrelated issue that I would like to bring to your
> >> attention
> >> is a jar file, called gtspp.jar (attached) suggested by Yuen for
> >> using IDV
> >> with the GTSPP data, has two files, datasource.xml and gstpp.ncml. I
> >> don't
> >> know too much about IDV. Should the filename, gstpp.ncml, be
> >> gtspp.ncml or
> >> it does not matter?
> >>
> >> Regards,
> >>
> >> Charles
> >>
> >> On 03/18/2012 07:38 AM, Brown, Murray wrote:
> >>>
> >>> Dear Folks,
> >>>
> >>> Saturday morning I began to experience some very strange PC behavior
> >>> (no,
> >>> I didn't suddenly become a Republican). My laptop, admittedly an
> >>> ancient
> >>> beast (5 yrs, a Compaq) started running extremely slowly and just
> >>> died on
> >>> big programs. With no online shopping, wild browsing or other usual
> >>> suspects for infection. Checking my memory usage I found that one
> >>> instance
> >>> of the notorious svchost.exe was steadily growing by the minute, and
> >>> had
> >>> reached 1.2 Gb. Yep. Rebooting just started the growth cycle again,
> >>> and I
> >>> could see the file grow about 1 Mb per second with nothing running.
> >>> If I
> >>> stop that particular svchost.exe (there are usually several) then
> >>> some other
> >>> stuff can run, but not IDV. It seems, and the Microsoft technical
> >>> confirmed, that the runaway instance of svchost.exe is the one that
> >>> usually
> >>> gets associated with IDV (a little Google searching turned up that
> >>> svchost.exe is the controller for dll's called by programs, among other
> >>> functions). [Other instances of svchost.exe remain unaffected and
> >>> seemingly
> >>> benign.] But it starts the weird behavior always, even when IDV isn't
> >>> started at all. I'm nuts, my PC is toast, or something weird was
> >>> alive and
> >>> well in the resources.
> >>>
> >>> The MS technician said they could clean svchost.exe, but they didn't
> >>> know
> >>> what was wrong with it and no virus checker (they used 3) could
> >>> identify
> >>> what it was. "It can be your old PC, or it can be the files you've
> >>> worked
> >>> with, but we just don't know." I've been working only with GTSPP
> >>> files and
> >>> IDV for the past 4 days or so (including the new jar), as you've all
> >>> seen.
> >>> But I'm up and running now, and the main svchost.exe is remaining
> >>> stable at
> >>> 28Kb. Bullet dodged.
> >>>
> >>> CHARLES: Sorry I sent that note about your servers, because now I
> >>> realize
> >>> it could have been the very beginning of my troubles. IDV loads
> >>> http://data.nodc.noaa.gov/thredds/catalog/gtspp/realtime/2012/catalog.xml
> >>> in
> >>> about 2 mins. now and all seems fine.
> >>>
> >>> Murray
> >>>
> >>>
> >>> ----- Original Message ----- From: "Unidata IDV Support"
> >>> <address@hidden>
> >>> To: <address@hidden>
> >>> Cc: <address@hidden>; <address@hidden>;
> >>> <address@hidden>; <address@hidden>;
> >>> <address@hidden>
> >>> Sent: Friday, March 16, 2012 4:12 PM
> >>> Subject: [IDV #ILT-566886]: vertical profiles with depth, not altitude
> >>>
> >>>
> >>>>> Wow. I am so grateful. But could you change the code/name so it is
> >>>>> GTSPP?
> >>>>>
> >>>>
> >>>> See the attached.
> >>>>
> >>>> Y
> >>>>>
> >>>>> ----- Original Message -----
> >>>>> From: "Unidata IDV Support" <address@hidden>
> >>>>> To: <address@hidden>
> >>>>> Cc: <address@hidden>; <address@hidden>;
> >>>>> <address@hidden>;
> >>>>> <address@hidden>; <address@hidden>;
> >>>>> <address@hidden>
> >>>>> Sent: Friday, March 16, 2012 1:11 PM
> >>>>> Subject: [IDV #ILT-566886]: vertical profiles with depth, not
> >>>>> altitude
> >>>>>
> >>>>>
> >>>>> >> Yuan,
> >>>>> >>
> >>>>> >> I am aware of it. The typo was in my format conversion program
> >>>>> to >>
> >>>>> >> create
> >>>>> >> the physical files.
> >>>>> >>
> >>>>> >> By the way, all files in the
> >>>>> >> http://data.nodc.noaa.gov/gtspp/realtime/2012/03 are the
> >>>>> corrected >> >>
> >>>>> >> ones.
> >>>>> >>
> >>>>> >> Thanks.
> >>>>> >>
> >>>>> >> Charles
> >>>>> >> On 3/16/2012 12:50 PM, Unidata IDV Support wrote:
> >>>>> >> >> Yuan,
> >>>>> >> >>
> >>>>> >> >> This is Charles Sun from US National Oceanographic Data Center.
> >>>>> >> >>
> >>>>> >> >> We are replacing the old GTSPP netcdf conventions, currently
> >>>>> >> >> >> >>
> >>>>> >> >> located
> >>>>> >> >> in
> >>>>> >> >> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/, by moving all
> >>>>> >> >> files
> >>>>> >> >> below the folder, netcdf4.0, one level up. This will not
> >>>>> happen >> >> >>
> >>>>> >> >> >> till
> >>>>> >> >> late next week.
> >>>>> >> >>
> >>>>> >> >> I will suggest use
> >>>>> >> >>
> >>>>> >> >>
> >>>>> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/netcdf4.0/indian/2012/01/gtspp_13237135_te_111.nc
> >>>>> >> >> for testing purpose. In addition, please be advised that a
> >>>>> typo >> >> in
> >>>>> >> >> >> >> the
> >>>>> >> >> z-variable attribute was found by Rich, which causes the
> >>>>> vertical
> >>>>> >> >> profile displayed up-side down.
> >>>>> >> > Charles,
> >>>>> >> > I checked the physical file, and the typo is in the file. Not
> >>>>> >> > sure how you plan to fix it.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Yuan
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Murray,
> >>>>> >> > See the attached jar file, this is a plugin created for this
> >>>>> >> > kind of typo files located locally or remotely. Please
> >>>>> install it
> >>>>> >> > >> > and
> >>>>> >> > restart the IDV, when you load the dataset, you need to
> >>>>> select the
> >>>>> >> > >> > data
> >>>>> >> > source type as "GSTPP data file". You should get what you
> >>>>> wanted.
> >>>>> >> >
> >>>>> >
> >>>>> > Sorry I forgot to add the attached file in my last email.
> >>>>> >
> >>>>> >
> >>>>> > Yuan
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Yuan
> >>>>> >> >> However, the typo has been corrected, again all files will
> >>>>> be >>
> >>>>> >> >> >> updated
> >>>>> >> >> by
> >>>>> >> >> late next week.
> >>>>> >> >>
> >>>>> >> >> Sorry for any inconvenience.
> >>>>> >> >>
> >>>>> >> >> Regards,
> >>>>> >> >>
> >>>>> >> >> Charles
> >>>>> >> >> On 3/16/2012 12:17 PM, Unidata IDV Support wrote:
> >>>>> >> >>>> Yuan,
> >>>>> >> >>>>
> >>>>> >> >>>> Please take a look at the attached figure (#1), which
> >>>>> doesn't >> >>>> >>
> >>>>> >> >>>> >>>> seem
> >>>>> >> >>>> like
> >>>>> >> >>>> much, but I'm so proud of it and so thankful to you and
> >>>>> Rich >>
> >>>>> >> >>>> >>>> Signell
> >>>>> >> >>>> for
> >>>>> >> >>>> helping me with all the tricks involved. It is based on the
> >>>>> >> >>>> "corrected"
> >>>>> >> >>>> NetCDF profile that Charles Sun provided (consisting of
> >>>>> only a
> >>>>> >> >>>> >> >>>> few
> >>>>> >> >>>> points).
> >>>>> >> >>>> But eventually, we could make these color-coded profiles
> >>>>> for >>
> >>>>> >> >>>> >>>> groups
> >>>>> >> >>>> of GTSPP
> >>>>> >> >>>> stations, grabbed online through THREDDS catalogs.
> >>>>> >> >>> Murray,
> >>>>> >> >>> The physical file (13237135.nc) is corrected from the
> >>>>> >> >>> beginning. The problem is the THREDDS server need to be
> >>>>> updated
> >>>>> >> >>> (I
> >>>>> >> >>> will check with TDS developer to make sure). There are two
> >>>>> URLs
> >>>>> >> >>> >> >>> here
> >>>>> >> >>> and confused me:
> >>>>> >> >>>
> >>>>> >> >>>
> >>>>> >> >>>
> >>>>> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/indian/2012/01/13237135.nc
> >>>>>
> >>>>> >> >>>
> >>>>> >> >>> and
> >>>>> >> >>>
> >>>>> >> >>>
> >>>>> >> >>>
> >>>>> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/netcdf4.0/indian/2012/01/gtspp_13237135_te_111.nc
> >>>>> >> >>>
> >>>>> >> >>>
> >>>>> >> >>> Which one are you planning to use?
> >>>>> >> >>>
> >>>>> >> >>>
> >>>>> >> >>> Yuan
> >>>>> >> >>>> This will be, for us in the water world, a huge step, in
> >>>>> terms
> >>>>> >> >>>> of
> >>>>> >> >>>> efficiency, potential for synthesis, and compatibility
> >>>>> with >> >>>> some
> >>>>> >> >>>> training
> >>>>> >> >>>> we're already doing. The hope (watch out Charles, here
> >>>>> comes my
> >>>>> >> >>>> soap box)
> >>>>> >> >>>> is that we'll have user-defined delivery of spatially and >>
> >>>>> >> >>>> >>>> temporally
> >>>>> >> >>>> selected individual NetCDF profiles. When the files have the
> >>>>> >> >>>> "positive"
> >>>>> >> >>>> syntax correction, then the use of 0 to -160 depths gives
> >>>>> >> >>>> really
> >>>>> >> >>>> FINE
> >>>>> >> >>>> figures. [I'm using Ocean Data View for mega-collections
> >>>>> based
> >>>>> >> >>>> >> >>>> on
> >>>>> >> >>>> Charles'
> >>>>> >> >>>> TGZ compressions...that's a different matter.]
> >>>>> >> >>>>
> >>>>> >> >>>> The second figure is one of the existing NC profiles on the
> >>>>> >> >>>> GTSPP
> >>>>> >> >>>> server, to
> >>>>> >> >>>> show you how much better they begin to look with more data >>
> >>>>> >> >>>> >>>> points.
> >>>>> >> >>>> That
> >>>>> >> >>>> figure comes directly from IDV/THREDDS, one click, without
> >>>>> any
> >>>>> >> >>>> >> >>>> ncml
> >>>>> >> >>>> fix. So
> >>>>> >> >>>> it's still upside down. I'll need some IDV help on applying a
> >>>>> >> >>>> layout that
> >>>>> >> >>>> makes the dots larger (perhaps scaled to temp); they ARE
> >>>>> >> >> >>>> >>>>
> >>>>> >> >>>> currently
> >>>>> >> >>>> colored
> >>>>> >> >>>> by temp, but it's not easy to see on these small figs.
> >>>>> >> >>>>
> >>>>> >> >>>> I really hope this dialogue, made necessary by my
> >>>>> blundering >>
> >>>>> >> >>>> >>>> around,
> >>>>> >> >>>> might
> >>>>> >> >>>> continue and lead to good IDV/UNIDATA-GTSPP/NODC discussions.
> >>>>> >> >>>> It
> >>>>> >> >>>> seems to
> >>>>> >> >>>> me that a lot could be accomplished.
> >>>>> >> >>>>
> >>>>> >> >>>> Murray
> >>>>> >> >>>>
> >>>>> >> >>>>
> >>>>> >> >>>
> >>>>> >> >>>
> >>>>> >> >>> Ticket Details
> >>>>> >> >>> ===================
> >>>>> >> >>> Ticket ID: ILT-566886
> >>>>> >> >>> Department: Support IDV
> >>>>> >> >>> Priority: Normal
> >>>>> >> >>> Status: Open
> >>>>> >> >>>
> >>>>> >> >>
> >>>>> >> >
> >>>>> >> > Ticket Details
> >>>>> >> > ===================
> >>>>> >> > Ticket ID: ILT-566886
> >>>>> >> > Department: Support IDV
> >>>>> >> > Priority: Normal
> >>>>> >> > Status: Closed
> >>>>> >> >
> >>>>> >>
> >>>>> >>
> >>>>> >
> >>>>> >
> >>>>> > Ticket Details
> >>>>> > ===================
> >>>>> > Ticket ID: ILT-566886
> >>>>> > Department: Support IDV
> >>>>> > Priority: Normal
> >>>>> > Status: Closed
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> Ticket Details
> >>>> ===================
> >>>> Ticket ID: ILT-566886
> >>>> Department: Support IDV
> >>>> Priority: Normal
> >>>> Status: Closed
> >>>
> >>>
> >>
> >
> >
> >
> 
> 


Ticket Details
===================
Ticket ID: ILT-566886
Department: Support IDV
Priority: Normal
Status: Closed