This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
> Hi Murray, > > I agree with you 100% that creating TGZ files on the fly is the best way > to go and will bring it to the NODC leadership's attention. > > Charles > On 03/18/2012 10:43 AM, Brown, Murray wrote: > > Gang (sounds right), > > > > Charles' other product line is covered by a different exercise at > > http://marinedataliteracy.org/odv/gtspp_bestnc.htm. These are TGZ > > compressed nc profiles all loaded at once into the display program > > OceanDataView. The TGZ files are available per-ocean basin, and > > per-month. > > > > But with that approach and (if we had it) with multi-profile NC files, > > the more important question is how do we reduce the file load to the > > spatial and temporal envelope of interest? I know beans about the > > technology, but it seems like creating TGZ compilations or NC > > compilations on the fly would be a good bet. I say this mainly > > because I'm thinking of our IOC students in Pago Pago who have just > > slightly better technology than those old phones we used to stick into > > rubber cradles (yeah, I know you remember Omnet). The global > > operational game isn't won until we can offer everybody a reasonable > > shot at any kind of data. I'm very very grateful about just this past > > week's revelations. > > > > Murray > > > > ----- Original Message ----- From: "Rich Signell" <address@hidden> > > To: "Charles Sun" <address@hidden> > > Cc: "Brown, Murray" <address@hidden>; > > <address@hidden>; <address@hidden>; "Reed, Greg" > > <address@hidden> > > Sent: Sunday, March 18, 2012 10:02 AM > > Subject: Re: PC issues > > > > > > Gang, > > > > With the new CF Discrete Sampling Geometry featureTypes implemented in > > NetCDF-Java Common Data Model, you can save many profiles in a single > > file, rather than separate files. For example: > > > > http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8372832 > > > > > > But I don't know if IDV can read these files and do anything useful > > with them yet. > > > > -Rich This is the best solution for this kind of dataset, and the IDV can certainly handle it better in this approach. Yuan > > > > > > On Sun, Mar 18, 2012 at 9:29 AM, Charles Sun <address@hidden> > > wrote: > >> Hi Murray, > >> > >> It is a typical issue for using IDV or IDV-like applications for reading > >> large number of small size files, either from local hard drive or over > >> Internet. > >> > >> One other maybe unrelated issue that I would like to bring to your > >> attention > >> is a jar file, called gtspp.jar (attached) suggested by Yuen for > >> using IDV > >> with the GTSPP data, has two files, datasource.xml and gstpp.ncml. I > >> don't > >> know too much about IDV. Should the filename, gstpp.ncml, be > >> gtspp.ncml or > >> it does not matter? > >> > >> Regards, > >> > >> Charles > >> > >> On 03/18/2012 07:38 AM, Brown, Murray wrote: > >>> > >>> Dear Folks, > >>> > >>> Saturday morning I began to experience some very strange PC behavior > >>> (no, > >>> I didn't suddenly become a Republican). My laptop, admittedly an > >>> ancient > >>> beast (5 yrs, a Compaq) started running extremely slowly and just > >>> died on > >>> big programs. With no online shopping, wild browsing or other usual > >>> suspects for infection. Checking my memory usage I found that one > >>> instance > >>> of the notorious svchost.exe was steadily growing by the minute, and > >>> had > >>> reached 1.2 Gb. Yep. Rebooting just started the growth cycle again, > >>> and I > >>> could see the file grow about 1 Mb per second with nothing running. > >>> If I > >>> stop that particular svchost.exe (there are usually several) then > >>> some other > >>> stuff can run, but not IDV. It seems, and the Microsoft technical > >>> confirmed, that the runaway instance of svchost.exe is the one that > >>> usually > >>> gets associated with IDV (a little Google searching turned up that > >>> svchost.exe is the controller for dll's called by programs, among other > >>> functions). [Other instances of svchost.exe remain unaffected and > >>> seemingly > >>> benign.] But it starts the weird behavior always, even when IDV isn't > >>> started at all. I'm nuts, my PC is toast, or something weird was > >>> alive and > >>> well in the resources. > >>> > >>> The MS technician said they could clean svchost.exe, but they didn't > >>> know > >>> what was wrong with it and no virus checker (they used 3) could > >>> identify > >>> what it was. "It can be your old PC, or it can be the files you've > >>> worked > >>> with, but we just don't know." I've been working only with GTSPP > >>> files and > >>> IDV for the past 4 days or so (including the new jar), as you've all > >>> seen. > >>> But I'm up and running now, and the main svchost.exe is remaining > >>> stable at > >>> 28Kb. Bullet dodged. > >>> > >>> CHARLES: Sorry I sent that note about your servers, because now I > >>> realize > >>> it could have been the very beginning of my troubles. IDV loads > >>> http://data.nodc.noaa.gov/thredds/catalog/gtspp/realtime/2012/catalog.xml > >>> in > >>> about 2 mins. now and all seems fine. > >>> > >>> Murray > >>> > >>> > >>> ----- Original Message ----- From: "Unidata IDV Support" > >>> <address@hidden> > >>> To: <address@hidden> > >>> Cc: <address@hidden>; <address@hidden>; > >>> <address@hidden>; <address@hidden>; > >>> <address@hidden> > >>> Sent: Friday, March 16, 2012 4:12 PM > >>> Subject: [IDV #ILT-566886]: vertical profiles with depth, not altitude > >>> > >>> > >>>>> Wow. I am so grateful. But could you change the code/name so it is > >>>>> GTSPP? > >>>>> > >>>> > >>>> See the attached. > >>>> > >>>> Y > >>>>> > >>>>> ----- Original Message ----- > >>>>> From: "Unidata IDV Support" <address@hidden> > >>>>> To: <address@hidden> > >>>>> Cc: <address@hidden>; <address@hidden>; > >>>>> <address@hidden>; > >>>>> <address@hidden>; <address@hidden>; > >>>>> <address@hidden> > >>>>> Sent: Friday, March 16, 2012 1:11 PM > >>>>> Subject: [IDV #ILT-566886]: vertical profiles with depth, not > >>>>> altitude > >>>>> > >>>>> > >>>>> >> Yuan, > >>>>> >> > >>>>> >> I am aware of it. The typo was in my format conversion program > >>>>> to >> > >>>>> >> create > >>>>> >> the physical files. > >>>>> >> > >>>>> >> By the way, all files in the > >>>>> >> http://data.nodc.noaa.gov/gtspp/realtime/2012/03 are the > >>>>> corrected >> >> > >>>>> >> ones. > >>>>> >> > >>>>> >> Thanks. > >>>>> >> > >>>>> >> Charles > >>>>> >> On 3/16/2012 12:50 PM, Unidata IDV Support wrote: > >>>>> >> >> Yuan, > >>>>> >> >> > >>>>> >> >> This is Charles Sun from US National Oceanographic Data Center. > >>>>> >> >> > >>>>> >> >> We are replacing the old GTSPP netcdf conventions, currently > >>>>> >> >> >> >> > >>>>> >> >> located > >>>>> >> >> in > >>>>> >> >> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/, by moving all > >>>>> >> >> files > >>>>> >> >> below the folder, netcdf4.0, one level up. This will not > >>>>> happen >> >> >> > >>>>> >> >> >> till > >>>>> >> >> late next week. > >>>>> >> >> > >>>>> >> >> I will suggest use > >>>>> >> >> > >>>>> >> >> > >>>>> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/netcdf4.0/indian/2012/01/gtspp_13237135_te_111.nc > >>>>> >> >> for testing purpose. In addition, please be advised that a > >>>>> typo >> >> in > >>>>> >> >> >> >> the > >>>>> >> >> z-variable attribute was found by Rich, which causes the > >>>>> vertical > >>>>> >> >> profile displayed up-side down. > >>>>> >> > Charles, > >>>>> >> > I checked the physical file, and the typo is in the file. Not > >>>>> >> > sure how you plan to fix it. > >>>>> >> > > >>>>> >> > > >>>>> >> > Yuan > >>>>> >> > > >>>>> >> > > >>>>> >> > Murray, > >>>>> >> > See the attached jar file, this is a plugin created for this > >>>>> >> > kind of typo files located locally or remotely. Please > >>>>> install it > >>>>> >> > >> > and > >>>>> >> > restart the IDV, when you load the dataset, you need to > >>>>> select the > >>>>> >> > >> > data > >>>>> >> > source type as "GSTPP data file". You should get what you > >>>>> wanted. > >>>>> >> > > >>>>> > > >>>>> > Sorry I forgot to add the attached file in my last email. > >>>>> > > >>>>> > > >>>>> > Yuan > >>>>> >> > > >>>>> >> > > >>>>> >> > Yuan > >>>>> >> >> However, the typo has been corrected, again all files will > >>>>> be >> > >>>>> >> >> >> updated > >>>>> >> >> by > >>>>> >> >> late next week. > >>>>> >> >> > >>>>> >> >> Sorry for any inconvenience. > >>>>> >> >> > >>>>> >> >> Regards, > >>>>> >> >> > >>>>> >> >> Charles > >>>>> >> >> On 3/16/2012 12:17 PM, Unidata IDV Support wrote: > >>>>> >> >>>> Yuan, > >>>>> >> >>>> > >>>>> >> >>>> Please take a look at the attached figure (#1), which > >>>>> doesn't >> >>>> >> > >>>>> >> >>>> >>>> seem > >>>>> >> >>>> like > >>>>> >> >>>> much, but I'm so proud of it and so thankful to you and > >>>>> Rich >> > >>>>> >> >>>> >>>> Signell > >>>>> >> >>>> for > >>>>> >> >>>> helping me with all the tricks involved. It is based on the > >>>>> >> >>>> "corrected" > >>>>> >> >>>> NetCDF profile that Charles Sun provided (consisting of > >>>>> only a > >>>>> >> >>>> >> >>>> few > >>>>> >> >>>> points). > >>>>> >> >>>> But eventually, we could make these color-coded profiles > >>>>> for >> > >>>>> >> >>>> >>>> groups > >>>>> >> >>>> of GTSPP > >>>>> >> >>>> stations, grabbed online through THREDDS catalogs. > >>>>> >> >>> Murray, > >>>>> >> >>> The physical file (13237135.nc) is corrected from the > >>>>> >> >>> beginning. The problem is the THREDDS server need to be > >>>>> updated > >>>>> >> >>> (I > >>>>> >> >>> will check with TDS developer to make sure). There are two > >>>>> URLs > >>>>> >> >>> >> >>> here > >>>>> >> >>> and confused me: > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> > >>>>> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/indian/2012/01/13237135.nc > >>>>> > >>>>> >> >>> > >>>>> >> >>> and > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> > >>>>> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/netcdf4.0/indian/2012/01/gtspp_13237135_te_111.nc > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> Which one are you planning to use? > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> Yuan > >>>>> >> >>>> This will be, for us in the water world, a huge step, in > >>>>> terms > >>>>> >> >>>> of > >>>>> >> >>>> efficiency, potential for synthesis, and compatibility > >>>>> with >> >>>> some > >>>>> >> >>>> training > >>>>> >> >>>> we're already doing. The hope (watch out Charles, here > >>>>> comes my > >>>>> >> >>>> soap box) > >>>>> >> >>>> is that we'll have user-defined delivery of spatially and >> > >>>>> >> >>>> >>>> temporally > >>>>> >> >>>> selected individual NetCDF profiles. When the files have the > >>>>> >> >>>> "positive" > >>>>> >> >>>> syntax correction, then the use of 0 to -160 depths gives > >>>>> >> >>>> really > >>>>> >> >>>> FINE > >>>>> >> >>>> figures. [I'm using Ocean Data View for mega-collections > >>>>> based > >>>>> >> >>>> >> >>>> on > >>>>> >> >>>> Charles' > >>>>> >> >>>> TGZ compressions...that's a different matter.] > >>>>> >> >>>> > >>>>> >> >>>> The second figure is one of the existing NC profiles on the > >>>>> >> >>>> GTSPP > >>>>> >> >>>> server, to > >>>>> >> >>>> show you how much better they begin to look with more data >> > >>>>> >> >>>> >>>> points. > >>>>> >> >>>> That > >>>>> >> >>>> figure comes directly from IDV/THREDDS, one click, without > >>>>> any > >>>>> >> >>>> >> >>>> ncml > >>>>> >> >>>> fix. So > >>>>> >> >>>> it's still upside down. I'll need some IDV help on applying a > >>>>> >> >>>> layout that > >>>>> >> >>>> makes the dots larger (perhaps scaled to temp); they ARE > >>>>> >> >> >>>> >>>> > >>>>> >> >>>> currently > >>>>> >> >>>> colored > >>>>> >> >>>> by temp, but it's not easy to see on these small figs. > >>>>> >> >>>> > >>>>> >> >>>> I really hope this dialogue, made necessary by my > >>>>> blundering >> > >>>>> >> >>>> >>>> around, > >>>>> >> >>>> might > >>>>> >> >>>> continue and lead to good IDV/UNIDATA-GTSPP/NODC discussions. > >>>>> >> >>>> It > >>>>> >> >>>> seems to > >>>>> >> >>>> me that a lot could be accomplished. > >>>>> >> >>>> > >>>>> >> >>>> Murray > >>>>> >> >>>> > >>>>> >> >>>> > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> Ticket Details > >>>>> >> >>> =================== > >>>>> >> >>> Ticket ID: ILT-566886 > >>>>> >> >>> Department: Support IDV > >>>>> >> >>> Priority: Normal > >>>>> >> >>> Status: Open > >>>>> >> >>> > >>>>> >> >> > >>>>> >> > > >>>>> >> > Ticket Details > >>>>> >> > =================== > >>>>> >> > Ticket ID: ILT-566886 > >>>>> >> > Department: Support IDV > >>>>> >> > Priority: Normal > >>>>> >> > Status: Closed > >>>>> >> > > >>>>> >> > >>>>> >> > >>>>> > > >>>>> > > >>>>> > Ticket Details > >>>>> > =================== > >>>>> > Ticket ID: ILT-566886 > >>>>> > Department: Support IDV > >>>>> > Priority: Normal > >>>>> > Status: Closed > >>>>> > >>>>> > >>>> > >>>> > >>>> Ticket Details > >>>> =================== > >>>> Ticket ID: ILT-566886 > >>>> Department: Support IDV > >>>> Priority: Normal > >>>> Status: Closed > >>> > >>> > >> > > > > > > > > Ticket Details =================== Ticket ID: ILT-566886 Department: Support IDV Priority: Normal Status: Closed