[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #TSI-527912]: nccopy advice - rechunking very large files

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #TSI-527912]: nccopy advice - rechunking very large files
  • Date: Wed, 10 Oct 2012 16:37:44 -0600
Hi Mike:

If you can make one of those file available on ftp or something, ill be able to 
profile the ncss. thanks!

John

> My use of "points" in my e-mail description might be misleading.  I am
> really meaning timeseries for individual grid cells from a gridded dataset.
> 
> Thanks,
> 
> Mike
> 
> --
> D. Michael Grogan
> -----------------------------------------
> STG, Inc. - Government ContractorData Access & Applications Branch
> Climate Services and Monitoring Division
> National Climatic Data address@hidden
> Phone: (828) 271-4680
> 
> 
> 
> On Wed, Oct 10, 2012 at 5:23 PM, Mike Grogan <address@hidden> wrote:
> 
> > THREDDS Support,
> >
> > We have been experimenting with different chunking schemes to support fast
> > timeseries access for points within a large (~34 year) physical
> > aggregation.  Please reference Dan Swank's recent discussions with Russ Rew
> > under ticket [netCDF #TSI-527912] in netCDF e-mail support.
> >
> > Dan has created a .nc4 version of this physical aggregation with chunk
> > sizes time/98128,x/8,y/6.
> >
> > When we use the OPeNDAP interface in TDS, we are able to retrieve the
> > entire ~ 34-year timeseries for a single x/y coordinate point in < 1/10th
> > of a second when hitting TDS from localhost.
> >
> > However, when we use the NetCDF Subset Service (NCSS) in TDS to retrieve
> > even just a one-month timeseries for a single lat/lon coordinate it takes
> > 40 seconds or more.
> >
> > What is going on under the hood here with NCSS that would cause this
> > discrepancy?  I know the nearest x/y has to be found in coordinate space
> > for the specified lat/lon.  Is this calculation happening for each timestep
> > for some reason?  Something else?
> >
> > We are using the tds-4.3.13-20120807.191518-1.war that Marcos built for
> > us a couple of months back.
> >
> > Sample URLs [unfortunately not public] are:
> >
> >
> > http://SERVER_NAME:8080/thredds/dodsC/NARRphysagg33/narr-TMP-850mb_221_yyyymmdd_hh00_000.grb.grb2.nc4.ts.d0.ascii?TMP_850mb[0:1:98127][40][240]
> >
> >
> > http://SERVER_NAME:8080/thredds/ncss/grid/NARRphysagg33/narr-TMP-850mb_221_yyyymmdd_hh00_000.grb.grb2.nc4.ts.d0?var=TMP_850mb&latitude=45&longitude=-83&time_start=1979-01-01T00%3A00%3A00.000Z&time_end=1979-01-31T21%3A00%3A00.000Z&vertCoord=&accept=csv
> >
> > Ignore the .grb and .grb2 parts of the filenames.  These ARE NC4 files.
> >  The grb and grb2 are just in there showing our original conversion path in
> > our experiments.
> >
> > Our d0 files are no compression.
> >
> > Thanks,
> >
> > Mike Grogan
> >
> >
> >
> > --
> >
> > D. Michael Grogan
> > -----------------------------------------
> > STG, Inc. - Government ContractorData Access & Applications Branch
> > Climate Services and Monitoring Division
> > National Climatic Data address@hidden
> > Phone: (828) 271-4680
> >
> >
> 
> 


Ticket Details
===================
Ticket ID: TSI-527912
Department: Support THREDDS
Priority: Normal
Status: Open