[netCDF #TSI-527912]: nccopy advice - rechunking very large files

thanks dan, i can use the one that russ generated and save 3 hours of 
download time.

On 10/11/2012 7:56 AM, Dan Swank wrote:
> New Client Reply: nccopy advice - rechunking very large files
>
> John,
> All files may be accessed here:
> http://nomads.ncdc.noaa.gov/data/offlinestage/custom/narr-physaggs/
> Same directory Russ got it from.
> narr-TMP-850mb*
> are our first test case for which  all variants are in place.
>
>            -Dan
>
>
> On Wed, Oct 10, 2012 at 6:37 PM, Unidata THREDDS Support
> <address@hidden> wrote:
>> Hi Mike:
>>
>> If you can make one of those file available on ftp or something, ill be able 
>> to profile the ncss. thanks!
>>
>> John
>>
>>> My use of "points" in my e-mail description might be misleading.  I am
>>> really meaning timeseries for individual grid cells from a gridded dataset.
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>> --
>>> D. Michael Grogan
>>> -----------------------------------------
>>> STG, Inc. - Government ContractorData Access & Applications Branch
>>> Climate Services and Monitoring Division
>>> National Climatic Data address@hidden
>>> Phone: (828) 271-4680
>>>
>>>
>>>
>>> On Wed, Oct 10, 2012 at 5:23 PM, Mike Grogan <address@hidden> wrote:
>>>
>>>> THREDDS Support,
>>>>
>>>> We have been experimenting with different chunking schemes to support fast
>>>> timeseries access for points within a large (~34 year) physical
>>>> aggregation.  Please reference Dan Swank's recent discussions with Russ Rew
>>>> under ticket [netCDF #TSI-527912] in netCDF e-mail support.
>>>>
>>>> Dan has created a .nc4 version of this physical aggregation with chunk
>>>> sizes time/98128,x/8,y/6.
>>>>
>>>> When we use the OPeNDAP interface in TDS, we are able to retrieve the
>>>> entire ~ 34-year timeseries for a single x/y coordinate point in < 1/10th
>>>> of a second when hitting TDS from localhost.
>>>>
>>>> However, when we use the NetCDF Subset Service (NCSS) in TDS to retrieve
>>>> even just a one-month timeseries for a single lat/lon coordinate it takes
>>>> 40 seconds or more.
>>>>
>>>> What is going on under the hood here with NCSS that would cause this
>>>> discrepancy?  I know the nearest x/y has to be found in coordinate space
>>>> for the specified lat/lon.  Is this calculation happening for each timestep
>>>> for some reason?  Something else?
>>>>
>>>> We are using the tds-4.3.13-20120807.191518-1.war that Marcos built for
>>>> us a couple of months back.
>>>>
>>>> Sample URLs [unfortunately not public] are:
>>>>
>>>>
>>>> http://SERVER_NAME:8080/thredds/dodsC/NARRphysagg33/narr-TMP-850mb_221_yyyymmdd_hh00_000.grb.grb2.nc4.ts.d0.ascii?TMP_850mb[0:1:98127][40][240]
>>>>
>>>>
>>>> http://SERVER_NAME:8080/thredds/ncss/grid/NARRphysagg33/narr-TMP-850mb_221_yyyymmdd_hh00_000.grb.grb2.nc4.ts.d0?var=TMP_850mb&latitude=45&longitude=-83&time_start=1979-01-01T00%3A00%3A00.000Z&time_end=1979-01-31T21%3A00%3A00.000Z&vertCoord=&accept=csv
>>>>
>>>> Ignore the .grb and .grb2 parts of the filenames.  These ARE NC4 files.
>>>>   The grb and grb2 are just in there showing our original conversion path 
>>>> in
>>>> our experiments.
>>>>
>>>> Our d0 files are no compression.
>>>>
>>>> Thanks,
>>>>
>>>> Mike Grogan
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> D. Michael Grogan
>>>> -----------------------------------------
>>>> STG, Inc. - Government ContractorData Access & Applications Branch
>>>> Climate Services and Monitoring Division
>>>> National Climatic Data address@hidden
>>>> Phone: (828) 271-4680
>>>>
>>>>
>>>
>>>
>>
>>
>> Ticket Details
>> ===================
>> Ticket ID: TSI-527912
>> Department: Support THREDDS
>> Priority: Normal
>> Status: Open
>>
>
>
>



Ticket Details
===================
Ticket ID: TSI-527912
Department: Support THREDDS
Priority: Normal
Status: Open