Re: 20050511: Large file access time

Unidata Support <address@hidden> writes:

> ------- Forwarded Message
>
>>To: address@hidden
>>From: J-M Epitalon <address@hidden>
>>Subject: Large file access time
>>Organization: UCAR/Unidata
>>Keywords: 200505111731.j4BHViP3026012
>
> Hello all,
>
> I experience a problem of latency when opening a file using Ferret.
> Ferret is a program that uses the NetCDF interface to visualize data. I
> submitted my problem to the Ferret users mailing list but nobody could
> answer it.
>
> I run the same following Ferret command
>    "use/regulart my_file_1500_Mb_large.nc"
> on two different Linux platforms.
>
> The purpose of this command is opening a NetCDF file, reading its header
> in order to determine its dimensions.
> The option "/regulart" tells Ferret to skip all time steps but the first
> and last one, since the time steps are assumed to be regular.
>
> On one platform, it executes immediately. On the second one, it takes 30
> seconds the first time, then once the file is in memory cache, it
> executes immediately.
> I mention that the file I want to open is 1.5Gb large.
>
> Please, is anybody willing to give me a possible cause for this ?
> Is there an easy way to debug this ?
>
> Is it possible that seeking the last time step is the process that takes
> that much time ?
>
> The file contains only one variable, whose dimensions are:
> ======================================
> netcdf EC2_1d_195001_199912_SUSURUSU {
> dimensions:
>         jpib = 21692 ;
>         jpjb = 1 ;
>         time_counter = UNLIMITED ; // (18262 currently)
> variables:
>         double time_counter(time_counter) ;
>                 time_counter:title = "Time" ;
>                 time_counter:long_name = "Time axis" ;
>                 time_counter:calendar = "gregorian" ;
>                 time_counter:units = "days since 1950-01-01 00:00:00" ;
>                 time_counter:time_origin = "1950-01-01 00:00:00" ;
>         float SUSURUSU(time_counter, jpjb, jpib) ;
>                 SUSURUSU:short_name = "RUISSELLEMEN" ;
>                 SUSURUSU:gridtype = "MEDIASGL" ;
>                 SUSURUSU:long_name = "" ;
>                 SUSURUSU:units = "" ;
>  
> ======================================
>
> Thanks in advance
>
> Below is the message I sent on Ferret users mailing list:
>
>> I notice that the option /regulart speeds up quite a bit.
>> On my personnal workstation, a Red-Hat PC running Ferret 5.51, the
>> instruction use/regulart executes instantaneously.
>>
>> However, on my LAS server, which is a quadriprocessor PowerEdge 2650
>> DELL PC with Red Hat Enterprise Linux, running Ferret 5.8, I cannot
>> achieve the same result:
>>
>> When opening a 1.5 Gb NetCDF file, it takes 20 to 30 seconds, even for
> a
>> local file!!!
>> For a NFS remote file, it takes 30 to 45s.
>>
>> Then, the second time I open it, whether in the same or another Ferret
>> session, it opens immediately.
>>
>> I tried different versions of Ferret for Linux Red-Hat 9: v5.53, 5.7
> and
>> 5.8 and I cannot see any difference.
>> I suppose the problem lies in the PC or in the OS...
>>
>> Please, could you help me understand this ?
>
> Jean-Marie Epitalon
> CERFACS
> Toulouse, France
>
> --
> NOTE: All email exchanges with Unidata User Support are recorded in the
> Unidata inquiry tracking system and then made publicly available
> through the web.  If you do not want to have your interactions made
> available in this way, you must let us know in each email you send to us.
>
> ------- End of Forwarded Message
>
>

Sorry, but I'm not familiar with the Ferret package, so I don't know
what it could be doing to take all that time.

Ed
-- 
Ed Hartnett  -- address@hidden