[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #ONQ-750372]: NetCDF NC_CHAR file double the size of ASCII file

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #ONQ-750372]: NetCDF NC_CHAR file double the size of ASCII file
  • Date: Fri, 23 May 2014 13:12:53 -0600
Hi Tim,

I'm assuming that you got the answer you needed from the netCDF postings about
the problem.  Please let us know if you have more questions ...

--Russ

> Dear Russ, thanks for your reply.
> 
> The result of the command you suggested is below:
> 
> netcdf plate {
> dimensions:
> Record_Lines = 4 ;
> Line_Symbols = 87 ;
> Record_Number = UNLIMITED ; // (3 currently)
> variables:
> char Record(Record_Number, Record_Lines, Line_Symbols) ;
> 
> // global attributes:
> :_Format = "classic" ;
> }
> 
> 
> 
> Please let me know how best to structure my file for best performance.
> 
> Cheers,
> 
> Tim.
> ______________________________________________
> Tim Stitt PhD
> User Support Manager (CRC)
> Research Assistant Professor (Computer Science & Engineering)
> Room 108, Center for Research Computing, University of Notre Dame, IN 46556
> Email: address@hidden
> 
> 
> 
> 
> 
> On 5/19/14, 5:22 PM, "Unidata netCDF Support"
> <address@hidden> wrote:
> 
> >Hi Tim,
> >
> >> I¹ve been trying to convert a large (40GB) ASCII text file (composed of
> >>multiple records of 4 line ASCII strings about 90 characters long) into
> >>NetCDF format. My plan was to rewrite the original serial code to use
> >>parallel NetCDF to have many MPI processes concurrently read records and
> >>process them in parallel.
> >>
> >> I was able to write some code to convert the ASCII records into
> >>[unlimited][4][90] NetCDF NC_CHAR arrays, which I was able to read
> >>concurrently via parallel NetCDF routines. My question is related to the
> >>size of the converted NetCDF file.
> >>
> >> I notice that the converted NetCDF file is always double the size of
> >>the ASCII file whereas I was hoping for it be to much reduced. I was
> >>therefore wondering if this is expected or is more due to my bad
> >>representation in NetCDF of the ASCII records? I am using
> >>nc_put_vara_text() to write my records. Maybe I need to introduce
> >>compression that I¹m not doing already?
> >
> >Doubling in size is not expected, and I need more information to explain
> >what's happening.  Could you send the output of running
> >
> >  ncdump -sh YOURFILE
> >
> >where YOURFILE is the netCDF file you've written.  That will provide
> >information that may make it clear what's happening, for example
> >inefficient chunk shapes if you've created a netCDF-4 classic model
> >file.  The size of the output from the above command shouldn't be
> >very large as it will only contain the file schema (metadata) and no
> >actual data.
> >
> >Thanks!
> >
> >--Russ
> >
> >Russ Rew                                         UCAR Unidata Program
> >address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> >Ticket Details
> >===================
> >Ticket ID: ONQ-750372
> >Department: Support netCDF
> >Priority: Normal
> >Status: Closed
> >
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: ONQ-750372
Department: Support netCDF
Priority: Normal
Status: Closed