[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[netCDF #SNE-995575]: NetCDF Compression
- Subject: [netCDF #SNE-995575]: NetCDF Compression
- Date: Tue, 11 Jun 2019 14:26:55 -0600
Since it should not be possible to compile HDF5 without libzip, this seems to
mean
that for some reason libzip is being called on AIX with a bad deflate value.
Is the AIX machine by any chance big-endian?
In any case, when you built the AIX netcdf library, did you enable logging?
If so, then try adding -L5 to the nccopy command to generate logging
information.
> The file input file on AIX or LINUX does NOT have _DeflateLevel attributes
>
> The output file on LINUX DOES have _DeflateLevel attributes
>
> the output file on AIX does NOT have _DeflateLevel attributes
>
>
> On 6/11/2019 4:00 PM, Unidata netCDF Support wrote:
> > You should see attributes for each variable saying
> > _DeflateLevel = 6
> > Since I do not see those, it looks like, for some reason russ.nc
> > is not compressed. See if the input file has _DeflateLevel attributes.
> >
> >
> >> Both the input and output file are 37574768 bytes after the nccopy
> >> operation
> >>
> >> /usr/local/netcdf-4.1.3/bin/ncdump -hs russ.nc
> >> netcdf russ {
> >> dimensions:
> >> lon = 2501 ;
> >> lat = 1251 ;
> >> variables:
> >> float lat(lat) ;
> >> lat:standard_name = "lat" ;
> >> lat:long_name = "latitude" ;
> >> lat:units = "degrees_north" ;
> >> lat:_FillValue = -999.f ;
> >> lat:_Storage = "contiguous" ;
> >> float lon(lon) ;
> >> lon:standard_name = "lon" ;
> >> lon:long_name = "longitude" ;
> >> lon:units = "degrees_east" ;
> >> lon:_FillValue = -999.f ;
> >> lon:_Storage = "contiguous" ;
> >> float TPW(lat, lon) ;
> >> TPW:name = "TPW" ;
> >> TPW:long_name = "Blended Total Precipitable Water" ;
> >> TPW:units = "mm" ;
> >> TPW:coordinates = "Latitude Longitude" ;
> >> TPW:_FillValue = -999.f ;
> >> TPW:_Storage = "contiguous" ;
> >> float observation_age(lat, lon) ;
> >> observation_age:name = "observation_age" ;
> >> observation_age:long_name = "age of
> >> observation(analysis_time - observation_time)" ;
> >> observation_age:units = "hours" ;
> >> observation_age:coordinates = "Latitude Longitude" ;
> >> observation_age:_FillValue = -999.f ;
> >> observation_age:_Storage = "contiguous" ;
> >> float Satellite_Number(lat, lon) ;
> >> Satellite_Number:name = "satellite_number" ;
> >> Satellite_Number:long_name = "the ID number of the
> >> satellite which made the observation" ;
> >> Satellite_Number:units = "none" ;
> >> Satellite_Number:coordinates = "Latitude Longitude" ;
> >> Satellite_Number:_FillValue = -999.f ;
> >> Satellite_Number:_Storage = "contiguous" ;
> >>
> >> // global attributes:
> >> :Metadata_Link =
> >> "BHP-TPW_v01r0_blend_s201906110131507_e201906111041388_c201906111148307.nc"
> >> ;
> >> :date_created = "2019-06-11-T11:48:30Z" ;
> >> :time_coverage_start = "2019-06-11-T01:31:50Z" ;
> >> :time_coverage_end = "2019-06-11-T10:41:38Z" ;
> >> :project = "Blended-HydroMet Products" ;
> >> :naming_authority = "NDE" ;
> >> :production_site = "DOC/NOAA/NESDIS/OSPO/ESPC/NDE,
> >> S-NPP Data Exploitation" ;
> >> :production_environment = "NDE Data Handling System" ;
> >> :platform = "Linux" ;
> >> :algorithm_version = "v3.0" ;
> >> :DAP_version = "v01r0" ;
> >> :Convention = "CF-1.6" ;
> >> :Metadata_Conventions = "CF-1.6, Unidata Dataset
> >> Discovery v1.0" ;
> >> :standard_name_vocabulary = "CF Standard Name Table
> >> (version 17, 24 March 2011)" ;
> >> :satellite_name =
> >> "NOAA-19,NOAA-20,METOP-A,METOPB,Suomi-NPP,GCOM-W1,GPM,Met-GPS,GOES-15" ;
> >> :instrument_name = "MiRS,ATMS,AMSR2,GPS,ABI GOES Imager" ;
> >> :title = "Blended Hydrometeorological Products (BHP)" ;
> >> :summary = "Unified TPW global map merges the TPW
> >> retrievals from various satellites/algorithms" ;
> >> :references =
> >> "https://www.ospo.noaa.gov/Products/bTPW/index.html"; ;
> >> :keywords = "Multi-Satellites merged product, NDE
> >> Operation, OSPO" ;
> >> :institution = "DOC/NOAA/NESDIS/OSPO/NDE, S-NPP Data
> >> Exploitation, OSPO, NESDIS, NOAA, U.S. Department of Commerce" ;
> >> :number_of_granules = "inapplicable" ;
> >> :geospatial_lat_min = "-90.0" ;
> >> :geospatial_lat_max = "90.0" ;
> >> :geospatial_lon_min = "-180.0" ;
> >> :geospatial_lon_max = "180.0" ;
> >> :geospatial_lat_units = "degrees_north" ;
> >> :geospatial_lon_units = "degrees_east" ;
> >> :creator_name = "DOC/NOAA/NESDIS/OSPO, Blended Hydro
> >> Team, Office of Satellite and Product Operations, OSPO, NESDIS, NOAA,
> >> U.S. Department of Commerce" ;
> >> :creator_email = "address@hidden" ;
> >> :creator_url =
> >> "https://www.ospo.noaa.gov/Products/bTPW/index.html"; ;
> >> :publisher_name = "DOC/NOAA/NESDIS/OSPO/NDE S-NPP Data
> >> Exploitation, NESDIS, NOAA, U.S. Department of Commerce" ;
> >> :publisher_email = "address@hidden" ;
> >> :publisher_url =
> >> "https://www.ospo.noaa.gov/Products/suites.html"; ;
> >> :history = "Created by BHP Version 3.0" ;
> >> :processing_level = "NOAA Level 4 data" ;
> >> :cdm_data_type = "GRID" ;
> >> :source = "MIRS_ATMS_IMG_33min; AMSR2_OCEAN " ;
> >> :PDA_short_name = "BHP-TPW" ;
> >> :missing_value = "-999.0" ;
> >> :compression_level = "0" ;
> >> :nc4_compression_flag = "on" ;
> >> :_Format = "netCDF-4 classic model" ;
> >> }
> >>
> >> On 6/11/2019 3:12 PM, Unidata netCDF Support wrote:
> >>> This will be difficult for us to debug since we do not have
> >>> access to an AIX system. Also, netcdf 4.1.3 is pretty old;
> >>> any chance you can upgrade to the latest version?
> >>> In any case, try this command and send us the output
> >>> (assuming its not proprietary):
> >>>> ncdump -hs <name of compressed file>
> >>>> Description of problem: My programmer issues the following command using
> >>>> NetCDF 4.1.3 to compress a NetCDF file on AIX and it does not compress
> >>>> at all.
> >>>>
> >>>> On Linux using NetCDF V4.6.1 it does compress successfully.
> >>>>
> >>>> /usr/local/netcdf-4.1.3/bin/nccopy -d 6
> >>>> BHP-TPW_v01r0_blend.nc russ.nc
> >>>>
> >>>> -rw-r--r-- 1 Pdpeas dpeas 37574958 Jun 11 18:06
> >>>> BHP-TPW_v01r0_blend.nc
> >>>>
> >>>> -rw-r--r-- 1 root system 37574768 Jun 11 18:55 russ.nc
> >>>>
> >>>>
> >>>>
> >>> =Dennis Heimbigner
> >>> Unidata
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: SNE-995575
> >>> Department: Support netCDF
> >>> Priority: Normal
> >>> Status: Open
> >>> ===================
> >>> NOTE: All email exchanges with Unidata User Support are recorded in the
> >>> Unidata inquiry tracking system and then made publicly available through
> >>> the web. If you do not want to have your interactions made available in
> >>> this way, you must let us know in eac
> > =Dennis Heimbigner
> > Unidata
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: SNE-995575
> > Department: Support netCDF
> > Priority: Normal
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the
> > Unidata inquiry tracking system and then made publicly available through
> > the web. If you do not want to have your interactions made available in
> > this way, you must let us know in each email you send to us.
> >
>
>
=Dennis Heimbigner
Unidata
Ticket Details
===================
Ticket ID: SNE-995575
Department: Support netCDF
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata
inquiry tracking system and then made publicly available through the web. If
you do not want to have your interactions made available in this way, you must
let us know in each email you send to us.