[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #AIA-724598]: Completeness of NC files

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #AIA-724598]: Completeness of NC files
  • Date: Tue, 13 Oct 2015 10:49:45 -0600
Hello Ulrich,

Unless the file simply fails to open, it would be very difficult to ensure that 
a file transferred via ftp/ssh/scp/etc is 'complete'.  If the transfer does not 
return an error indicating that the file is truncated or otherwise incomplete.  
Ways to verify the file you downloaded is at least a full duplicate of the 
remote file would be file size/history information (as you point out), or 
perhaps mdf5 checksums *if* the institution uses them; sadly, most do not.  

I believe that you are correct that the file was replaced on the remote server; 
I can't say that for certain, however.  I'm sorry I can't provide a more 
positive answer; have a great day.

-Ward

> 
> Dear colleagues,
> 
> how can I make sure that a given netcdf data set is complete?
> 
> Recently, I got a data set (about 200 Mbytes) from NCAR-RAL by ftp.
> I found that the data that were announced to be in that file were in fact not 
> in the file.
> After some debate, I downloaded the data set again (the history information 
> was identical)
> 
> The new data set was larger in bytes (factor 2.5) and now contained the 
> information that I was missing.
> 
> Do you have an explanation for this? Can it happen that a data gets 
> transferred by ftp incomplete and such that the user does not realize this 
> when opening the file?
> 
> Alternatively (and perhaps more likely), the file was changed in the ftp 
> server from where I downloaded. Unfortunately, I cannot check this myself.
> 
> 
> Thank you,
> 
> Ulrich Schumann
> 
> 
> 
> --------------------------
> Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
> German Aerospace Center
> Institute of Atmospheric Physics | Oberpfaffenhofen | 82234 Wessling | Germany
> 
> Prof. Dr. Ulrich Schumann
> Telephone +49 173 28 75 353 | Telefax +49 8153 28 1841 | 
> address@hidden<mailto:address@hidden>
> www.DLR.de<http://www.dlr.de/>
> 
> 
> 
> 

Ticket Details
===================
Ticket ID: AIA-724598
Department: Support netCDF
Priority: Normal
Status: Closed