[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #ULD-680356]: parallel netcdf: writing with increasing number of cores seems to be slower than fewer cores.

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #ULD-680356]: parallel netcdf: writing with increasing number of cores seems to be slower than fewer cores.
  • Date: Fri, 14 Sep 2018 12:58:49 -0600
Thank you very much for submitting this information and the code! I will take a 
look at it and see if there is in fact a bug on our end! Depending on data 
sizes, disk storage format (SSD, spinning drive, etc), and total number of 
cores available, it's not unknown to see this sort of counterintuitive result.  
During my own graduate work, I observed a loss of performance when I added too 
many processors to a bioinformatics analysis I was doing.   It was eventually 
tracked to a known issue in the hardware scheduler for that intel chipset.

That said, it is far more likely that there is an issue somewhere in the 
software.  Thanks again for reporting this!  It definitely bears examination.

Have a great day,

-Ward

> Dear Sir, Madam,
> 
> 
> I am writing you because I am performing some tests with parallel netcdf. I 
> am writing a 5 dimension array in parallel. The interesting part is that, 
> when I run the code with 2 cores, it performs faster than, say 20 cores.
> 
> 
> Attached is a folder containing the source code for my test. You will notice 
> I am using the Eigen library, as these experiments are related to some code 
> from my PhD project (where we use this specific array format).
> 
> 
> If you'd like to run the code, please just change the paths to the Eigen 
> directory and parallel netcdf directory in the makefile before compiling.
> 
> 
> Maybe I'm just missing a simple bug, but maybe there is actually an issue?
> 
> 
> Many thanks for your help,
> 
> 
> Alexandre Szenicer.
> 
> 
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: ULD-680356
Department: Support netCDF
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.