No speedup observed when increasing MPI processes

[cl25796]

Dear Prof. Zhong,

I'm using HamEPC for EPC calculations and encountering a scaling issue. The runtime is nearly identical when using mpirun -np N HamEPC --config EPC_input.yaml for N values from 1 to 12.

I've experimented with various OMP_NUM_THREADS (from 1-4) and -np N (from 1 to 12) combinations, as suggested by the hybrid parallelization documentation, but the lack of speedup persists.

I've tried two different systems and both have the same problem.
Small system:

Large system:

Are there known scaling characteristics that might explain this? Any advice on diagnosing this would be appreciated.

Thank you for your time and guidance.

Sincerely,
Chen

[cl25796]

Dear Prof. Zhong,

I found that the multiple cores are doing the same calculations for HamEPC as below. I'm using the same compilers_and_libraries_2018.5.274 as you mentioned in HamGNN and I can successfully run openmx_postprocess with command mpirun -np 8 openmx_postprocess openmx.dat.

Do you know why this is happening? Thank you so much!

Sincerely,
Chen

mpirun -np 1 HamEPC --config EPC_input.yaml
################################################## EPC Calculation ##################################################
----- k_path report begin ----------
real-space lattice vectors
 [[ 6.08492  0.       0.     ]
 [-3.04246  5.26969  0.     ]
 [ 0.       0.       9.82658]]
k-space metric tensor
 [[0.03601 0.01801 0.     ]
 [0.01801 0.03601 0.     ]
 [0.      0.      0.01036]]
internal coordinates of nodes
 [[0.  0.  0. ]
 [0.5 0.  0. ]]
reciprocal-space lattice vectors
 [[0.16434 0.09488 0.     ]
 [0.      0.18976 0.     ]
 [0.      0.      0.10176]]
cartesian coordinates of nodes
 [[0.      0.      0.     ]
 [0.08217 0.04744 0.     ]]
list of segments:
  length = 0.09488  from  [0. 0. 0.]  to  [0.5 0.  0. ]
node distance list: [0.      0.09488]
node index list:    [0 1]
----- k_path report end ------------

Ignore block transport.
Ignore block mobility.
100%|██████████| 2/2 [02:03<00:00, 61.83s/it]

---

mpirun -np 2 HamEPC --config EPC_input.yaml
################################################## EPC Calculation ##################################################
----- k_path report begin ----------
real-space lattice vectors
 [[ 6.08492  0.       0.     ]
 [-3.04246  5.26969  0.     ]
 [ 0.       0.       9.82658]]
k-space metric tensor
 [[0.03601 0.01801 0.     ]
 [0.01801 0.03601 0.     ]
 [0.      0.      0.01036]]
internal coordinates of nodes
 [[0.  0.  0. ]
 [0.5 0.  0. ]]
reciprocal-space lattice vectors
 [[0.16434 0.09488 0.     ]
 [0.      0.18976 0.     ]
 [0.      0.      0.10176]]
cartesian coordinates of nodes
 [[0.      0.      0.     ]
 [0.08217 0.04744 0.     ]]
list of segments:
  length = 0.09488  from  [0. 0. 0.]  to  [0.5 0.  0. ]
node distance list: [0.      0.09488]
node index list:    [0 1]
----- k_path report end ------------

Ignore block transport.
Ignore block mobility.
  0%|          | 0/2 [00:00<?, ?it/s]----- k_path report begin ----------
real-space lattice vectors
 [[ 6.08492  0.       0.     ]
 [-3.04246  5.26969  0.     ]
 [ 0.       0.       9.82658]]
k-space metric tensor
 [[0.03601 0.01801 0.     ]
 [0.01801 0.03601 0.     ]
 [0.      0.      0.01036]]
internal coordinates of nodes
 [[0.  0.  0. ]
 [0.5 0.  0. ]]
reciprocal-space lattice vectors
 [[0.16434 0.09488 0.     ]
 [0.      0.18976 0.     ]
 [0.      0.      0.10176]]
cartesian coordinates of nodes
 [[0.      0.      0.     ]
 [0.08217 0.04744 0.     ]]
list of segments:
  length = 0.09488  from  [0. 0. 0.]  to  [0.5 0.  0. ]
node distance list: [0.      0.09488]
node index list:    [0 1]
----- k_path report end ------------

100%|██████████| 2/2 [02:06<00:00, 63.01s/it]
100%|██████████| 2/2 [02:04<00:00, 62.14s/it]

[QuantumLab-ZY]

Hi @cl25796,

Currently, not all computational modes in HamEPC support parallel acceleration. If you're performing a band structure calculation, please note that it is still executed in serial. Our parallel processing efforts have mainly been focused on optimizing the summation loops in calculations related to mobility and superconductivity.

Best,
Yang Zhong

[cl25796]

Hi Prof. Zhong,

Thank you so much for your reply. I'm trying to run mobility and superconductivity calculations with parallel computing but still have some problems.

For the GaAs example, I could successfully run the mobility calculations with command "HamEPC --config EPC_input.yaml".

HamEPC --config EPC_input.yaml
################################################## Mobility Calculation ##################################################
Sampling with Cauchy distribution.
Using ERTA method to calculate mobility.
CBM = -2.117707678129024 eV
Fermi energy = -2.218457076513638 eV, Carrier density = 1.0000001532017732e+16 cm^(-3).

But when I used command "mpirun -np N HamEPC --config EPC_input.yaml" (N=1,2,4). I got the following error:

mpirun -np 1 HamEPC --config EPC_input.yaml

=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 3987920 RUNNING AT ztcomputation-MD72-HB0-00
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 3987920 RUNNING AT ztcomputation-MD72-HB0-00
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

   Intel(R) MPI Library troubleshooting guide:
      https://software.intel.com/node/561764

I tried to re-install HamEPC but it didn't help. Can you please give me some suggestions to fix this error? Thank you so much!

Best regards,
Chen

[cl25796]

Dear Prof. Zhong,

I want to follow up with the previous question. I tried to re-install HamEPC with different Intel libraries but still can't run parallel calculations for mobility and superconductivity. I can successfully run parallel calculations for openmx_postprocess with command mpirun -np 8 openmx_postprocess openmx.dat. It takes too long to run mobility and superconductivity with a single core. Can you please give me some suggestions?

Thank you so much! @QuantumLab-ZY

Best regards,
Chen

[QuantumLab-ZY]

Hi Chen,

If you do not see duplicate outputs in your log files, that usually means parallelization is working correctly.

Both the mobility and superconductivity calculations involve very dense grids. For example, the k and q grids used in the mobility calculation are both 300x300x300, which is extremely demanding computationally. Using only 9 CPUs is insufficient for such large grids. Additionally, the error message "EXIT CODE: 9" indicates that the job ran out of memory.

To address this, I recommend using at least 100 CPUs for both mobility and superconductivity calculations. For the superconductivity calculations specifically, you can combine MPI and OpenMP for better performance and memory usage. This can be done by adjusting the number of MPI tasks in your mpirun command and setting the OMP_NUM_THREADS environment variable appropriately. This hybrid approach can help to conserve memory and achieve better speedup.

Best,
Yang Zhong