mpi4py is a python API for MPI. MPIPoolExecutor is a subclass of mpi4py.futures to create MPI processes to execute calls asynchronously

The MPIPoolExecutor class uses a pool of MPI processes to execute calls asynchronously. By performing computations in separate processes, it allows to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned. The __main__ module must be importable by worker processes

Test Script

The test script shows a way to calculate $2^{x}$ in a loop

1
2
3
4
5

from mpi4py.futures import MPIPoolExecutor

executor = MPIPoolExecutor(max_workers=100)
for result in executor.map(pow, [2]*32, range(32)):
    print(result)
  • executor is an instance of MPIPoolExecutor with max_workers of 100
  • Calculate $2^x$ in a map where $x$ set from 0 to 31

Time length running

1
2
3
4

starttime = time.time()
......
endtime = time.time()
print('Running for %6.3fs' % endtime - starttime)

Run this script via mpiexec

script for MPIPoolExecutor could run via mpiexec in command line

1

mpiexec -n 100 python -m mpi4py.futures MpiPool.py

the script would be execute with 100 cores

Comparision

  • 2.7007s running with 1 core
  • 0.1527s running with 100 core

Test script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

from mpi4py.futures import MPIPoolExecutor
import time

if __name__ == '__main__':
    executor = MPIPoolExecutor(max_workers=100)
    starttime = time.time()
    for result in executor.map(pow, [2]*32, range(32)):
        print(result)
    endtime = time.time()
    print('Running for %6.3fs' % endtime - starttime)