WRITELOOP

PYTHON CONCURRENCY WITH CONCURRENT.FUTURES 101

2017 March 18

We would want to use the ProcessPoolExecutor for CPU intensive tasks. The ThreadPoolExecutor is better suited for network operations or I/O. While the API is similar, we must remember that the ProcessPoolExecutor uses the multiprocessing module and is not affected by the Global Interpreter Lock. However, we can not use any objects that is not picklable. So we need to carefully choose what we use/return inside the callable passed to process pool executor. (reference: http://masnun.com/2016/03/29/python-a-quick-introduction-to-the-concurrent-futures-module.html)