🚀 Python Tutorial — concurrent.futures (ThreadPool & ProcessPool)

Introduction 🌟

The concurrent.futures module provides a high-level interface for running tasks concurrently using:

  • 🧵 ThreadPoolExecutor → Best for I/O-bound tasks
  • 🧠 ProcessPoolExecutor → Best for CPU-bound tasks

Note

💡 Simple API compared to raw threading/multiprocessing
💡 Future objects represent results of running tasks
💡 Provides map(), submit(), as_completed()

1. Importing concurrent.futures 🧱

import_cf.py

import concurrent.futures

2. ThreadPoolExecutor — I/O Bound Tasks ⚡

thread_pool.py

from concurrent.futures import ThreadPoolExecutor
import time

def fetch_data(n):
    print(f"Fetching {n}...")
    time.sleep(1)
    return f"Data {n}"

with ThreadPoolExecutor(max_workers=3) as executor:
    results = executor.map(fetch_data, [1,2,3,4,5])

print(list(results))

✔ Runs tasks concurrently
✔ Automatically manages threads

3. ProcessPoolExecutor — CPU Bound Tasks 🧮

process_pool.py

from concurrent.futures import ProcessPoolExecutor
import math

def compute(n):
    return sum(math.sqrt(i) for i in range(n))

with ProcessPoolExecutor() as executor:
    results = executor.map(compute, [10000, 20000, 30000])
    print(list(results))

✔ True parallelism using multiple CPU cores

4. submit() — Running Single Tasks 🎯

submit_example.py

from concurrent.futures import ThreadPoolExecutor
import time

def task(n):
    time.sleep(1)
    return n * 2

with ThreadPoolExecutor() as executor:
    future = executor.submit(task, 5)
    print(future.result())

✔ submit() returns a Future object
✔ result() blocks until task completes

5. as_completed() — Process Results as They Finish 🏁

as_completed_example.py

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

def job(n):
    time.sleep(n)
    return f"Finished {n}"

with ThreadPoolExecutor() as ex:
    futures = [ex.submit(job, i) for i in [3,1,2]]

    for f in as_completed(futures):
        print(f.result())

✔ Outputs results in order of completion, not submission

6. Handling Exceptions in Futures ⚠️

future_exception.py

from concurrent.futures import ThreadPoolExecutor

def bad():
    raise ValueError("Oops!")

with ThreadPoolExecutor() as ex:
    f = ex.submit(bad)
    try:
        f.result()
    except Exception as e:
        print("Caught:", e)

✔ Exceptions propagate through result()

7. Cancelling Futures 🛑

cancel_future.py

from concurrent.futures import ThreadPoolExecutor
import time

def long_task():
    time.sleep(3)

with ThreadPoolExecutor() as ex:
    f = ex.submit(long_task)
    canceled = f.cancel()  # only works before running
    print("Canceled?", canceled)

Note

❗ Once a Future has started executing, it cannot be cancelled.

8. Timeout Handling ⏱️

timeout.py

from concurrent.futures import ThreadPoolExecutor
import time

def wait():
    time.sleep(5)

with ThreadPoolExecutor() as ex:
    f = ex.submit(wait)
    try:
        print(f.result(timeout=2))
    except Exception as e:
        print("Timeout:", e)

9. Difference Between map() and submit() ⚡

Featuremap()submit()
ReturnsResults in orderFuture objects
Start immediately?NoYes
Exception accessWhen retrieving resultsDirect via future
Handles multiple argsSingle iterableAny args

10. Real-World Example — Downloading Web Pages 🌐

web_scraping.py

from concurrent.futures import ThreadPoolExecutor
import requests

def download(url):
    print("Downloading", url)
    return requests.get(url).status_code

urls = ["https://google.com", "https://example.com"]

with ThreadPoolExecutor(max_workers=2) as ex:
    for status in ex.map(download, urls):
        print("Status:", status)

11. Real-World Example — Parallel Image Processing 🖼️

image_processing.py

from concurrent.futures import ProcessPoolExecutor
from PIL import Image, ImageFilter

def blur_image(path):
    img = Image.open(path)
    img = img.filter(ImageFilter.BLUR)
    img.save("blur_" + path)
    return path

images = ["img1.jpg", "img2.jpg", "img3.jpg"]

with ProcessPoolExecutor() as ex:
    for result in ex.map(blur_image, images):
        print("Processed:", result)

12. Executor Shutdown 💼

shutdown_example.py

executor = ThreadPoolExecutor()
# ...
executor.shutdown(wait=True)  # wait=False = don't wait for tasks

13. concurrent.futures vs Threading vs Multiprocessing ⚔️

FeatureThreadPoolExecutorProcessPoolExecutor
Best forI/O-bound workCPU-bound work
Parallel?No (GIL)Yes
Lightweight?YesNo
Shared memory?YesNo

Best Practices 💡

  • ✔ Use ThreadPoolExecutor for I/O tasks
  • ✔ Use ProcessPoolExecutor for CPU-heavy tasks
  • ✔ Use map() for simple iterable tasks
  • ✔ Use submit() for more control
  • ✔ Always handle exceptions from futures
  • ✔ Avoid creating too many workers

Conclusion 🎉

>>“concurrent.futures makes concurrency simple — giving you high-level thread and process pools with minimal effort.” ✨

You now fully understand Concurrent Futures in Python! Want the next topic? Try AsyncIO + Executors, Job Scheduling, or Parallel Algorithms. Just tell me! 😊