🚀 Python Tutorial — Multiprocessing

Introduction 🌟

Multiprocessing allows Python programs to run tasks in true parallelismby using multiple CPU cores. Unlike threading (limited by the GIL), multiprocessing is perfect for CPU-intensive tasks such as:

  • ✔ Mathematical computations
  • ✔ Image/video processing
  • ✔ Machine learning workloads
  • ✔ Data transformations

Note

💡 Multiprocessing avoids the GIL
💡 Runs each process in its own memory space
💡 Best for heavy CPU work

1. Importing the Module 🧱

import_mp.py

import multiprocessing

2. Basic Multiprocessing Example ▶️

basic_mp.py

import multiprocessing
import time

def worker():
    print("Worker started")
    time.sleep(1)
    print("Worker finished")

if __name__ == "__main__":
    p = multiprocessing.Process(target=worker)
    p.start()
    p.join()

✔ Each process runs independently
✔ Must use if __name__ == "__main__" on Windows

3. Passing Arguments to Processes 🎯

mp_args.py

def show(name, count):
    for i in range(count):
        print(f"{name}: {i}")

if __name__ == "__main__":
    p = multiprocessing.Process(target=show, args=("TaskA", 3))
    p.start()
    p.join()

4. Running Multiple Processes Concurrently ⚡

multiple_processes.py

def task(num):
    print(f"Processing {num}")

if __name__ == "__main__":
    processes = []
    for i in range(5):
        p = multiprocessing.Process(target=task, args=(i,))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

5. Using Process Pools — Easiest Method 💼

Pool manages a group of worker processes for you.

pool_example.py

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.map(square, [1, 2, 3, 4, 5])
        print(results)

✔ Load-balanced parallel execution

6. Using apply() and map() 🧠

apply_map.py

pool.apply(square, (5,))   # runs once
pool.map(square, [1, 2, 3])        # runs in parallel

7. Sharing Data Between Processes 🔄

Using Value & Array (Shared Memory)

shared_memory.py

from multiprocessing import Process, Value, Array

def modify(val, arr):
    val.value += 1
    arr[0] = 99

if __name__ == "__main__":
    num = Value("i", 10)
    arr = Array("i", [1, 2, 3])

    p = Process(target=modify, args=(num, arr))
    p.start()
    p.join()

    print(num.value)  # 11
    print(arr[:])     # [99, 2, 3]

✔ Best for numeric shared data

8. Using Manager for Shared Objects 🗂️

manager_example.py

from multiprocessing import Manager, Process

def update(shared_list):
    shared_list.append(100)

if __name__ == "__main__":
    with Manager() as manager:
        lst = manager.list([1, 2, 3])

        p = Process(target=update, args=(lst,))
        p.start()
        p.join()

        print(lst)

✔ Works with lists, dicts, namespaces, queues

9. Using Queue for Inter-Process Communication 📬

queue_example.py

from multiprocessing import Process, Queue

def worker(q):
    q.put("Result from worker")

if __name__ == "__main__":
    q = Queue()
    p = Process(target=worker, args=(q,))
    p.start()
    print(q.get())   # receives message
    p.join()

✔ Safe way for processes to exchange data

10. Using Pipes (Two-way Communication) 🔌

pipe_example.py

from multiprocessing import Process, Pipe

def child(conn):
    conn.send("Hello from child")
    print(conn.recv())

if __name__ == "__main__":
    parent_conn, child_conn = Pipe()
    p = Process(target=child, args=(child_conn,))
    p.start()

    print(parent_conn.recv())
    parent_conn.send("Hello from parent")

    p.join()

11. Lock for Process Synchronization 🔒

process_lock.py

from multiprocessing import Process, Lock

def printer(lock, msg):
    with lock:
        print(msg)

if __name__ == "__main__":
    lock = Lock()
    for i in range(5):
        Process(target=printer, args=(lock, f"Message {i}")).start()

✔ Prevents mixed output in the console

12. Real-World Example — CPU-Intensive Task 🧮

cpu_task.py

import math
from multiprocessing import Pool

def compute(n):
    return sum(math.sqrt(i) for i in range(n))

if __name__ == "__main__":
    with Pool() as pool:
        results = pool.map(compute, [10000, 20000, 30000, 40000])
        print(results)

✔ True parallel CPU execution

13. multiprocessing vs threading ⚔️

ThreadingMultiprocessing
Shares memorySeparate memory
Limited by GILNo GIL limitation
Great for I/O tasksBest for CPU tasks
LightweightHeavy (separate processes)

14. Important Notes ⚠️

Note

✔ Always protect entry point withif __name__ == "__main__" (especially on Windows)
✔ Processes are heavier than threads
✔ Data must be serialized (pickled)

15. Best Practices 💡

  • ✔ Use Pool for parallel CPU tasks
  • ✔ Use Managers or Queues for sharing complex objects
  • ✔ Avoid unnecessary shared memory
  • ✔ Prefer multiprocessing.dummy when you need threads with same API
  • ✔ Combine logging for debugging multiprocessing apps

Conclusion 🎉

>>“Multiprocessing unlocks Python’s full CPU power — giving true parallel execution beyond the GIL.” ✨

You now fully understand Multiprocessing in Python! Want the next topic? Try AsyncIO, Process Pools, Concurrency Models, or GIL Explained. Just tell me! 😊