Multiprocessing in Python with the `multiprocessing` module

Multiprocessing refers to the ability of a system to run multiple programs or processes concurrently. In the context of Python, the `multiprocessing` module provides a powerful way to spawn processes, allowing them to execute tasks in parallel, effectively leveraging multiple CPU cores. This is particularly crucial in Python due to the Global Interpreter Lock (GIL), which limits a single Python process to executing only one thread at a time, even on multi-core machines.

Why use `multiprocessing`?

1. CPU-bound tasks: For computations that heavily rely on the CPU (e.g., complex calculations, data processing), `multiprocessing` allows these tasks to run on different CPU cores simultaneously, leading to significant speed improvements.
2. Bypassing the GIL: Unlike multithreading, where all threads within a single Python process are subject to the GIL, each process created by the `multiprocessing` module has its own Python interpreter and its own GIL. This means that CPU-bound operations in separate processes can truly run in parallel without being bottlenecked by the GIL.

Key Components of the `multiprocessing` module:

- `Process` Class: The fundamental way to create and manage individual processes. You instantiate `Process` with a target function and its arguments, then call `start()` to run it and `join()` to wait for its completion.
- `Pool` Class: A higher-level abstraction that simplifies the parallel execution of a function across a pool of worker processes. It's ideal for 'map-reduce' style operations where you want to apply the same function to many different inputs in parallel. Methods like `map()`, `apply()`, `map_async()`, and `apply_async()` are commonly used.
- Inter-Process Communication (IPC): Since processes have separate memory spaces, special mechanisms are needed for them to communicate or share data:
- `Queue`: A multi-producer, multi-consumer queue, safe for use across processes. Useful for passing messages or data chunks.
- `Pipe`: A simpler two-way communication channel between two processes.
- `Manager`: Allows Python objects to be shared between processes, providing synchronization primitives and shareable data structures (e.g., shared lists, dictionaries).
- Synchronization Primitives: Similar to threading, `multiprocessing` provides tools like `Lock`, `Semaphore`, and `Event` to coordinate activities between processes, especially when they need to access shared resources like files or external databases (though not shared memory directly).

When a new process is spawned, a copy of the parent process's state (including loaded modules and global variables) is typically created. This ensures isolation and prevents unintended side effects between processes. The `multiprocessing` module handles the complexities of process creation, management, and communication, making it relatively straightforward to write parallel Python applications.

import multiprocessing
import time
import os

def cpu_bound_task(number):
    """A CPU-bound task that performs a calculation."""
     Simulate a calculation by squaring numbers in a loop
    pid = os.getpid()
    print(f"Process {pid}: Starting task for {number}...")
    result = 0
    for i in range(1_000_000):
        result += (number - i)  0.5  A dummy calculation
    print(f"Process {pid}: Finished task for {number} with a partial result.")
    return f"Task for {number} completed by Process {pid}"

def main_process_example():
    print("\n--- Demonstrating `Process` class ---")
    processes = []
    numbers = [5, 10, 15, 20]

    start_time = time.perf_counter()

    for num in numbers:
        p = multiprocessing.Process(target=cpu_bound_task, args=(num,))
        processes.append(p)
        p.start()  Start the process

    for p in processes:
        p.join()  Wait for the process to complete

    end_time = time.perf_counter()
    print(f"All individual processes finished in {end_time - start_time:.4f} seconds.")

def main_pool_example():
    print("\n--- Demonstrating `Pool` class ---")
    numbers = [5, 10, 15, 20, 25, 30, 35, 40]
     Determine the number of processes based on CPU cores
    num_cores = multiprocessing.cpu_count()
    print(f"Using a pool of {num_cores} processes.")

    start_time = time.perf_counter()

     Create a pool of worker processes. 'with' ensures cleanup.
    with multiprocessing.Pool(processes=num_cores) as pool:
         `map` applies the function to each item in the iterable
         and collects results in order.
        results = pool.map(cpu_bound_task, numbers)

    end_time = time.perf_counter()
    print(f"Pool operations finished in {end_time - start_time:.4f} seconds.")
    print("Results from Pool:", results)

if __name__ == "__main__":
     It's crucial to put the main execution logic under `if __name__ == "__main__":`
     on Windows, to prevent infinite recursion when processes are spawned.
    main_process_example()
    main_pool_example()