Parallel Processing with joblib

Parallel processing is a method of computation where multiple instructions or processes are executed simultaneously. Its primary goal is to speed up overall execution time by dividing a large problem into smaller, independent sub-problems that can be solved concurrently. This approach is particularly effective for "embarrassingly parallel" problems, where tasks require little to no inter-process communication.

`joblib` is a set of tools from the SciPy ecosystem to provide light-weight pipelining in Python. It is especially designed to handle numerical computations efficiently. One of its most powerful features is its ability to parallelize functions using multiprocessing, making it an excellent choice for speeding up tasks that can be broken down into independent units.

Key components for parallel processing with `joblib`:

1. `Parallel` class: This class allows you to execute a function on a pool of worker processes. You instantiate it by specifying `n_jobs`, which determines the number of parallel jobs to run concurrently. Common values include `1` (no parallelism, sequential execution), positive integers (specific number of processes), or `-1` (use all available CPU cores).
2. `delayed` function: This function acts as a wrapper for your function calls. Instead of immediately executing the function, `delayed` creates a "delayed" object that contains the function and its arguments. The `Parallel` class then takes an iterable of these `delayed` objects and distributes them among its worker processes.

`joblib` is widely used in machine learning libraries like scikit-learn for tasks such as hyperparameter tuning (`GridSearchCV`, `RandomizedSearchCV`) and ensemble methods, where multiple models or parameter combinations can be trained in parallel. It efficiently handles data passing between processes and offers caching mechanisms to avoid recomputing results for identical inputs, further optimizing performance.

import time
from math import sqrt
from joblib import Parallel, delayed

 A computationally intensive function that also simulates some delay
def calculate_square_root_and_sleep(number):
    time.sleep(0.1)   Simulate some I/O or computation delay
    return f"Calculated sqrt({number}) = {sqrt(number):.2f}"

 List of numbers to process
numbers_to_process = list(range(1, 16))  15 numbers for demonstration

print("--- Sequential Processing ---")
start_time_seq = time.time()
results_seq = []
for num in numbers_to_process:
    results_seq.append(calculate_square_root_and_sleep(num))
end_time_seq = time.time()

 for res in results_seq:
     print(res)  Uncomment to see all results
print(f"Sequential processing took: {end_time_seq - start_time_seq:.2f} seconds\n")

print("--- Parallel Processing with joblib ---")
 Use n_jobs=-1 to use all available CPU cores
 The delayed function wraps the function call and its arguments
start_time_par = time.time()
results_par = Parallel(n_jobs=-1)(delayed(calculate_square_root_and_sleep)(num) for num in numbers_to_process)
end_time_par = time.time()

 for res in results_par:
     print(res)  Uncomment to see all results
print(f"Parallel processing took: {end_time_par - start_time_par:.2f} seconds\n")

print("Note: The actual speedup depends on your CPU cores and the nature of the task.")
print("For very short tasks, the overhead of creating processes might reduce or negate benefits.")
print("For CPU-bound tasks, a speedup proportional to the number of cores is often observed.")