Python Multiprocessing: Parallel Programming Guide

The Python multiprocessing module provides a powerful toolkit for creating parallel programs that can efficiently utilize multi-core processors. Unlike standard threading, which is limited by the Global Interpreter Lock (GIL), multiprocessing allows you to run multiple processes simultaneously, each with its own memory space and independent execution.

Core Principles of Multiprocessing

Multiprocessing in Python is based on creating separate processes that can run in parallel on different CPU cores. Each process operates in an isolated memory space, eliminating the shared data issues common in multithreading.

Key components of the multiprocessing module include:

• Process — a class for creating and managing individual processes
• Pool — a process pool for efficiently executing multiple tasks
• Queue — queues for safely passing data between processes
• Lock — synchronization mechanisms for controlling resource access
• Manager — objects for managing shared data

When to Use Multiprocessing

Multiprocessing is most effective in the following scenarios:

CPU-intensive tasks: mathematical computations, image processing, data analysis, scientific calculations. These tasks benefit greatly from parallel execution across multiple CPU cores.

Large-scale data processing: when you need to apply the same function to many elements, multiprocessing can significantly speed up execution.

Independent tasks: operations that don't require frequent data exchange between processes.

For I/O-intensive tasks (file operations, network requests), consider using asynchronous programming (asyncio) or multithreading instead.

Creating and Managing Processes

Basic Example with the Process Class

import multiprocessing
import time

def worker_task(name, duration):
    """Function to run in a separate process"""
    print(f"Process {name} started")
    time.sleep(duration)
    print(f"Process {name} finished")

if __name__ == "__main__":
    # Create processes
    process1 = multiprocessing.Process(target=worker_task, args=("Worker-1", 3))
    process2 = multiprocessing.Process(target=worker_task, args=("Worker-2", 2))
    
    # Start processes
    start_time = time.time()
    process1.start()
    process2.start()
    
    # Wait for processes to finish
    process1.join()
    process2.join()
    
    end_time = time.time()
    print(f"Total execution time: {end_time - start_time:.2f} seconds")

Inheriting from the Process Class

import multiprocessing
import os
import time

class CustomProcess(multiprocessing.Process):
    def __init__(self, name, work_count):
        super().__init__()
        self.name = name
        self.work_count = work_count
    
    def run(self):
        """Method executed in the process"""
        print(f"Process {self.name} (PID: {os.getpid()}) started")
        
        for i in range(self.work_count):
            print(f"{self.name}: executing task {i+1}")
            time.sleep(1)
        
        print(f"Process {self.name} finished")

if __name__ == "__main__":
    processes = []
    
    # Create multiple processes
    for i in range(3):
        p = CustomProcess(f"Process-{i+1}", 3)
        processes.append(p)
        p.start()
    
    # Wait for all processes to finish
    for p in processes:
        p.join()

Working with Process Pools

The Pool class provides a convenient interface for managing