Parallel Programming in C++ with MPI and OpenMP

Introduction

Michael Quinn's book "Parallel Programming in C++ with MPI and OpenMP" (2003) is a classic textbook on parallel programming that remains relevant today. Despite being over 20 years old, the fundamental concepts of distributed memory (MPI) and shared memory (OpenMP) have not undergone radical changes. The book offers a balanced approach: from parallelism theory to practical examples in C++. It is ideal for students, engineers, and developers who want to master parallel computing at a serious level.

What the book is about

The book consists of 15 chapters covering all key aspects of parallel programming. The first part is devoted to the basics: parallel computing models, performance evaluation, synchronization issues, and data races. The author then moves on to the two main technologies:

• MPI (Message Passing Interface) — for clusters and distributed systems. Point-to-point and collective communications, derived data types, and virtual topologies are discussed in detail.
• OpenMP — for multi-core processors and shared memory. Compiler directives, thread management, synchronization, and loop parallelization are covered.

Separate chapters are devoted to hybrid programming (MPI+OpenMP), debugging, profiling, and optimization. Each topic is accompanied by code examples that can be immediately run and modified.

Who this book is for

The book is aimed at readers already familiar with C++ (preferably at an intermediate level or higher). Useful knowledge includes working with pointers, dynamic memory, and templates. Prior knowledge of parallelism is not required — the author introduces concepts gradually. The main audience:

• students in technical fields (computer science, computational mathematics);
• developers of high-performance computing (HPC) systems;
• engineers working with simulations, data processing, and machine learning;
• instructors of parallel programming courses.

What the reader will learn

After studying the book, the reader will be able to:

• understand parallel computing models (SIMD, MIMD, SPMD);
• write programs with MPI for data exchange between processes;
• use OpenMP for parallelizing loops and tasks;
• combine MPI and OpenMP for hybrid applications;
• measure speedup, efficiency, and scalability;
• avoid common mistakes: deadlock, race condition, false sharing;
• use profilers (gprof, Intel VTune) for optimization.

Strengths

• Practical focus. Each concept is supported by working code. Examples range from simple (array sum) to complex (parallel matrix multiplication, Floyd's algorithm).
• Structured approach. Chapters are logically connected: from simple to complex. Each chapter ends with a summary and exercises (with answers available online).
• Depth. The author does not just list MPI/OpenMP functions but explains why they work the way they do. Performance, latency, and bandwidth issues are discussed.
• Platform independence. Examples compile on Linux, Windows (MSVC, MinGW), and macOS. Only standard libraries are used.
• Relevance. Despite being from 2003, MPI-2 and OpenMP 2.0 remain the foundation for modern standards (MPI-4, OpenMP 5.x). The book provides a solid foundation.

Who it is recommended for

The book will be useful for anyone who wants to seriously study parallel programming. If you are a beginner, start with the first chapters, which explain basic concepts. If you already have experience, skip the introduction and focus on the chapters on hybrid programming and optimization. I especially recommend it to instructors: the book's structure is ideal for a semester-long course with