mpi

MPI

Purpose

Guide agents through MPI (Message Passing Interface) programming: point-to-point and collective communication, non-blocking operations, subcommunicators, MPI+OpenMP hybrid patterns, process launching with mpirun, debugging techniques, MPI-IO, and common performance issues.

When to Use

• Parallelizing across multiple nodes or sockets
• Implementing distributed algorithms (matrix decompose, FFT)
• Combining MPI process parallelism with OpenMP thread parallelism
• Running HPC jobs with Slurm/PBS + mpirun
• Debugging deadlocks and message mismatches
• Parallel file I/O with MPI-IO

Workflow

1. Minimal MPI program

#include <mpi.h>
#include <stdio.h>

int main(int argc, char **argv) {
    MPI_Init(&argc, &argv);

    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    printf("Hello from rank %d of %d\n", rank, size);

    MPI_Finalize();
    return 0;
}

mpicc -o hello hello.c
mpirun -np 4 ./hello
# or
mpiexec -n 4 ./hello

2. Point-to-point

if (rank == 0) {
    int data = 42;
    MPI_Send(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
} else if (rank == 1) {
    int recv;
    MPI_Recv(&recv, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
    printf("rank 1 got %d\n", recv);
}

Tagged messages: match tag and source for MPI_Recv.

3. Collectives

int local = rank + 1;
int global_sum;

MPI_Allreduce(&local, &global_sum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);

// Broadcast
if (rank == 0) data = 100;
MPI_Bcast(&data, 1, MPI_INT, 0, MPI_COMM_WORLD);

// Scatter/Gather
MPI_Scatter(sendbuf, sendcount, MPI_INT, recvbuf, recvcount, MPI_INT, 0, MPI_COMM_WORLD);
MPI_Gather(sendbuf, sendcount, MPI_INT, recvbuf, recvcount, MPI_INT, 0, MPI_COMM_WORLD);

Collective	Purpose
MPI_Bcast	One-to-all
MPI_Scatter	Distribute chunks
MPI_Gather	Collect chunks
MPI_Allreduce	Reduce + broadcast result
MPI_Barrier	Synchronization
MPI_Alltoall	All-to-all exchange