Back to workDistributed Systems
Distributed All-Reduce
C++ and MPI implementation of core distributed deep learning reduction algorithms.
The problem
Synchronizing gradients across multi-node distributed training clusters creates severe network bottlenecks if communication topology is not optimized for message size and latency.
Approach
Implemented and benchmarked multiple All-Reduce collective communication strategies in C++ using MPI: a Star-topology baseline, a latency-optimal Binomial Tree All-Reduce O(log N), and a bandwidth-optimal Ring All-Reduce (Scatter-Reduce followed by All-Gather). Includes runtime configuration and Doxygen-generated documentation.
At a glance
Star, Tree, Ring
Topologies
Latency vs. bandwidth optimized
Constant per node
Ring Complexity
Bandwidth-optimal for large tensors
C++ / OpenMPI
Language & API
Low-level distributed memory
Doxygen + Makefile
Tooling
Automated docs and benchmarking
Stack
C++MPIOpenMPIDistributed SystemsParallel ComputingDoxygen