horovod

The framework implements ring-allreduce gradient synchronization, an algorithm that distributes gradient aggregation work evenly across all participating workers, eliminating the bottleneck of a central parameter server and achieving near-linear training throughput scaling.

Horovod supports TensorFlow, Keras, PyTorch, and Apache MXNet, with a consistent API across all frameworks so teams don't need to learn framework-specific distributed training primitives.

Converting a single-GPU training script to distributed Horovod training typically requires fewer than ten lines of additional code: initialize Horovod, pin each process to its GPU, wrap the optimizer with Horovod's DistributedOptimizer, and broadcast initial variable values to synchronize starting weights.

Horovod handles all inter-process communication using MPI or Gloo for CPU communication and NCCL for GPU-to-GPU transfers via NVLink or InfiniBand, automatically selecting the fastest available communication backend for the hardware topology.

Uber open-sourced Horovod in 2018 after using it internally to reduce training time for large-scale ML models from days to hours.

The framework integrates with cloud infrastructure providers (AWS SageMaker, Azure ML, Google Cloud AI Platform, Databricks) and popular ML platforms (Kubernetes, Spark, Ray), making it practical for both research clusters and production training pipelines.

Companies including Microsoft, Alibaba, and Lyft adopted Horovod for training recommendation systems, natural language models, and computer vision networks at scale.