Training on unreliable mixed GPUs across the internet

Audience: Users who do not have access to top-tier multi-gpu/multi-node servers and want to scale training across different GPU types, or across the internet.

1: Training across multiple machines over the internet

Quick setup to start training on multiple machines.

basic

2: Speed up training by enabling under-the-hood optimizations

Learn which flags to use with the HivemindStrategy to speed up training.

intermediate

3: Optimize Memory and Communication using compression hooks

Enable gradient buffer optimizations and communication improvements to reduce bottlenecks in communication.

expert