Compressing Gradients by Exploiting Temporal Correlation in Momentum-SGD

An increasing bottleneck in decentralized optimization is communication. Bigger models and growing datasets mean that decentralization of computation is important and that the amount of information exchanged is quickly growing. While compression techniques have been introduced to cope with the latte...

Ausführliche Beschreibung