Overlapping Communication With Computation in Parameter Server for Scalable DL Training

Scalability of distributed deep learning (DL) training with parameter server (PS) architecture is often communication constrained in large clusters. There are recent efforts that use a layer by layer strategy to overlap gradient communication with backward computation so as to reduce the impact of c...

Ausführliche Beschreibung