Packrat: Automatic Reconfiguration for Latency Minimization in CPU-based DNN Serving

In this paper, we investigate how to push the performance limits of serving Deep Neural Network (DNN) models on CPU-based servers. Specifically, we observe that while intra-operator parallelism across multiple threads is an effective way to reduce inference latency, it provides diminishing returns....

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	arXiv.org 2023-11
Hauptverfasser:	Bhardwaj, Ankit, Phanishayee, Amar, Narayanan, Deepak, Tarta, Mihail, Stutsman, Ryan
Format:	Artikel
Sprache:	eng
Schlagworte:	Artificial neural networks Central processing units CPUs Inference Optimization Reconfiguration Servers
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!