InstCache: A Predictive Cache for LLM Serving

Large language models are revolutionizing every aspect of human life. However, the unprecedented power comes at the cost of significant computing intensity, suggesting long latency and large energy footprint. Key-Value Cache and Semantic Cache have been proposed as a solution to the above problem, b...

Ausführliche Beschreibung

Bibliographische Detailangaben

Hauptverfasser:	Zou, Longwei, Liu, Tingfeng, Chen, Kai, Kong, Jiangang, Deng, Yangdong
Format:	Artikel
Sprache:	eng
Schlagworte:	Computer Science - Computation and Language Computer Science - Distributed, Parallel, and Cluster Computing
Online-Zugang:	Volltext bestellen
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!