MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregress...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	arXiv.org 2024-09
Hauptverfasser:	Liu, Akide, Liu, Jing, Pan, Zizheng, He, Yefei, Haffari, Gholamreza, Bohan Zhuang
Format:	Artikel
Sprache:	eng
Schlagworte:	Compression ratio Inference Large language models State vectors
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!