Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM

Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience while preserving intelligent capabilities. However, such a task exhibits single-batch computing with incredibly low arith...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	arXiv.org 2024-09
Hauptverfasser:	Yu, Zhongkai, Liang, Shengwen, Ma, Tianyun, Cai, Yunke, Ziyuan Nan, Huang, Di, Song, Xinkai, Hao, Yifan, Zhang, Jie, Tian Zhi, Zhao, Yongwei, Du, Zidong, Hu, Xing, Guo, Qi, Chen, Tianshi
Format:	Artikel
Sprache:	eng
Schlagworte:	Computation Computer architecture Inference Large language models Robotics Tiling
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!