PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
By Yuchen Yang · Paper · cs.LG
Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements of the model weights and the growing KV cache. We