Elastic Precision: Software-Hardware Co-Design of a Mixed-Precision Architecture for Long-Context LLM Inference

arXiv:2611.00915v1 Announce Type: new Abstract: Low-bit quantization reduces the data movement and computation costs of large language model (LLM) inference, but uniform widths can waste precision and degrade model quality. Existing mixed-precision methods do not fully account for how the other input of the current GEMM tile affects output error when selecting one input's width. Using both inputs to guide precision introduces additional estimation and selection overheads on conventional hardware. We propose Coffee, a runtime precision allocation architecture for long-context LLM inference.