TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout

arXiv:2607.04031v2 Announce Type: replace Abstract: Large Language Model (LLM) inference is bottlenecked by the capacity and bandwidth of GPU High-Bandwidth Memory (HBM). Recent proposals, such as High-Bandwidth Flash (HBF) and RoMe, offer higher capacity or bandwidth than HBM, but require a minimum access granularity of kilobytes. We show that these Large-Granularity Memory Systems (LGMS) can degrade LLM inference performance by up to an order of magnitude. The root cause is read amplification during tiled matrix multiplication, in which each memory request fetches far more data than the compute tile needs.