EdgeCIM: A Hardware-Software Co-Design for CIM-Based Acceleration of Small Language Models

arXiv:2604.11512v2 Announce Type: replace Abstract: The growing demand for deploying Small Language Models (SLMs) on edge devices, including laptops, smartphones, automotive and robotic computing platforms, and high-performance embedded systems, has exposed fundamental inefficiencies in existing accelerators. While GPUs handle prefill workloads efficiently, the autoregressive decoding phase is dominated by GEMV operations that are inherently memory-bound, resulting in poor utilization and prohibitive energy costs at the edge. In this work, we present EdgeCIM, a hardware-software co-design framework for accelerating autoregressive decoding of SLMs on edge devices in latency-critical applications.