Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference
Reducing model size does not always accelerate inference, because irregular sparse operations can introduce substantial overhead. We introduce S-SALAAD, which learns low-rank components together with hardware-compatible structured sparsity during training. Budget-aware compression and dedicated CPU and GPU kernels let a single checkpoint serve devices with different memory limits. Experiments on Llama2 and Qwen3, spanning pretraining and distillation, demonstrate improved trade-offs among perplexity, memory consumption, and inference latency relative to the evaluated baselines, supporting efficient deployment of large models on edge devices.