HACompBench: Co-designed Multimodal DNN Compression Evaluation for Edge Devices

Deployment of deep neural networks on edge devices faces challenges from heterogeneous hardware and multimodal tasks, where existing compression evaluation frameworks overlook hardware co-design, leading to suboptimal performance. To address this, we introduce HACompBench, a new hardware-aware framework that defines compression evaluation as a multi-objective optimization problem and combines hardware metrics such as quantization efficiency ξ and sparsity compatibility η with a dynamic scoring function J. We performed comprehensive experiments across four leading SoC platforms: Snapdragon 888, Snapdragon 765G, Kirin 970, and Jetson Nano P3450, and tested ten DNN models covering vision, text, and speech modalities using compression techniques such as quantization, pruning, and weight sharing, revealing hardware-induced performance gaps, such as quantization yields J=32.3% on Snapdragon 888 but J=68.0% on Jetson Nano P3450 due to INT8 emulation overhead. These results systematically highlight compression variations from differences in parallel processing capabilities. HACompBench innovates by linking hardware features, supporting multimodal tasks, and surpassing MLPerf Tiny’s single-modality focus and AIoTBench’s lack of co-design through embedded metrics ξ and η, while modality-specific corrections improve accuracy by up to 12.6%. It provides a unified and robust framework for edge deployment.