Recently, a research paper titled Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models by faculty from the School of Intelligent Manufacturing and Electrical Engineering and the Electrical Information Team of the Henan Provincial Collaborative Innovation Center for Intelligent Explosion-Proof Equipment, has been accepted by the 43rd International Conference on Machine Learning (ICML 2026). ICML is one of the most prestigious and influential top-tier academic conferences in the fields of international artificial intelligence and machine learning, with broad impact across foundation models, deep learning, statistical learning, and reinforcement learning. The paper was completed in collaboration with Nanyang Normal University, King’s College London (UK), and the University of Luxembourg. NYNU is the primary completion unit, and Dr. Gao Bo from NYNU is the first and corresponding author.

This paper addresses the problem of performance degradation and inaccurate key information capture when the Softmax self-attention mechanism processes ultra-long sequences. The authors propose a novel attention calculation method, LSSAR, designed to enhance the model’s ability to process long texts and complex information. Experimental results show that this method significantly outperforms the Softmax self-attention mechanism in tasks such as length extrapolation, long-text understanding, and information retrieval. Notably, it demonstrates a unique advantage in learning physical laws: on a model with only 109 million parameters (109M), using the LSSAR method successfully reproduced the core structure of Newton’s law of gravitation from planetary movement data. In contrast, a trillion-parameter model using the traditional Softmax self-attention mechanism failed to achieve equivalent results on a similar task. This indicates that the performance improvement of AI models depends not only on parameter scale but also on underlying methodological design.

Written & Photo by: Gao Bo & Yang Shiwei

Source: NYNU Academic Activities (Chinese)

https://www.nynu.edu.cn/info/1048/31624.htm