RBS-Attention: A New Method to Enhance Long-Context Language Models

RBS-Attention introduces a novel sparse-prefill approach to improve the efficiency of long-context language model inference.

The method addresses the 'mean dilution' problem in sparse block selection.

It uses two complementary selection branches to enhance performance.

No additional training is required, making it easily adoptable.

Source: https://arxiv.org/abs/2609.20971