Recent advances in large language models have significantly improved their
ability to process long-context input, but practical applications are
challenged by increased inference time and resource consumption, particularly
in resource-constrained environments. To address these challenges, we propose
MOOSComp, a token-classification-based long-context compression method that
enhances the performance of a BERT-based compressor by mitigating the
over-smoothing problem and incorporating outlier scores. In the training phase,
we add an inter-class cosine similarity loss term to penalize excessively
similar token representations, thereby improving the token classification
accuracy. During the compression phase, we introduce outlier scores to preserve
rare but critical tokens that are prone to be discarded in task-agnostic
compression. These scores are integrated with the classifier’s output, making
the compressor more generalizable to various tasks. Superior performance is
achieved at various compression ratios on long-context understanding and
reasoning benchmarks. Moreover, our method obtains a speedup of 3.3x at a 4x
compression ratio on a resource-constrained mobile device.

Cet article explore les excursions dans le temps et leurs implications.

Télécharger PDF: