Volume 2 - Issue 3, May - June 2026

Sentiment classifiers must resolve long-range dependencies and subtle polarity cues that sequential models handle poorly: RNNs propagate sentiment signals through a fixed-width hidden state, while CNNs are limited to a fixed receptive field. Transformers sidestep both bottlenecks via self-attention, yet a precise account of why they work well for sentiment—and how small they can be made—is largely missing from the literature. We address this gap in two ways. First, we formalize self-attention as a learnable kernel smoother and prove that a transformer encoder can represent any sentiment function over boundedlength sequences, and that it strictly subsumes every finite-order Markov model regardless of state size. Second, we introduce SentiFormer, a 22M-parameter transformer trained from scratch, incorporating a polarity-aware positional encoding and a pertoken gating mechanism that suppresses neutral words. On SST5, IMDb, Yelp, and Twitter, SentiFormer reaches 92.3%, 94.0%, 95.6%, and 86.5% accuracy—matching or exceeding BERT-base at one-fifth of its parameter count and without pre-training. Sparse attention reduces the per-forward-pass cost to O(n) in sequence length, enabling 8,000 sentences per second on a single V100.

Nandini Gupta, Karan Gupta, Bhoomi Agrawal, Saurabh Shrivastava,"A Theoretical Framework for Transformer-Based Sentiment Analysis: Attention, Expressivity, and Efficiency" International Journal of Advanced Multidisciplinary Research and Educational Development, V2(3): Page(764-768) May-June 2026. ISSN: 3107-6513. www.ijamred.com. Published by Scientific and Academic Research Publishing.