Why Attention Fails: The Degeneration of Transformers into MLPs in Time Series Forecasting

Status: completed

Authors: Zida Liang, Jiayi Zhu, Weiqiang Sun

Venue / Year: arXiv:2509.20942, Sep 2025 (rejected from ICLR 2026)

Affiliations: Shanghai Jiao Tong University

Links: arXiv | PDF | Code | OpenReview (ICLR 2026 Rejection)

Tags: [[time series forecasting]], [[transformer]], [[attention mechanism]], [[MLP degeneration]], [[representation learning]], [[empirical analysis]]

One-Sentence Summary

Through systematic ablation experiments, this paper demonstrates that attention mechanisms in time-series Transformers (PatchTST, iTransformer, etc.) contribute negligibly to forecasting performance—effectively degenerating Transformer blocks into MLPs—and attributes this failure to the inadequacy of linear embeddings in constructing a structured latent space.

Problem

Core Challenge

Transformer-based models have achieved remarkable success in NLP and CV, but their effectiveness in time series forecasting remains controversial. Prior work (Zeng et al., 2023) showed that simple linear models can outperform Transformers, yet the underlying reasons for Transformer failure have not been thoroughly investigated at the mechanism level.

Motivation

The motivation is scientific rather than engineering-driven: understanding why attention fails in time series is critical before proposing fixes. The authors argue that the community has been proposing increasingly complex attention variants (Auto-correlation, Fourier-enhanced, sparse attention) without addressing the root cause—whether attention is even functioning as intended.

Prior Work Gaps

1. Zeng et al. (2023): Showed linear models outperform Transformers but did not investigate the attention mechanism's internal behavior
2. SAMformer (2024): Identified attention as causing sharp minima but did not analyze degeneration
3. PITS (2024): Used patch-wise MLPs but did not explain why MLPs suffice
4. Kim et al. (2024): Eliminated self-attention but used cross-attention as replacement, not studying the degeneration phenomenon
5. RAM (2024): Replaced attention with MLP but observed performance drops, which this paper argues is due to parameter reduction rather than attention's actual contribution

Method

Core Innovation

This is primarily an empirical analysis paper with a conceptual contribution. The key innovations are:

1. Degeneration phenomenon discovery: Transformer blocks in time-series models effectively operate as MLPs
2. Multi-perspective experimental validation: Four complementary experiment types confirming degeneration
3. Interpretable toy dataset: State-machine-based synthetic data revealing attention's failure to capture inter-patch dependencies
4. Theoretical analysis of linear embedding failure: Mathematical argument for why linear embeddings cannot provide structured latent spaces

Architecture Analysis

The paper does not propose a new architecture. Instead, it analyzes existing ones:

Component	PatchTST	iTransformer
Tokenization	Patch-wise (unfold)	Channel-wise (entire series per variable)
Embedding	Linear projection	Linear projection
Positional Encoding	Learnable additive	None (channel tokens)
Attention	Multi-head self-attention (residual)	Multi-head self-attention
FFN	2-layer MLP with GELU	2-layer MLP with ReLU
Normalization	RevIN + BatchNorm	RevIN + LayerNorm
Output Head	Flatten + Linear	Linear per channel

Key Experiments

Experiment 1: Attention Replacement (Section 3.1)

Replace the attention matrix in with:

1. Zero attention: (reduces block to FFN only)
2. Identity attention: (each token sees only itself)
3. Mean attention: (uniform averaging, no context)
4. Trainable fixed attention: is a learnable parameter matrix (context-independent)

Experiment 2: Attention Perturbation (Section 3.2)

Post-hoc perturbation of trained models:

where and .

Experiment 3: Varying Patch Length (Section 3.3)

Increasing patch length until a single token remains, reducing the entire model to an MLP.

Experiment 4: Positional Encoding Zeroing (Section 3.4)

Zeroing learned positional encodings in trained models to test whether attention utilizes positional information.

Toy Dataset Design (Section 4)

A synthetic time series with three components:

• Carrier: Sinusoidal wave , ,
• Events: Triangular waveforms controlled by a state machine (4 states, period )
• Noise:

The state machine has deterministic transitions, making the "correct" attention pattern known a priori.

Theoretical Analysis (Section 5)

The paper argues that linear embedding is an isomorphic linear transformation when input and output dimensions match. Since the latent space (encoding semantic/structural information) cannot be isomorphic to the raw time series space, linear embedding is fundamentally insufficient.

Key argument: In NLP, embedding maps discrete one-hot vectors to continuous space (a non-trivial transformation). In time series, the input is already continuous, so a linear layer merely applies an affine transformation without creating meaningful structure.

Evidence

Claims → Evidence Mapping

Claim	Type	Evidence	Strength	Risk
Attention replacement does not degrade performance	Empirical	Tables 7-14 (PatchTST, iTransformer across 5 datasets, 4 prediction lengths)	Strong	Consistent across settings, but ECL/Traffic show slight drops with zero attention
FFN perturbation degrades performance far more than attention perturbation	Empirical	Figure 1 (3D surface plots)	Strong	Only shown for PatchTST on ETTm2
Increasing patch length to single-token does not hurt performance	Empirical	Figure 3b	Strong	Only PatchTST, stride=patch_len
Positional encoding zeroing has no effect on PatchTST	Empirical	Table 2	Strong	Contrasts sharply with ViT (89.8% → 32.4%)
Attention fails to focus on event patches in toy dataset	Empirical	Figure 7 (attention visualization)	Medium	Single toy dataset, qualitative
Linear embedding is neither effective nor necessary	Empirical + Theoretical	Tables 23-24 (frozen embedding), Section 5.1 argument	Medium	Theoretical argument is informal, not a formal proof
ViT's early blocks degenerate but later blocks use attention	Empirical	Table 3 (selective smoothing)	Medium	Only CIFAR-10, depth-5 ViT
Degeneration extends to foundation models	Empirical	Tables 19-22 (Lag-Llama, SAMformer, Timer, Moirai)	Medium	Limited datasets per model, no error bars for some

Key Results

Attention Replacement on PatchTST (pred_len=96):

Dataset	Metric	Raw	Eye	Zero	Mean
ETTh1	MSE
ETTh2	MSE
ETTm2	MSE
Weather	MSE

Remarkably, zero/eye/mean attention often improves over raw attention, suggesting the attention mechanism introduces noise rather than signal.

Foundation Model Results (Lag-Llama, CRPS):

Dataset	RAW	MEAN	EYE	ZERO
ECL	0.050	0.068	0.044	0.051
Traffic	0.114	0.114	0.114	0.120
Weather	0.149	0.150	0.153	0.147

Foundation models show more mixed results—some datasets show degradation with attention replacement, suggesting the degeneration may be less severe in larger, pre-trained models.

Limitations

1. No constructive solution: The paper identifies the problem but offers no concrete fix. Appendix D discusses VQ-VAE/RQ-VAE as potential solutions but provides no experimental validation.

2. Limited model diversity: Primary experiments focus on PatchTST and iTransformer. Foundation model experiments (Section C.3) cover more models but with fewer datasets and less rigorous error bars.

3. Informal theoretical analysis: The "proof" of linear embedding inadequacy (Section 5.1) is an informal argument about isomorphism, not a rigorous theorem. The math_proof/validation.py only numerically checks a trigonometric identity for .

4. Single toy dataset: The state machine dataset (Section 4) is custom-designed and may not generalize to real-world time series characteristics.

5. No causal intervention on representations: The paper shows attention is unused but does not experiment with alternative embedding strategies that might "activate" attention.

6. Confounding from residual connections: PatchTST uses residual attention (Realformer-style), where attention scores accumulate across layers. The analysis does not fully account for how this affects the degeneration.

7. Missing statistical tests: While error bars are reported for main experiments, no formal statistical significance tests (e.g., paired t-tests, Wilcoxon) are performed to confirm that differences between attention variants are truly negligible.

Critical Assessment

Strengths

• Compelling core finding: The degeneration phenomenon is well-demonstrated and the experimental design (replacing rather than ablating attention) cleverly controls for parameter count
• Multi-angle validation: Four complementary experiments converge on the same conclusion, strengthening the claim
• ViT contrast experiment: The comparison with ViT (Section 5.3) provides an insightful counterexample showing that attention can work when representations are properly structured
• Practical relevance: The finding has direct implications for the time series forecasting community, potentially redirecting research effort from attention design to representation learning
• Extensive appendix: Detailed experimental settings, additional results, and discussion of concurrent work

Weaknesses & Risks

Fatal Flaws:

• None that invalidate the core finding

Significant Concerns:

1. Overclaiming: The paper claims attention "fails" universally, but the foundation model results (Tables 19-22) show more nuanced behavior. Lag-Llama on ECL shows CRPS degradation from 0.050 (RAW) to 0.068 (MEAN), suggesting attention does contribute in some settings.
2. Theoretical weakness: The isomorphism argument (Section 5.1) conflates "linear transformation" with "trivial transformation." A learned linear projection can create useful latent spaces (e.g., PCA, random projections for compressed sensing). The argument that "linear space cannot encode semantic structure" is not rigorously justified.
3. Incomplete ViT analysis: The ViT experiment uses CIFAR-10 (32×32 images, patch size 4), which is far from standard ViT settings (ImageNet, patch size 16). The "early blocks degenerate, late blocks use attention" finding may not generalize.
4. Missing counterfactual: If linear embedding is the problem, what happens with a better embedding? The multi-embedding experiment (conv, MLP, residual block) in the code shows degeneration persists, but this is not discussed in the main paper.

Minor Issues:

1. Writing quality: Several typos ("forcasting", "Attetion"), inconsistent notation
2. Reproducibility concerns: The perturbation notebook references softmax_layer attribute that does not exist in the provided backbone code
3. Hardcoded values: The fixed attention matrix shape [1, n_heads, 22, 22] is hardcoded for a specific patch configuration

Reviewer Feedback (ICLR 2026)

The paper was rejected from ICLR 2026. While I could not access the specific reviews due to OpenReview access restrictions, the rejection is consistent with the following likely concerns:

1. Lack of constructive contribution (diagnosis without cure)
2. Insufficient theoretical rigor for the "linear embedding failure" claim
3. The degeneration phenomenon, while interesting, may be considered incremental over Zeng et al. (2023) and RAM (2024)
4. Foundation model experiments are too limited to support the broad claims

Innovation Score: 5/10

The paper makes a solid empirical observation (attention degeneration) with good experimental support, but the contribution is primarily diagnostic. The theoretical analysis is informal, and no constructive solution is proposed. The finding is valuable for the community but represents an incremental advance over prior negative results about Transformers for time series.

Reusable Ideas

1. Attention replacement as ablation: Replacing attention with identity/mean/zero matrices while preserving parameter count is a cleaner ablation than removing the attention module entirely
2. Toy dataset with known ground truth: Designing synthetic data where the "correct" attention pattern is known a priori is a powerful interpretability tool
3. ViT vs TST contrast: Comparing the same architectural components across domains (vision vs time series) to isolate domain-specific failure modes
4. Perturbation sensitivity analysis: The 2D grid of attenuation × noise for comparing module importance is a general technique applicable to other architectures

Related Work

Paper	Main Contribution	vs This Paper
Zeng et al. (2023) - DLinear	Linear models outperform Transformers	Identified the symptom; this paper investigates the mechanism
SAMformer (2024)	Sharp minima cause poor generalization	Different diagnosis (optimization landscape vs representation)
PITS (2024)	Patch-wise MLPs work well	Empirical validation without degeneration analysis
RAM (2024)	Replace attention with MLP	Ablation causes performance drop; this paper argues parameter reduction is the confound
Kim et al. (2024)	Self-attention ineffective, cross-attention works	Complementary finding; different replacement strategy
TSMixer (2023)	All-MLP architecture for forecasting	Validates that attention is unnecessary, but does not explain why
Pathformer (2024)	Multi-scale adaptive pathways	This paper shows Pathformer also degenerates (Tables 15-18)

Positioning: This paper sits at the intersection of empirical analysis and mechanistic interpretability for time-series Transformers. It goes beyond prior negative results by providing multi-perspective evidence and a (partial) explanation rooted in representation learning.

Personal Notes

Key Takeaway

The core insight—that attention degenerates because linear embeddings fail to create a structured latent space—is intuitively compelling but theoretically underdeveloped. The paper correctly identifies that the problem is upstream of attention (in the embedding), not in attention itself.

Connection to Own Research

For precipitation/radar nowcasting, this finding is relevant because:

• Many weather forecasting Transformers (MetNet, DGMR, etc.) use learned embeddings
• The degeneration phenomenon may extend to spatiotemporal forecasting
• The ViT contrast suggests that when patches have clear spatial structure (as in radar images), attention may work better than in pure time series

Follow-up Decision

• [x] Worth referencing: The degeneration phenomenon and experimental methodology
• [ ] Would not directly build upon: The paper is diagnostic, not constructive
• [ ] The VQ-VAE/RQ-VAE direction (Appendix D) is worth exploring independently

Code Verification

Claims → Code Mapping

Claim	Paper Description	Code Implementation	Status
Attention replacement (zero/eye/mean/fix)	Section 3.1, four strategies	Attention_Replacement/PatchTST/PatchTST_backbone.py: env-var controlled branching in _ScaledDotProductAttention.forward()	Match
Attention perturbation	Section 3.2, noise + attenuation	Attention_Perturbation/PatchTST.ipynb: forward hooks on attention dropout layers	Partial match (hook target softmax_layer not in backbone)
Positional encoding zeroing	Section 3.4	PosEnc_Zeroing_Out/PatchTST.ipynb: replaces W_pos with constant 0.5 tensor	Match
ViT contrast experiment	Section 5.3	ViT/ViT.py: forward hooks to zero attention in selected layers	Match
Toy dataset	Section 4.1, state machine	Toy-Datatset-Exp/data_loader.py: Dataset_Patch_dependent class with StateMachine	Match
Multi-embedding types	Not in main paper	Attention_Replacement/multi-emb-type/: linear, conv, MLP, residual block embeddings	Extra

Hyperparameter Consistency

Hyperparameter	Paper Report	Code Default	Discrepancy
d_model	Not explicitly stated	128	N/A
n_heads	Not explicitly stated	16	N/A
e_layers	Not explicitly stated	3	N/A
patch_len	Not explicitly stated	16	N/A
stride	Not explicitly stated	16	N/A
dropout	Not explicitly stated	0.0 (backbone) / 0.2 (shell)	Minor
learning_rate	Not explicitly stated	0.0001	N/A
Fixed attention shape	Not stated	[1, n_heads, 22, 22] hardcoded	Problematic

Implementation Quality

• Structure: Modular but minimal—no package structure, no __init__.py
• Documentation: Inline comments only, no docstrings
• Dependency management: No requirements.txt or pyproject.toml
• Error handling: Minimal; hardcoded device, hardcoded tensor shapes
• Testing: No unit tests

Reproducibility Assessment

• Code availability: Public GitHub repo with anonymous backup
• Randomness control: Seed 3407 for ViT; no explicit seed for PatchTST experiments
• Environment: No environment specification; dependencies must be inferred from imports
• Data: Uses standard ETT/Weather/ECL benchmarks (publicly available)
• Overall: Moderate reproducibility—the core experiments can be replicated but require manual setup