Superposition in Complex Time-Series Domains
Background: Transformers for time series forecasting are often evaluated on standard benchmarks where simple linear models remain highly competitive, raising questions about whether transformers utilize complex representational mechanisms like superposition. Sparse autoencoders (SAEs) successfully uncover superposed features in language models, but similar signatures are often absent in smaller, task-specific time series models.
Question / Future Work: It remains unknown whether the lack of detectable superposition in standard time series benchmarks is a fundamental characteristic of the data or a failure of current interpretability methods. Future research must determine if high-stakes, high-complexity time series domains—such as clinical or financial forecasting—exhibit representational superposition that justifies the use of deep transformer architectures.
Why It Matters: This is critical for understanding whether current forecasting benchmarks are too simple to warrant advanced transformer architectures and for defining the scope of future foundation models.
Evidence: While our analysis reveals that standard forecasting benchmarks do not demand complex sparse representations, this finding points toward domains where the opposite may hold. … Medical time series … may exhibit precisely the kind of representational complexity that these benchmarks lack.
Metadata & Links
- created_at
- 2026-05-09T05:56:39Z