OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking
OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking
Authors: Yiding Ma, Chengyun Ruan, Kaibo Huang, Zhongliang Yang, Linna Zhou Date: 2026-05-05 Paper ID: openalex:2605.03762
Summary
OracleProto is a reproducible framework designed to evaluate the forecasting capabilities of large language models while mitigating information leakage from pretraining data. By combining knowledge-cutoff-aligned sample admission, temporal masking, and content-level leakage detection, the framework transforms resolved historical events into reliable forecasting benchmarks. Experimental results demonstrate that OracleProto maintains leakage at the 1% level, enabling rigorous, auditable, and comparative assessments of LLM forecasting performance.
Key Contributions
- Introduces OracleProto, a reproducible benchmarking framework that reconstructs resolved events into time-bounded forecasting samples.
- Achieves leakage rates as low as 1%, representing a significant improvement over standard tool-only temporal filtering techniques.
- Provides a unified evaluation interface for measuring forecasting quality, sampling stability, and cost efficiency across multiple contemporary LLMs.
Open Questions & Future Work
Key Concepts
- oracleproto: A framework for creating reproducible LLM forecasting benchmarks by reconstructing resolved events with strict temporal masking and leakage detection.
Archivist Review
I have approved OracleProto as a central concept because it provides a systematic methodological framework for a critical issue in LLM-based time-series forecasting. FutureX-Past is approved as a central dataset benchmark. I have also added a high-level open question regarding the fundamental difficulty of disentangling predictive reasoning from memorized knowledge in LLMs, which is a major ongoing concern for the field.
Approved Concepts
- OracleProto: It establishes a systematic methodology for creating leakage-free, time-bounded forecasting benchmarks for LLMs by aligning data with model knowledge cutoffs, addressing a critical bottleneck in evaluating agentic forecasting.
Approved Open Questions
- LLM Forecasting Leakage Evaluation: This is a foundational problem for ensuring that LLM forecasting benchmarks measure intelligence rather than memorization.
Datasets
Links
Metadata & Links
- url
- https://arxiv.org/abs/2605.03762
- paper_id
- 2605.03762
- paper_source
- openalex
- domain
- nlp
- tags
- time-series-question-answering-tsqaagentic-time-series-forecasting
- architectures
-
- datasets
- FutureX-Past
- concept_slugs
- oracleproto
- dataset_slugs
- futurex-past
- skill
- TimeSeriesSkill
- created_at
- 2026-05-08T05:43:50Z