Skip to content

LLM Forecasting Leakage Evaluation

Home / Open Questions / LLM Forecasting Leakage Evaluation

Background: Evaluating the forecasting ability of LLMs is complicated by the presence of training data that may contain the answers to future-dated events, often leading to performance inflation via pretraining memorization.

Question / Future Work: Establishing a robust boundary between a model’s intrinsic predictive reasoning and its pre-existing knowledge of historical outcomes remains a significant challenge for auditing LLM-based forecasting systems.

Why It Matters: This is a foundational problem for ensuring that LLM forecasting benchmarks measure intelligence rather than memorization.

Evidence: retrospective benchmarks are reproducible, but they cannot reliably distinguish genuine forecasting from facts a model may have already learned during pretraining.

Metadata & Links