Skip to content

Forecasting Downstream Performance of LLMs With Proxy Metrics

Home / Papers / Forecasting Downstream Performance of LLMs With Proxy Metrics

Forecasting Downstream Performance of LLMs With Proxy Metrics

Authors: Arkil Patel, Siva Reddy, Marius Mosbach, Dzmitry Bahdanau Date: 2026-05-18 Paper ID: openalex:2605.18607

Summary

This paper addresses the challenge of accurately forecasting downstream language model performance without expensive direct evaluation. The authors propose proxy metrics based on token-level statistics, such as entropy and token rank, computed over expert-written solutions rather than relying on standard cross-entropy loss. These proxies consistently outperform traditional compute-based and loss-based baselines in model selection, data curation, and training-time extrapolation scenarios. The results demonstrate that leveraging expert-curated data provides a robust, low-cost signal for assessing LLM capabilities across the development life cycle.

Key Contributions

  • Introduces proxy metrics derived from token-level statistics on expert-written solutions to forecast downstream LLM performance.
  • Achieves mean Spearman Rho of 0.81 for cross-family model selection, significantly outperforming cross-entropy loss (0.36).
  • Demonstrates pretraining data selection at 10,000x lower compute cost than direct evaluation, improving the Pareto frontier.
  • Enables reliable training-time accuracy extrapolation across 18x compute horizons with reduced error compared to baseline methods.

Key Concepts

  • expert-trajectory-proxy-metrics: A forecasting framework that estimates downstream LLM capabilities by aggregating token-level statistics from the model’s output distribution on expert-curated solution sequences.

Archivist Review

I have approved the core methodological concept of using expert-written trajectories as a signal for LLM forecasting, as it provides a superior alternative to cross-entropy loss for downstream capability estimation. I have rejected the generic tags and task categories provided in the original input, as they do not meet the criteria for standalone technical contributions in the vault.

Approved Concepts

  • Expert-Trajectory-Proxy-Metrics: Provides a computationally efficient and more aligned alternative to cross-entropy loss for forecasting downstream LLM performance.

Rejected Candidates

  • [concept] llm-training-efficiency (llm-training-efficiency) - generic: This is a generic field label rather than a specific algorithmic contribution.
  • [concept] performance-forecasting (performance-forecasting) - generic: This is a high-level task category rather than a specific methodology.
  • [concept] model-selection (model-selection) - generic: A standard model engineering practice, not a unique contribution.
  • [concept] data-curation (data-curation) - generic: A common machine learning pipeline phase.

Metadata & Links

url
https://arxiv.org/abs/2605.18607
paper_id
2605.18607
paper_source
openalex
domain
nlp
tags
llm-training-efficiencyperformance-forecastingmodel-selectiondata-curation
architectures
datasets
concept_slugs
expert-trajectory-proxy-metrics
dataset_slugs
skill
TimeSeriesSkill
created_at
2026-05-21T06:26:41Z