Skip to content

Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Home / Papers / Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Authors: Efstathios Karypidis, Spyros Gidaris, Nikos Komodakis Date: 2026-04-13 Paper ID: openalex:2604.11707

Summary

Re2Pix is a hierarchical video prediction framework that addresses the trade-off between semantic consistency and visual fidelity by decomposing the forecasting process. Instead of predicting pixels directly, the model first forecasts future scene dynamics within the feature space of a pre-trained foundation model and subsequently uses these representations to condition a latent diffusion model for frame synthesis. To overcome the accumulation of errors in autoregressive prediction, the authors introduce nested dropout and mixed supervision techniques. Experiments on autonomous driving benchmarks show that this semantics-first approach significantly enhances both perceptual quality and temporal coherence.

Key Contributions

  • Introduces Re2Pix, a hierarchical prediction framework that decouples semantic feature forecasting from latent diffusion-based pixel synthesis.
  • Proposes nested dropout and mixed supervision strategies to mitigate the train-test distribution shift inherent in autoregressive representation prediction.
  • Demonstrates improved temporal semantic consistency and visual fidelity on autonomous driving benchmarks compared to standard diffusion-based video prediction baselines.

Open Questions & Future Work

Key Concepts

Archivist Review

The concept of decoupled hierarchical prediction for video generation was approved as a core contribution to the field of generative models. The open question was reframed to focus on the fundamental challenge of train-test mismatch in multi-stage generation, which is a known, significant bottleneck in such architectures. Other candidates were rejected for being overly model-specific (Re2Pix) or generic application-level extensions (controllability).

Approved Concepts

  • Semantic Representation Prediction for Video Synthesis: The paper formalizes a decoupled hierarchical paradigm that is likely to become a standard pattern in generative video modeling, distinct from monolithic frame-to-frame approaches.

Approved Open Questions

  • Hierarchical Forecasting Distribution Shift: Addressing the error accumulation and distributional mismatch between stages is critical for the practical viability of multi-stage generative models.

Rejected Candidates

  • [concept] Re2Pix (re2pix) - subcomponent_of_broader_mechanism: This is a specific model architecture name; the underlying technique (hierarchical decoupled forecasting) is the more reusable and significant concept.
  • [open_question] Expanding VFM Feature Guidance Diversity (expand-vfm-feature-guidance-diversity) - low_impact: This is framed more as a set of incremental experiments (using different models) rather than a fundamental theoretical bottleneck.
  • [open_question] Integrating Controllability in Hierarchical Generation (integrating-controllability-in-hierarchical-generation) - generic: Controllability is a general generative modeling challenge; the candidate does not define a specific, novel technical bottleneck unique to hierarchical video architectures.

Metadata & Links

url
https://arxiv.org/abs/2604.11707
paper_id
2604.11707
paper_source
openalex
domain
computer-vision
tags
computer-visiongenerative-modelsvideo-predictiondiffusion-models
architectures
datasets
concept_slugs
semantic-representation-prediction-video-synthesis
dataset_slugs
skill
TimeSeriesSkill
created_at
2026-04-16T05:47:06Z