Compress to Remember
Learning Compact Memory via On-Policy Distillation
for Long Video Generation
1 Carnegie Mellon University2 Rice University* Equal contribution
arXiv
PDFAbstract
Long video generation needs a memory of what happened earlier. Retaining the entire history is expensive; selecting a few frames can discard useful information. Prediction-Aligned Context Compaction (PACC) learns a small set of memory tokens that preserve what a frozen video generator needs to predict next.
We train the compressor to match the generator's full-history predictions on its own generated continuations. At inference, completed video blocks become a compact memory archive. The same generator retrieves from this archive as the video grows, improving memory consistency on both Causal-rCM and Causal Forcing.
Method
Keep the generator. Learn what it should remember.
Data rollout
The teacher generates the initial chunks; the student continues from compressed memory and appends each new chunk to the rollout prefix H.
Training
Match predictions along that rollout.
Instead of reconstructing old frames, we ask a more useful question: can the model predict the same next step from a compact memory? Only the compressor is trained.
At each denoising state, ask the frozen teacher what it would predict with the full prefix. Match that prediction, then average the losses across the rollout to train the compressor.
Generation
From short training videos to long generation
Once a block has been generated, compress it and keep its memory. For each new chunk, retrieve a memory block and retain the raw beginning and recent frame as anchors.
Results
Memory improves on two frozen backbones
With the generator frozen, PACC improves memory consistency on both Causal-rCM and Causal Forcing.
MBench M-score ↑
Citation
@article{wu2026compress,
title={Compress to Remember: Learning Compact Memory via
On-Policy Distillation for Long Video Generation},
author={Wu, Xiaoyu and Guo, Weihang and Wang, Yifei and
Feng, Xinze and Kavraki, Lydia E. and Wu, Zhiwei Steven},
journal={arXiv preprint arXiv:2609.36364},
year={2026}
}