Movendi Research Note No. 4
Persistent Predictive State: A Research Framework for Reality Models
Research Hypothesis
A compact, persistent representation of a bounded physical environment may support reliable physical prediction when its state is maintained through the joint inference of dynamics, correspondence, and uncertainty.
Abstract
A physical environment persists even when its observations do not. Sensors sample different surfaces as viewpoints change. Objects become occluded and reappear. Measurements arrive late, disappear, or contradict an earlier estimate. A useful Reality Model must maintain continuity through these changes while remaining able to revise what it believes.
This note develops a research framework for studying that requirement. It defines predictive state relative to a bounded operating domain, a set of prediction targets, and explicit resource constraints. It investigates persistent attributed points as a candidate representation, distinguishing temporary sensor measurements from model elements whose continuity is inferred over time. It then proposes testable hypotheses and an initial experimental protocol.
The purpose is to turn the direction established in the first three Movendi Research Notes into questions that can be implemented, compared, and challenged. This note presents a proposed framework and experiments; it does not report new empirical results.
Foreword
MRN-001 introduced prediction as the foundation of our research. MRN-002 narrowed the problem to bounded physical environments and defined the properties a Reality Model should possess. MRN-003 explored persistent points as a possible substrate for representing physical state, with correspondence as a central challenge.
This fourth note asks how those ideas can be tested.
Leroy & Leroy (2026) [11] develop the broader position that models of physical sites must discover and revise the state needed for prediction. The present note examines persistent attributed points and a bounded experimental protocol within that research direction. The position paper does not prescribe points or empirically validate the hypotheses proposed here.
What must a model remember? What should it predict? How can we tell whether it preserves physical continuity rather than producing a plausible sequence of observations? And under what conditions would a specialized representation justify its complexity?
As with the preceding notes, the framework remains provisional. It is intended to support experiments and discussion, including results that lead us to revise the underlying assumptions.
Executive Summary
A Reality Model should maintain the information required to predict the aspects of its environment that matter. That information may include geometry, motion, relationships, hidden properties, past observations, and unresolved alternatives. It should persist across interruptions in sensing without becoming detached from new evidence.
We propose organizing the research around four requirements:
- A defined operating domain. Specify the environment, sensing conditions, available actions, prediction horizons, and expected variation.
- A maintained predictive state. Retain enough information from the observation history to answer the required prediction queries.
- Revisable continuity. Preserve useful structure through time while allowing uncertain associations, mistaken estimates, and obsolete state to change.
- Measurable tradeoffs. Evaluate accuracy, uncertainty, memory, computation, adaptation, and engineering effort together.
Persistent attributed points provide one candidate representation. Their value must be established against recurrent latent states, object-based representations, geometric trackers, and other appropriate alternatives. The first experiment should isolate the effects of persistence and predictive correspondence before attempting to establish the superiority of points themselves.
From Observations to Predictive State
Consider an object moving behind an obstruction in a workspace. A camera stops observing it, but a model may still need to estimate its position, anticipate its reappearance, and account for an action that could change its trajectory.
When the object becomes visible again, the sensor generally does not recover the same samples it measured before. The physical structure may be continuous while the measurements are not.
This is the distinction between observation and state.
An observation is evidence acquired at a particular time and from a particular sensing configuration. State is the model's maintained estimate of what that evidence implies about the environment. It may contain several competing explanations rather than a single confident reconstruction.
We use the following working definition:
A Reality Model is a learned computational system that maintains a compact predictive state of a specified physical environment, updates that state from new evidence, and represents uncertainty about its present and future evolution.
The definition describes a research target within the broader study of world models. Its usefulness will depend on the experiments and systems it enables.
Bounding the Problem
A bounded environment is not necessarily a small space. A room containing unconstrained human activity may be harder to predict than a larger machine with a limited set of operating modes.
For each experiment, we should specify:
- The physical region and processes being modeled.
- The available sensors, their timing, and their limitations.
- The actions supplied to the model and the external interventions it cannot observe directly.
- The future quantities and events to be predicted.
- The prediction horizons and expected environmental variation.
- The permitted memory, computation, latency, training data, and adaptation effort.
These choices define what the model needs to preserve. Fine texture may be unnecessary for predicting a collision, while an initially hidden material property may be essential. A representation can therefore be sufficient for one family of predictions and inadequate for another.
We should define this operating domain before testing, then include deliberate departures from it. A model should be evaluated both on predictions within its domain and on how it expresses uncertainty when its assumptions no longer hold.
A Minimal Problem Formulation
Let ot denote the observation at time t, at an available action input, and st the maintained state. A generic state update can be written as:
st = Fθ(st−1, ot, at−1, Δt).
The state may contain spatial elements, learned features, explicit memory, and uncertainty over alternative configurations. Elapsed time matters when observations are asynchronous. Unknown interventions must remain unknown influences rather than being silently supplied as action inputs.
The predictor then uses that state, a specified query, and any known future actions to produce a distribution over a future outcome. Queries might concern position, occupancy, contact, or the occurrence of an event within a stated interval.
The ideal is predictive sufficiency: the maintained state should support the same relevant predictions that could be made from the complete available history. This is a target for evaluation, not a property established by the notation. No representation can uniquely recover information the sensors never reveal.
This predictive view of state follows the formulation developed by Littman, Sutton, and Singh [1], who represent a system through predictions of action-conditioned future observations.
Compactness is similarly a tradeoff. We should compare prediction quality across memory, computation, and latency budgets. End-to-end cost includes perception, runtime state, required pretrained components, and any output decoding needed by the task. Training and adaptation costs should be reported separately.
A smaller internal representation is valuable when it retains the necessary predictive information and reduces the total cost of achieving the required capability.
Persistent Points and Uncertain Correspondence
Observation points and model points
Observation points are temporary sensor measurements. Model points are elements of a maintained estimate. They may carry a position, motion features, appearance information, confidence, observation history, and relationships with neighboring elements.
A model point need not correspond to a particular material particle or sensor sample. Its identifier records a continuity hypothesis. Where individual identity is ambiguous, continuity may be better represented at the level of a surface region, neighborhood, or object.
Points introduce a geometric assumption: location and spatial relationships are useful for the prediction targets. They do not remove all assumptions, guarantee physical understanding, or make other representations unnecessary. Objects, meshes, fields, and latent states remain important alternatives and possible complements.
Graph Network-based Simulators learn dynamics over particle graphs [7], while 3D-IntPhys couples a visual frontend to point-based dynamics prediction [8]. Kipf et al. [5] demonstrate object segmentation and tracking from video using optical flow and initial object-location cues. Persistent Dynamic 3D Gaussians [9] provide a related precedent for tracking moving scene elements.
Prediction proposes continuity; perception corrects it
We propose studying an update cycle with three coupled functions:
- Predict. Estimate how the current state may evolve before the next observation arrives.
- Associate. Evaluate how new measurements relate to the predicted structure, including multiple possible associations and unmatched evidence.
- Correct. Revise state, uncertainty, and continuity hypotheses using the new evidence.
Correspondence should not require a one-to-one mapping between scans. Several observations may support one region of the model. A model element may remain unobserved. New measurements may reveal new structure or challenge an earlier interpretation.
Prediction and correspondence can help one another, but they can also fail together. A mistaken prediction can bias association; a mistaken association can reinforce the prediction. Unexpected motion, similar-looking objects, and conflicting evidence must therefore be part of the evaluation.
Persistence includes revision and forgetting
A missing observation can mean occlusion, sensor failure, departure from the monitored region, or an incorrect earlier estimate. Existence, visibility, and confidence should be treated separately.
Retaining every element indefinitely creates false structure. Deleting every unobserved element destroys persistence. The model must learn or implement a defensible policy for maintaining, revising, and retiring hypotheses.
Model elements may also split or merge when spatial resolution changes. Such operations should preserve the quantities needed for prediction. An internal change of indexing should not be scored as loss of physical identity when the task requires only region-level continuity.
Visualization as a readout
Images, depth maps, meshes, and object descriptions can help inspect and use the model. Rendering losses may also support learning. But visual quality alone does not establish continuity or predictive accuracy.
The evaluation should expose cases in which a convincing reconstruction assigns the wrong identity or predicts the wrong event, as well as cases in which an abstract representation predicts accurately without reconstructing fine appearance.
As demonstrated by Assran et al. [4] with V-JEPA 2, predictive features learned from video can support an action-conditioned latent model for robotic planning after additional training on robot trajectories.
Learning at Different Timescales
Prediction error may arise from measurement noise, uncertain association, mistaken state, inadequate dynamics, or a changing environment. One residual does not necessarily identify its cause.
We should therefore distinguish immediate state correction, explicit memory updates, environment-specific adaptation, and changes to shared model parameters. Updating an object's estimated position need not require retraining the model; discovering a persistent change in its behavior may justify a different update.
This separation follows the distinction among state, memory, and parameters, and the problem of attributing prediction error, discussed by Leroy & Leroy (2026) [11].
This framework does not prescribe an alternative to transformers or backpropagation. Those methods remain candidates whose value should be tested under the stated requirements.
Candidate learning signals include subsequent observations, agreement across views, temporal consistency, and outcomes of known actions. Claims of self-supervised learning must still identify any calibration inputs, pretrained encoders, segmentation masks, labels, and physical priors. Evaluation-only ground truth must remain separate from training inputs.
Uncertainty should be measured for specific prediction targets. Action-conditioned forecasts should likewise be distinguished from causal identification: claims about causal effects require controlled interventions and appropriate assumptions.
Research Hypotheses and Failure Tests
The following hypotheses make the program testable. Each includes a comparison that could weaken or reject it.
H1. Persistence improves prediction through missing observations
Compare a model that retains state across occlusion with the same model whose state is reset or truncated. Hold training and available observations constant. The hypothesis is weakened if persistence produces no reproducible improvement in post-occlusion estimates or future-query accuracy across the predefined conditions.
H2. Predictive correspondence improves continuity
Compare association informed by predicted motion with a comparable association mechanism lacking that information. Measure both familiar motion and unexpected interventions. Benefits that disappear or reverse under disturbances would reveal an important limitation.
H3. Attributed points offer a useful accuracy–resource tradeoff
Compare points with recurrent latent states, object-based models, and suitable geometric alternatives under matched inputs and documented budgets. If alternative representations consistently achieve better predictions at lower cost, the point-based hypothesis is weakened for those tasks.
H4. Domain specialization reduces required resources
Compare general and specialized systems on the same held-out in-domain queries, including adaptation costs and pretraining dependencies. A gain that disappears after accounting for sensing, data access, and end-to-end computation does not support the hypothesis.
H5. Separating correction from adaptation improves recovery
Compare state-only updates, parameter adaptation, and their combination after specified environmental changes. Measure sustained recovery and regression on previously learned conditions. Faster adjustment is not sufficient if it causes persistent loss elsewhere.
These hypotheses concern distinct capabilities. Evidence for one should not be presented as proof of all five, or as a demonstration of general physical intelligence.
A First Experimental Environment
A useful starting point is a calibrated tabletop workspace with rigid objects, a movable occluder, and repeatable actions. The model receives designated camera or depth observations and, in action-conditioned tests, the commanded intervention. Separate sensing or instrumentation can provide evaluation references where feasible, with its own uncertainty documented.
Sequences should vary motion, viewpoint, sampling density, sensor dropout, and occlusion duration. Similar objects and unexpected disturbances should test whether the model preserves alternatives or becomes confidently wrong. Deformation and changing topology can follow as separate extensions.
PLATO offers a precedent for structured tests of physical expectations, including persistence through occlusion [6]. Its use of supplied segmentation and tracking also makes clear why the inputs available to each method must be documented.
The first experiment should focus on H1 and H2. Keep the representation and dynamics family fixed, then vary persistence and motion-informed association. This establishes whether those mechanisms help before attributing a benefit to points themselves.
Separate estimation, prediction, and recovery
- Online estimation: use observations available through the current timestamp to estimate state.
- Open-loop prediction: forecast from a declared cutoff without access to later observations.
- Recovery: resume observations after a gap and measure how the model revises its state.
Future observations must not enter association, camera-pose refinement, or reconstruction before a forecast is scored. Train and test sets should separate trajectories and acquisition sessions, with additional tests for new object instances, intervention patterns, and longer occlusions.
Compare with competent alternatives
Initial baselines should include geometric tracking with simple motion extrapolation, persistent state without learned dynamics, and a recurrent learned predictor. Object-based and alternative spatial representations become essential when testing H3. Analytical dynamics may be particularly competitive in the rigid-object setting.
Hafner et al. [2] demonstrate learning control behaviors through imagined trajectories in Dreamer, while Wu et al. [3] demonstrate world-model learning directly on physical robots in DayDreamer. DROID-SLAM jointly estimates camera pose and depth [10]; it is a geometric estimation reference rather than a dynamics predictor.
Shared perception components should be used where practical. Where methods require different inputs or frontends, those differences and their costs must be explicit. Simple baselines that perform well are informative outcomes.
Measure continuity and prediction directly
Report state-estimation error, prediction error across horizons, recovery after occlusion, false persistence of absent structure, and identity continuity where identity is well defined. Where point identity lacks a meaningful ground truth, use region-level or object-level continuity. Unordered geometric distance alone cannot measure identity.
For probabilistic outputs, measure appropriate proper scoring rules and calibration. Use repeated trials and uncertainty intervals. Report model storage, peak runtime memory, update latency including tail latency, training cost, and adaptation effort on stated hardware.
The intended result is a set of accuracy–resource comparisons and failure analyses. Numerical targets and decision criteria should be fixed in the experimental protocol before the corresponding tests are run.
Scientific Context
The framework draws on established research. Littman, Sutton, and Singh [1] formalize state in terms of action-conditioned future observations. Hafner et al. [2] demonstrate learned world models for control in Dreamer, and Wu et al. [3] extend this approach to physical robot learning in DayDreamer. Assran et al. [4] demonstrate self-supervised video representations and action-conditioned prediction for robotic planning with V-JEPA 2.
The object-centric video model developed by Kipf et al. [5] motivates representing and tracking entities over time, while PLATO [6] provides structured tests of physical expectations. Graph Network-based Simulators and 3D-IntPhys establish important precedents for point-based dynamics [7], [8]. Dynamic 3D Gaussians demonstrates persistent moving scene elements for reconstruction and tracking [9], while systems such as DROID-SLAM illustrate the importance of joint geometric estimation [10].
The contribution sought by this research program is a tested combination of requirements and mechanisms for bounded physical prediction. It must be distinguished experimentally from these foundations. A new name, a point representation, or a prediction–correction loop alone does not establish novelty.
Open Questions and Limits
Bounding an environment may improve efficiency while reducing transfer. A collection of specialized models could require more total engineering and maintenance than a general system. Adaptation between environments therefore matters alongside performance within one environment.
Spatial points may also be insufficient for hidden stresses, temperature, forces, material properties, or human intent. Additional variables, sensors, and physical priors may be required. Increasing point density cannot resolve information that is unobservable.
Persistent state can become stale, and uncertainty calibrated in one domain may fail outside it. These limitations should guide the experiments rather than being deferred until deployment.
We have not yet established that the proposed representation satisfies predictive sufficiency, that predictive correspondence improves robustness, or that specialization lowers total cost. The framework is designed to make those claims answerable.
From Research Direction to Evidence
The first three notes established a direction: learn predictive representations of bounded physical reality, and investigate points as a substrate for persistent state.
The next step is to determine what survives controlled comparison.
Our immediate research question is whether maintaining predictive state and using it to interpret new measurements improves continuity through occlusion and resampling. A bounded experiment can make that question concrete, reveal its failure modes, and establish which assumptions deserve further investment.
A Reality Model should be judged by the physical questions it can answer, the uncertainty it preserves, and the resources required to do so.
References
- Michael L. Littman, Richard S. Sutton, and Satinder Singh. “Predictive Representations of State.” Advances in Neural Information Processing Systems 14, 2001. Read source
- Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. “Mastering diverse control tasks through world models.” Nature 640, 647–653, 2025. Read source
- Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. “DayDreamer: World Models for Physical Robot Learning.” PMLR 205, 2226–2240, 2023. Read source
- Mido Assran et al. “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning.” arXiv:2506.09985, 2025. Read source
- Thomas Kipf et al. “Conditional Object-Centric Learning from Video.” arXiv:2111.12594, 2021. Read source
- Luis S. Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. “Intuitive physics learning in a deep-learning model inspired by developmental psychology.” Nature Human Behaviour 6, 1257–1267, 2022. Read source
- Alvaro Sanchez-Gonzalez et al. “Learning to Simulate Complex Physics with Graph Networks.” PMLR 119, 2020. Read source
- Haotian Xue et al. “3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes.” Advances in Neural Information Processing Systems 36, 2023. Read source
- Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. “Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis.” arXiv:2308.09713, 2023. Read source
- Zachary Teed and Jia Deng. “DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras.” arXiv:2108.10869, 2021. Read source
- Luc H. Leroy and Maximilien Leroy. “Position: World Models of Physical Sites Must Discover Their Own State.” World Models in Physical AI Workshop, NeurIPS 2026, 2026. Workshop poster. Read source