Research update · completed experiment plus untested directions. This follows MORPH: Model of Object Representation and Predictive Homomorphisms. The measurements below come from the completed O1 campaign. The populations, moving sensors, and recurrent hierarchy discussed afterward have not been implemented or evaluated in that campaign.

MORPH started with an ambitious question: can an agent build persistent object models whose representations change predictably when it acts? The first experiments reduced that question to something much smaller: encode a simple image, rotate part of its embedding, and decode the expected rotated view.

The latest result is encouraging and incomplete. We now have a model that reconstructs geometric shapes and predicts their rotated views accurately on new instances of familiar shape families. But the encoder does not produce sufficiently consistent pose coordinates across those views. All five final seeds passed 18 of 19 registered checks. The same remaining check failed in every seed.

That is a negative result for the full experiment. It is also a useful separation of two things I had initially hoped would improve together: good image prediction and a dependable internal coordinate system.

What O1 actually tested

O1 was a bounded architecture and optimization search, not the complete MORPH system. It used synthetic white geometric silhouettes on black backgrounds, rendered with antialiasing to 32 × 32 pixels. The familiar families included ellipses, polygons, concave shapes, holed shapes, and unions of primitives.

The learner received an initial image and known rotation increments in radians. It predicted future views without receiving the intervening target images as inputs. Training included reconstruction and prediction at horizons 1, 2, 4, and 8, plus feature-invariance and pose-equivariance losses. There was no object memory, ownership inference, recurrence, or learned motor calibration.

The encoder split its output into two parts:

• An invariant feature code, intended to retain information that survives rotation.
• An equivariant pose code, intended to change in a prescribed way under rotation.

Those names describe training objectives. They do not prove that one branch contains pure identity and the other contains pure pose.

The selected recipe used a fully connected encoder with separate feature and pose heads, followed by a convolutional decoder. Its latent contained 32 invariant values and 32 pose values. The pose values formed 16 disjoint coordinate pairs, with four pairs assigned to each harmonic from 1 through 4.

Each pair rotates by its harmonic times the supplied angle. These are rotations of activation coordinates in the embedding, not rotations of the network’s learned weights. The group action is fixed; the encoder and decoder must learn to work with it.

The search evaluated 24 configurations across three encoder families: convolutional encoders with separate dense heads, convolutional encoders with a shared projection and output split, and fully connected encoders with separate heads. Two finalists were evaluated across five seeds, with one recipe selected using validation data before the final audit. This does not establish that dense encoders are generally better than convolutional encoders.

The complete campaign consumed 260,000 optimizer updates and 20.83 accounted hours, within its authorized 55-hour cap. Source, data, and checkpoint hashes were frozen, and 4,960 saved prediction tickets were checked for prediction-before-score chronology. That establishes an auditable run; it does not turn an unsuccessful criterion into a successful one.

The measured result

The primary frozen test contained 512 new objects drawn from familiar shape families.

MSE is mean squared pixel error on intensities from zero to one. The foreground metric uses a renderer-defined support mask; prediction comparisons use the union of source and target foreground. These masks are evaluator information, not learner inputs. Scores are aggregated with family and action-schedule balancing. The confidence interval uses paired seed/family-object bootstrap resampling. An MSE of 0.11 is not an 11% classification error.

Saved model outputs from the final report, not a scripted illustration. The panel shows seed 0 on the primary test: the first object from each family and the worst object by eight-step foreground error. Rows show inputs, reconstructions, eight-step targets, and predictions. The final column makes a failure visible: the model rounds away the shape’s protrusions.

The distinction between kinds of unseen data matters. New instances from familiar families performed well. Entirely withheld families performed substantially worse:

These diagnostic distributions were excluded from recipe selection and primary acceptance. Their errors limit the claim: O1 learned useful prediction within a distribution, not broad geometric generalization.

How can the pixels be right while the coordinates are wrong?

Let E be the encoder, D the decoder, u the feature code, and p the pose code. Let R rotate the actual image by angle theta, and let rho apply the prescribed rotation to the pose code. The desired relationship is:

In words: rotating the image and then encoding it should produce approximately the same pose code as encoding it first and rotating that code. The reported pose ratio normalizes the root-mean-square discrepancy by within-object pose-code variation across views, on objects whose rendered views change. It is not an angular error or a percentage of incorrect poses.

The decoded prediction instead tests:

These are different requirements. A decoder can map several different latent codes to similar images. Consequently, good reconstruction and good prediction do not guarantee that the two latent routes agree. Our aggregate image scores also do not prove that the routes decode identically on every example.

Two explanations remain open. The discrepancy might lie mostly in redundant directions that barely affect decoded pixels. Alternatively, the decoder might compensate for a meaningful mismatch between the encoder and the prescribed transformation. Both could occur together. We have not yet measured which dominates.

Replacing either the feature or pose branch damaged predictions, so both branches are being used. But replacement can create codes outside the training distribution. It is not proof of clean disentanglement.

There is another distinction here: our rotation matrices already satisfy the group composition law by construction. Applying two rotations composes correctly. What failed was the learned encoder’s alignment with those rotations. A later memory or planning module consuming the latent directly may care about that discrepancy even if the image decoder can tolerate it.

What the homomorphism autoencoder paper establishes

Keurti et al.’s Homomorphism Autoencoder learns action representations jointly with observation representations. Its toroidal translation experiment uses discrete cyclic x/y shifts, corresponding to a subgroup of SO(2) × SO(2). It reports separate rotation blocks for those axes, a further rotation factor in another setting, identity separation among three shapes, and 128-step prediction after training on two-step trajectories. It also studies SO(3) rotations of a 3D bunny.

Those are stronger demonstrations of learned transformation structure than O1 currently provides. But the paper does not report our normalized pose-consistency gate, and its BCE reconstruction measurements cannot be numerically ranked against our foreground MSE. A matched reproduction is needed to compare performance directly.

The torus also clarifies a possible next task. Horizontal and vertical wraparound movement are two independent circular coordinates. Our current harmonics all respond to one rotation angle; adding more harmonics does not create an independent movement axis.

Could populations tolerate imperfect consistency?

One possibility is to represent several plausible poses instead of forcing one estimate. This becomes especially relevant for a tiny moving sensor: a patch showing a straight edge may correspond to several places on an object.

A population could retain candidate locations and weights. Movement transports those candidates; each predicts a future patch; new pixels change their weights. Agreement between sensors would mean that their hypotheses support compatible object states after accounting for relative sensor positions. Equal confidence scores alone would not mean agreement.

This is an untested hypothesis for MORPH. It might help if a single-state model fails because observations are genuinely ambiguous. It would not automatically remove redundant latent coordinates or repair a systematically wrong action model. We have not established ambiguity as the cause of O1’s failed gate.

A single Gaussian describes uncertainty around one region. Multiple distinct explanations may require multiple peaks or an explicit weighted population, especially when pose wraps around. The useful outcome would be accurate predictions and appropriate uncertainty reduction as new evidence arrives. A distribution that simply widens enough to accommodate everything would not be a success.

O1’s failed gate should remain failed. A population experiment would need its own registered question and criteria, including whether the correct alternative survives an ambiguous sequence and gains support when a distinguishing landmark appears.

From moving patches to a shared recurrent model

Another direction is to move a small sensor over a hidden image and accumulate its glimpses. Imagine seeing a duck through a pinhole. A recurrent encoder could retain observations and displacement information, while a decoder reconstructs the larger region.

Two capabilities must be measured separately. Remembering and placing pixels already observed is spatial memory; a canvas that pastes patches at known coordinates is an essential baseline. Predicting the unseen body from a glimpse of the beak is learned completion and should remain uncertain. A plausible invented duck is not the same as recovering the hidden image.

Multiple local recurrent encoders could then feed a parent recurrent autoencoder. The parent would receive child embeddings and their relative positions, compress them into a larger-scale state, and reconstruct or predict the child observations. Joint training would let parent prediction errors encourage child representations that are easier to combine, while local reconstruction would protect detail.

The following is a proposed computation, not an implemented model:

flowchart TD
    O["1. New patches and displacement"] --> C["2. Local recurrent encoders"]
    L["Previous local memories"] --> C
    C --> P["3. Parent recurrent encoder with relative positions"]
    H["Previous parent memory"] --> P
    P --> D["4. Predict future child patches"]
    D --> S["5. Freeze predictions, then score newly revealed pixels"]
    C --> L
    P --> H

Reconstructing embeddings that the parent just received tests compression. Predicting withheld or future patches tests whether it learned useful relationships. Pixel-level checks remain necessary: an embedding-only objective could improve because the children discard information.

Shared weights alone do not establish a shared object model or coordinate frame. Two paths ending at the same location should support compatible predictions, but their memories can legitimately differ if one path gathered more evidence. The parent’s identity hypothesis can help interpret local patches; its persistence is not independent confirmation that the identity is correct.

Likewise, a parent prediction sent back to a child is context, not a new observation. Recurrent feedback must not count the same evidence repeatedly. That discipline from the original MORPH proposal becomes more important as the hierarchy grows.

What should earn the next layer?

Stacking this pattern across spatial and temporal scales is an appealing direction. Each level could retain features, track transformations, accumulate uncertain evidence, and predict the level below. But longer timescales need explicit objectives, and actions involving contact, deformation, or irreversible change will need dynamics beyond group rotations.

The immediate research choices are smaller. A frozen-model diagnostic can determine whether O1’s actual latent discrepancies affect decoded geometry. A moving-patch experiment can test whether recurrence retains observed detail and improves future prediction. A population comparison can isolate the value of preserving ambiguity. A two-level comparison can test whether parent feedback helps beyond a flat recurrent model with comparable information and capacity.

These should be separate, registered tests rather than one large system whose failures are impossible to attribute. None has started as part of this update.

The useful advance is concrete: structured prediction now works reasonably well for familiar geometric families. The unresolved issue is equally concrete: accurate output has not yet given us the common internal coordinates MORPH would like to use. The next mechanisms need to earn their place by explaining or overcoming that limitation, with new observations providing the evidence.

Experimental follow-up: MORPH, Part 2: Good Predictions, Inconsistent Coordinates reports the completed O1 campaign and separates its measured findings from proposed recurrent and population-based extensions.

Working research note · architecture under active review. MORPH — Model of Object Representation and Predictive Homomorphisms — is an exploratory architecture, not an experimentally validated model. It grew out of a narrower question about object identity: how can an embodied learner continuously create new identities, learn from them aggressively, and still prevent its own recurrent hypotheses from returning as counterfeit evidence?

The proposal has now sharpened into three coupled problems:

1. Persistent identity: create a new object memory immediately, without adding a new classifier output or retraining the whole network.
2. Structured sensorimotor dynamics: use one shared equivariant encoder so actions move a representation along lawful, compositional transformation trajectories rather than through an arbitrary next-state predictor.
3. Controlled plasticity: when gradient descent improves the encoder, migrate old memories and learned dynamics into the new representational coordinates instead of silently invalidating them.

The central proposal is:

Persistent identities should be fast nonparametric memories interpreted by a shared equivariant predictive substrate. Physical action should induce structured motion through that substrate. When the substrate itself changes, old knowledge should migrate through an explicit compatibility transform and the update should remain in escrow until both old identities and old dynamics still work. A hypothesis may influence what is predicted or tested next, but evidence used to create that hypothesis cannot also validate it.

This separates three jobs:

1. Fast memory creates and updates individual object hypotheses without gradient descent.
2. Fast equivariant dynamics map actions into predictable transformations of patch representations.
3. Slow shared plasticity updates the encoder, decoder, identity–pose factorizer, group-action model, retrieval metric, vigilance machinery, grouping machinery, and later abstraction layers under anti-forgetting constraints.

The decoder is part of the predictive substrate: it maps a predicted future embedding back into real sensor space so the prediction can be compared with what the robot actually sees or feels. MORPH’s relationship to Homomorphism AutoEncoders is discussed in section 15.

MORPH therefore tries to preserve several attractive properties at once:

• one-shot or few-shot creation of a previously unknown identity;
• broad gradient updates that improve general machinery across many identities;
• action-conditioned prediction that reuses geometric structure instead of relearning every transition independently;
• explicit migration of old memories when the learned representation changes;
• temporal evidence accounting that prevents recurrent hypotheses from validating themselves.

Two different transformation structures appear and should not be conflated.

The physical transformation group describes lawful action and motion, for example \(SE(3)\) for rigid 3-D rotation and translation. Its action on the learned representation predicts how features should move when the camera, robot, or object moves.

The representation-migration group describes a change of coordinates between encoder versions. A useful first approximation is an orthogonal transform in feature space. It need not be the same group as the physical motion model.

The first toy architecture still begins with fixed sensor patches, sparse approximate-nearest-neighbor retrieval, ART-like vigilance, active experiments, and an unknown number of persistent identities. The longer-term hypothesis is that the same proposal/challenge/resonance and controlled-migration rules can recurse into parts, objects, relations, scenes, affordances, and still higher abstractions.

1. The problem: open-ended identity without local-only learning

A conventional classifier usually ends in a fixed output vector:

That assumes the identity vocabulary is known when the model is built.

An embodied learner has a different problem. It may encounter object \(K+1\) tomorrow, \(K+2\) next week, and millions more over its lifetime. Rebuilding and retraining a softmax head whenever a new identity appears is the wrong abstraction.

A more natural starting point is a shared encoder:

with persistent identities represented separately in memory:

A new object can then be created by allocating a new memory artifact rather than a new output neuron.

But this creates a second problem.

If every persistent identity owns its own private predictor, matcher, or transition model, then experience with one object updates only that object. Learning becomes narrow. A new observation does not improve the machinery used by thousands of related and unrelated things.

That loses one of the major strengths of deep learning:

A single prediction error can update a large shared parameter set simultaneously.

MORPH therefore deliberately makes persistent identities mostly memory, while keeping most trainable machinery shared.

2. Architecture at a glance

MORPH uses a shared representation with an explicit identity–pose factorization. Here, identity means a persistent individual thing; pose latent means the current view or transformation state relative to learned anchors. It need not be a calibrated position and orientation in meters and radians.

flowchart TD
    X["1. Sensor patch histories"] --> E["2. Shared equivariant encoder"]
    E --> Z["3. Joint latent field"]
    Z --> F["4. Joint grouping and identity–pose inference"]
    F --> I["Identity descriptor and candidate memory"]
    F --> P["Pose latent and ownership uncertainty"]
    I --> ANN["5. ANN candidates plus UNKNOWN"]
    ANN --> H["6. Competing instance hypotheses"]
    P --> H
    H --> D["7. Transform latent and decode predicted observation"]
    D --> T["8. Freeze prediction, act, then score new evidence"]
    T --> M["9. Update memory without gradients"]
    M --> ANN
    T --> U["10. Train candidate shared model"]
    U --> R["Review old identities, dynamics, and migration"]
    R --> E

Stages 1–3 encode the observation before assigning object ownership. Stages 4–6 keep competing identity, grouping, and pose explanations. Stages 7–9 test those explanations against future observations. Stage 10 improves the shared machinery only after scoring, with a separate review before deployment.

For patch \(i\) at time \(t\), the encoder produces:

The input is a history of length \(L+1\). Its population code over patches is:

A factorizer \(F_\eta\), using that field and provisional ownership weights \(w_{ij,t}\), proposes:

Here \(u_j\) is an approximately invariant identity descriptor and \(p_j\) is the pose latent for candidate instance \(j\). Ownership and factorization are refined together; neither is assumed known initially. A local retrieval head can still produce coarse patch keys \(k_{i,t}=I_\phi(z_{i,t})\), but those are candidate cues rather than complete invariant object identities.

A composition function \(C_\eta\) combines identity and pose into an object latent. For group-modeled motion:

The group element is \(g_{j,t}=\exp(\xi_{j,t}\Delta t)\), with generator \(\xi_{j,t}=f_\psi(a_t,c_t,H_{j,t})\). Context \(c_t\) includes available motion and sensor information; the hypothesis \(H_{j,t}\) specifies candidate ownership and relative state. A motor command does not directly specify every object’s motion.

The composition is constrained to respect the group:

An additional shared dynamics residual \(r_\omega\) predicts effects outside that model:

A shared decoder \(D_\delta\) maps the predicted latent field \(\hat Z_{t+1}\), predicted ownership/visibility \(\hat W_{t+1}\), and sensor context back to real observation space:

For vision, this means image intensities or a distribution over pixels; touch and other sensors need corresponding outputs. This is the sensor-space prediction, not another identity key. A spatial decoder must handle features moving between patches and visibility changes; the same fixed patch does not necessarily observe the same surface at the next time step. An object-centered decoder alone needs a projection/compositing step to predict the camera image.

Persistent memory stores identity descriptors, view/orbit anchors, transition traces, encoder versions, relations, and uncertainty. It supplies data interpreted by shared functions, rather than a private deep network for each object.

Encoder evolution has a separate compatibility transform:

The transform \(Q\) carries old coordinates forward; \(u_{\mathrm{new}}\) denotes candidate new representational capacity. For a first migration model, \(Q=\exp(A)\) with \(A^\top=-A\). New capacity and the mismatch left after fitting \(Q\) are distinct from the dynamics residual.

3. Fixed sensor patches and sparse identity retrieval

The first toy version divides the visual field into fixed patches. Each patch encodes a short history into a compact population code \(z_{i,t}\), then proposes sparse local matches:

For example, patch 17 might retrieve objects 42, 781, 19, and 103, plus UNKNOWN. The index searches stored local prototypes and maps each hit back to its parent identity. Different patches can retrieve different sets; a global list of every identity is unnecessary.

From a variant patch to an invariant identity

An equivariant encoder does not automatically deliver a unique object ID. A small patch showing a uniform blue surface may belong to a stapler, a mug, or a wall. No transformation matrix can recover identity information that the observation does not contain.

The desired relationships are:

The first retains predictable transformation information. The second reads out information stable under the modeled transformations. A readout can learn that stability from reliably associated views without receiving the absolute pose at inference time. It can also marginalize over candidate transformations or compare multiple stored views. Invariance alone is insufficient: a constant output is invariant too, so reconstruction and discrimination constraints must preserve useful information.

For realistic 3-D objects, rotation changes visibility and can move evidence across patch boundaries. A fixed crop is generally not closed under the object’s rotation group. Consequently, the exact equations apply most cleanly to an object-centered latent or the whole latent field with correspondence, while local patch keys are only approximate, partial evidence.

MORPH therefore proposes a bounded joint inference loop:

1. Retrieve locally: appearance, short-term continuity, and stored view anchors generate candidate identities without requiring a known pose.
2. Propose ownership: neighboring patches and coherent motion support several possible instance groupings.
3. Infer pose per candidate: compare the grouped evidence with candidate anchors, retaining alternative poses or axes when ambiguous.
4. Predict and challenge: test each identity–ownership–pose explanation through future views or actions; only afterward update persistent memory.

This resolves the execution-order circularity by keeping alternatives, rather than assuming identity must be settled before pose or grouping can be estimated. It does not guarantee a correct or efficient solution. The cost is limited by candidate counts and recurrence budgets, and ambiguous observations may remain unknown.

A rotation is also defined relative to a frame and origin. Rotating about an unknown object center is not necessarily a rotation about the camera origin. A rigid transform must include the corresponding translation, and relative camera/object motion must be inferred or measured. The pose latent can encode those relationships without initially exposing metric coordinates. One shared motion hypothesis must coordinate the population code; arbitrary independent rotations of each feature vector would not enforce object coherence.

Identity as an orbit, with pose along that orbit

A persistent object can store multiple anchors:

In the ideal group model, one consistent object’s anchors lie on its transformation orbit:

Identity is the equivalence class of lawful views; pose specifies a location along that orbit. An orbit is a manifold under suitable regularity assumptions, not generally a ball with simple minimum and maximum embedding coordinates. A practical memory covers only sampled reachable views; noise and model error make the accepted region a tolerance around that coverage.

A candidate can be evaluated by orbit distance:

candidates = ANN(local_key, K)
for identity in candidates:
    propose compatible ownership and pose states
    compare observed features with transformed memory anchors
keep bounded alternatives and UNKNOWN
freeze their action-conditioned predictions before observing the outcome

ANN retrieves a shortlist of anchors or invariant descriptors; it does not solve general manifold membership. The minimization above is a subsequent constrained alignment or learned matching step. Visually indistinguishable instances may need trajectory history or interaction even when the orbit model is accurate.

Symmetric objects also have several equally valid poses. MORPH should maintain pose uncertainty or symmetry classes rather than enforce a globally unique canonical orientation. Learning useful orbit structure, ownership, and pose factorization together is a central research question.

4. ART-like vigilance is a challenge, not a verdict

Nearest neighbor is only candidate generation.

A retrieved candidate must still survive a vigilance test:

If

candidate \(M_j\) is reset for that patch and another candidate may be tested.

If no stored identity passes vigilance, the legal outcome is:

This is inspired by Adaptive Resonance Theory (ART), where bottom-up evidence activates a candidate category, the category supplies a top-down expectation, and mismatch can reset the candidate and continue search. High vigilance produces finer categories; lower vigilance permits broader categories.

MORPH generalizes the interpretation:

A candidate identity is allowed to propose an expectation. The expectation itself is not evidence. It must survive comparison with evidence outside the proposal.

That distinction becomes crucial once recurrence is introduced.

5. Temporal escrow: predict first, validate later

The central architectural rule is simple:

Suppose evidence through time \(t\) proposes object hypothesis \(H\).

That evidence is sufficient to create the hypothesis:

It is not allowed to confirm the hypothesis.

Before the next observation exists, the system commits a prediction:

The prediction and relevant model state are placed in a conceptual escrow record:

prediction ticket
-----------------
time: t
candidate identity: H
available evidence: E≤t
chosen action: a_t
predicted future evidence: Ê_{t+1}
model version: θ_t

Then the robot acts, or time passes.

Only afterward does reality produce:

Now the hypothesis can earn evidence:

The important causal fact is that \(E_{t+1}\) could not have been used to construct the prediction that is now being tested against it.

This prevents a dangerous loop:

hypothesis wins
    ↓
train model on the same evidence
    ↓
model fits that evidence better
    ↓
better fit is treated as confirmation
    ↓
hypothesis wins harder

MORPH requires the opposite ordering:

hypothesis proposed
    ↓
prediction frozen
    ↓
new evidence arrives
    ↓
prediction scored
    ↓
only now may learning occur

The escrow rule is not itself a neural learning algorithm. It is a causal accounting rule for validation.

6. Recurrent belief may circulate; evidence may not multiply

A recurrent network can pass a hypothesis through many nodes:

A → B → C → D → A

That creates a classic danger in loopy inference: evidence originating at A can eventually return to A through another path and appear independent.

MORPH does not initially try to solve arbitrary provenance bookkeeping.

Instead it imposes an evidence-conservation rule:

Internal messages may redistribute, refine, suppress, or select hypotheses. They cannot manufacture new likelihood. New confidence must ultimately be paid for by a new designated evidence event.

A returned message may cause A to test a different prediction. It cannot increase global confidence merely because the hypothesis completed another circuit.

The recurrent network is free to ask:

• which candidate should be tested?
• which stored identity is most relevant?
• which neighbor should be queried?
• which action would distinguish two hypotheses?
• what should be predicted next?

But confirmation requires a new observation, a held-out observation, or another explicitly independent evidence channel.

The toy V1 uses future sensory evidence as an auditable causal boundary: it was unavailable when the prediction was frozen. Nearby frames can still be statistically correlated, so temporal separation alone does not guarantee independent evidence.

7. Active inference turns ambiguity into an experiment

A hypothesis is a structured claim:

This evidence belongs to identity \(I_j\), this is its current pose latent, and given action or transition \(a_t\), its latent representation and resulting observations should change in this particular way.

One possible record is:

where \(w_{j,t}\) describes candidate patch ownership, \(p_{j,t}\) is the relative pose latent, \(\Sigma_{j,t}\) records uncertainty, and \(\theta_t\) identifies the shared model version used for prediction. The identity descriptor comes from the candidate memory. A hypothesis refers to a current instance and its evidence trail, rather than only naming an object or a category.

The shared transition model uses this record and the action to produce a distribution over future latents and visible observations. Two hypotheses may name different identities, different poses of the same identity, or different ownership assignments. Their predictions can disagree even when their present appearance scores are similar.

Suppose two object hypotheses currently fit:

They predict similar present evidence, but different consequences under action \(a\):

The system can choose an action maximizing disagreement:

or more generally maximizing expected information gain.

Conceptually:

H1: "move right; the handle should enter these patches"
H2: "move right; no handle should appear"

               ↓

          move right

               ↓

          observe world

               ↓

     resonance / reset

This gives recurrent inference an escape hatch:

When internal messages become circular or ambiguous, ask the world a question.

The action is not merely motor output. It is an experiment.

8. Fast identity memory and shared equivariant learning

Temporal escrow would be too slow if every object learned only through its own private weights.

MORPH instead has three complementary update paths.

Fast path: persistent identity memory

A newly observed identity can be allocated immediately:

Allocation and memory updates use no gradient descent. They store observations, update statistics, or revise associations. Gradient descent belongs to the separate shared-learning path.

A provisional identity may be promoted, revised, merged, or deleted as future escrowed evidence accumulates.

Fast path: action-conditioned group dynamics

The same encoder used for identity should also support structured prediction.

A physical action is mapped into a local generator:

The exponential map produces a finite group element:

The representation then moves under a learned group representation:

For rigid 3-D motion, a natural candidate is the special Euclidean group \(\,SE(3)\,\) with Lie algebra \(\,\mathfrak{se}(3)\,\). A twist combines infinitesimal translation and rotation. Unit dual quaternions are one compact way to represent the corresponding finite rigid transforms.

The important architectural claim is not that every sensory transition is rigid-body motion. It is that the system should explain as much transition structure as possible through reusable, compositional transformations before spending unconstrained model capacity on residual dynamics.

A more realistic predictor is therefore:

Slow path: shared gradient learning

Once new evidence has been scored, the same transition can update many shared functions simultaneously.

A toy loss can now include identity, prediction, equivariance, migration, and grouping terms:

The decoder provides two complementary losses. Reconstruction checks that the current representation retains enough information to reproduce the current observation. Future prediction checks the action-conditioned representation against the subsequent real observation:

Here \(\ell\) is an observation-space error or negative log-likelihood, for example image mean-squared error with a suitable visibility treatment. A probabilistic decoder can express uncertainty about occluded or unseen surfaces; it should not avoid errors by declaring every difficult pixel invisible. Reconstruction trains representation quality but cannot validate an identity using its own proposal data. The future prediction is frozen and scored before those future observations enter training.

joint_latent = encoder(observation_history)
identity, pose, ownership = factorize(joint_latent, candidate_memory)
future_pose = action_group(action, context) * pose
future_latent = compose(identity, future_pose) + dynamics_residual
predicted_observation = decoder(future_latent, predicted_visibility)
freeze prediction and model version
act; acquire real future observation
score sensor-space error and matched latent-space error
only then train the candidate shared model

Latent prediction loss adds transformation consistency; it does not replace decoding and comparison in real sensor space. A decoder that ignores the transformed latent would likewise fail the action-conditioned prediction test.

On matched observations where the chosen group model applies, the equivariance loss asks the next observation to agree with the action-conditioned transformation:

These losses can update overlapping shared parameters during candidate training, with the staged residual schedule below controlling which branches receive gradients.

So learning object \(\,M_{1001}\,\) does not mean:

update the circuitry belonging to object 1001.

It means:

use this encounter as another constraint on the general machinery for encoding, retrieving, predicting, transforming, distinguishing, grouping, and composing things.

This is why one encoder can support both identity and dynamics, provided that the factorization and its losses are actually learned. The encoder supplies a joint latent; shared inference extracts an identity descriptor and a pose latent.

flowchart TD
    X["Observation"] --> E["Equivariant encoder"]
    E --> Z["Joint latent"]
    Z --> F["Identity–pose factorization"]
    F --> I["Invariant identity descriptor"]
    F --> P["Pose latent"]
    I --> M["Persistent memory"]
    A["Action and context"] --> G["Group action ρp(exp ξ)"]
    P --> G
    G --> PP["Predicted pose latent"]
    I --> C["Compose identity and predicted pose"]
    PP --> C
    C --> ZP["Predicted future latent"]
    ZP --> O["Decoder: predicted real observation"]
    O --> L["Compare with subsequent sensor observation"]

The group acts on the pose factor while the identity descriptor remains stable. Composition reconstructs the joint latent; the decoder predicts the resulting observation. This is an intended factorization, not a guarantee that an arbitrary equivariant network exposes two clean coordinate blocks.

Training uses reliably associated temporal views, multi-step action prediction, and reconstruction of observations. Same-instance views constrain identity stability, confirmed distinct instances provide separation, and composition constrains the pose factor to retain transformation information. Ownership remains provisional, so uncertain tracks should have reduced weight and be challenged before becoming training associations. A reconstruction or variance-preserving objective is necessary because equivariance loss alone admits collapsed codes. A learned factorization need not be globally identifiable; local charts and symmetry-aware pose uncertainty may be sufficient.

Training the dynamics residual separately

The dynamics residual is a shared prediction head, not necessarily a separate encoder. First train the encoder, factorizer, composition/decoder, and group operator on matched, visible transitions where the chosen group model is appropriate. Do not force clean group equivariance on every occlusion or contact event.

Next freeze that candidate group path and compute the prediction error for additional scored transitions:

The frozen target encoder \(E_{\theta^-}\) fixes the coordinate system during this fit. Stop-gradient means this residual-training phase updates \(\omega\), not the shared encoder or group operators. The penalty limits residual use; correspondence and visibility masks exclude comparisons of unrelated surfaces. Also apply the decoded future-observation loss through the frozen decoder: gradients can pass through its input into the residual head while the decoder’s weights stay fixed. This tests whether a latent correction actually improves sensor-space prediction.

score the frozen prediction against the new observation
fit candidate group path on suitable matched transitions
freeze candidate encoder, factorizer, and group path
target = encoded_future - group_prediction
fit residual_head to target with magnitude/capacity penalties
test group-only and combined predictions on held-out episodes

This staged schedule is one practical way to separate the objectives. Later joint fine-tuning can be tested with a small residual, clean-transition group loss, and explicit gradient controls; unconstrained backpropagation through every branch risks residual takeover. A large residual may also signal bad correspondence or a wrong motion estimate, rather than genuine non-group dynamics.

9. Hard negatives turn local novelty into neighborhood learning

ANN retrieval provides more than computational efficiency.

It also tells the learner which existing memories were most confusing.

Suppose a new object retrieves:

If future evidence confirms that none of them was correct, those are useful hard negatives.

A contrastive loss can use the confirmed identity \(M_+\) against its confusing neighbors:

This gives each new identity a neighborhood learning radius.

The update does not touch only the new artifact. It teaches the shared representation why this experience should separate from the things it most resembled.

10. Replay and migration provide a global learning radius

Shared weights create a familiar problem: catastrophic interference.

A gradient step trained only on the newest object can make the global representation worse for older objects. Replay remains useful because it exposes candidate updates to old experiences and hard negatives.

A simple batch might contain:

25% current experience
25% ANN-confusable prior experiences
25% recent experiences
25% broadly sampled older experiences

The exact sampling policy is an experiment.

But replay is no longer the only anti-forgetting mechanism.

MORPH also asks whether the change between encoder versions can be explained by an explicit migration transform. A candidate update should preserve old knowledge either because replay keeps it stable or because old embeddings can be transported into the new coordinates with low distortion.

That gives four learning radii:

NEW EXPERIENCE
      │
      ├── fast/local
      │     update persistent memory artifact
      │
      ├── sensorimotor
      │     refine shared group dynamics / residual dynamics
      │
      ├── neighborhood
      │     contrast against ANN-confusable memories
      │
      └── global
            backprop current + replay losses
            fit/test representation migration
            commit only if old knowledge survives

This resembles the computational motivation behind Complementary Learning Systems: rapid storage of individual experiences alongside slower distributed learning that extracts shared structure through interleaved experience.

MORPH does not require a literal mapping from these artificial components onto hippocampus and neocortex. The relevant idea is computational: fast memory, lawful state transformation, and broad overlapping representation learning solve different problems.

11. Encoder drift becomes explicit representation migration

If object memories store embeddings,

while \(\,E_\theta\,\) continues to learn, old stored vectors eventually become stale.

The coordinate system itself moves.

Rather than treating this only as a maintenance problem, MORPH makes migration compatibility part of the learning objective.

Let the frozen previous encoder be \(\,E_0\,\) and a candidate updated encoder be \(\,E_1\,\). MORPH tries to decompose representational change into:

Here:

• \(\,Q\,\) is a shared, invertible coordinate migration that carries old knowledge forward;
• \(\,u_{\mathrm{new}}(x)\,\) is residual plasticity that can add distinctions the old representation could not express.

A simple first choice is an orthogonal migration:

parameterized through the Lie algebra:

Because an orthogonal map preserves inner products and Euclidean distances,

the old identity geometry is preserved exactly inside the migrated subspace.

Train SGD to prefer migratable updates

During an escrowed encoder update, keep \(\,E_0\,\) frozen and jointly train \(\,E_1\,\) and the migration transform.

A protected-subspace compatibility loss can be:

Here \(\mathcal A\) contains historical replay anchors and \(P_{\mathrm{old}}\) selects the capacity assigned to old knowledge. On held-out anchors, the vector

measures migration mismatch. It is an error to audit, not an unrestricted learned term subtracted away to make compatibility look good.

A useful stability/plasticity split is:

New capacity can use reserved dimensions or an expanded latent, under explicit memory/compute budgets. The protected block can move coherently while the new block learns distinctions the old code lacked. An exact orthogonal transform cannot improve distances within the old block; improved discrimination must use added capacity or a separately tested relaxation.

Old stored embeddings do not contain the new information. Their new block remains missing until a raw anchor is re-encoded or the object is revisited; a query matcher must handle partial/versioned representations. The square change-of-basis and conjugation equations below apply to the protected block. Expanded capacity needs its own learned dynamics and compatibility tests.

Commit only after migration review

After the candidate update, MORPH evaluates at least four things:

1. old identity compatibility — do historical objects still retrieve and resonate correctly?
2. migration residual — how much old-state movement cannot be explained by \(\,Q\,\)?
3. old action dynamics — do previously learned transformations still predict correctly?
4. new utility — did the update actually improve the failure that triggered plasticity?

Conceptually:

candidate SGD update
        ↓
fit / refine Q
        ↓
migrate old anchors in escrow
        ↓
test identity + dynamics + new task
        ↓
   commit / reject

If the update is accepted, stored full-latent prototypes can migrate:

and invariant ANN keys can either be regenerated from the migrated prototypes or migrated through their own explicitly learned compatibility map.

This database migration can be eager or versioned/lazy. A memory can store the encoder version that created it, and the required migration transforms can be composed when that memory is touched.

The learned dynamics must migrate too

Suppose the old representation obeys:

After the coordinate change

the same physical transformation should be represented as:

This is a change of basis, not new physics.

It gives MORPH a strong anti-forgetting condition: an encoder update should preserve not only old object identities but also the lawful transition structure previously learned in the latent space.

The decoder and residual predictor must remain compatible too. On the protected block, an exact change of basis would require:

These equations suppress unchanged sensor context and ownership arguments for readability. They are tested compatibility conditions; conjugating the group operator alone does not enforce them. Candidate updates must also check decoded historical observations, the identity–pose factorizer, and retrieval keys.

Physical motion and encoder migration are different groups

This distinction is important.

physical symmetry
    SE(3), SO(3), learned local groups
            ↓
    action-conditioned latent motion

representation migration
    SO(d) or another feature-space group
            ↓
    encoder-version compatibility

The same Lie-group language is useful in both places, but they solve different problems.

The physical group says:

how should a representation change because the world or robot moved?

The migration group says:

how should an old representation be transported because the encoder changed?

MORPH needs both.

12. Spatial grouping can exploit local transformation consistency

The architecture still needs to bind patch evidence into current instances.

The first version does not require pairwise cross-prediction among every patch. Each patch still produces sparse evidence for retrieved identities:

Those scores form a spatial evidence field for each candidate.

But the equivariant architecture adds a second grouping cue: patches belonging to the same rigid or approximately coherent thing should often be explainable by the same underlying physical transformation.

A global camera or object twist can be written:

Its local effect on patch \(\,i\,\) can depend on depth, image position, orientation, object ownership, and other local context:

followed by:

The matrices or learned operators \(\,J_i\,\) need not literally be analytic camera Jacobians in the first prototype. The important constraint is that local patch transitions should be coordinated consequences of a shared cause, not unrelated transforms invented independently by every patch.

This suggests a useful grouping signal:

If several patches are best explained by one shared object-motion hypothesis, that transformation coherence is evidence that they belong to the same current thing.

Conceptually:

shared object / camera motion
          ↓
   local frame effects
    /      |       \
 patch 1 patch 2  patch 3
    \      |       /
     coherent transition
          ↓
     grouping evidence

A local morphological operation in log space can still encourage coherent regions without making it part of the identity-learning mechanism.

For example, a max-plus dilation can be written:

with erosion as the corresponding min-plus dual.

Opening, closing, connected components, transformation consistency, or a later learned grouping module can turn sparse patch evidence into candidate current instances.

This keeps five questions separate:

1. retrieval: what stored things might explain this patch?
2. vigilance: does the raw evidence actually match?
3. equivariance: does the patch change as the action model predicts?
4. grouping: which nearby patches share a coherent current cause?
5. identity: which persistent memory best survives future prediction?

The anti-forgetting migration should be more global than the physical patch dynamics. MORPH should not begin by allowing an arbitrary independent encoder-migration transform for every patch; that would make it too easy to preserve patches individually while destroying cross-patch geometry. A shared migration \(\,Q\,\) plus small constrained local residuals is the safer starting point.

13. A toy learning episode

Imagine a robot encounters a blue stapler that it has never seen before.

Step 1: encode

Fixed patches produce local equivariant representations:

The identity readout produces retrieval keys:

Step 2: retrieve

Several patches retrieve familiar neighbors:

red stapler
tape dispenser
hole punch
UNKNOWN

Step 3: vigilance

None of the known candidates explains enough of the observation.

A provisional identity is created:

The current observations may initialize its prototypes, identity keys, and orbit anchors, but they are marked as proposal evidence.

They cannot validate the identity they just created.

Step 4: commit an action-conditioned prediction

The robot intends to move the camera right.

The action model maps that command into a Lie-algebra element:

The finite transformation is:

The candidate grouping and pose determine locally conditioned consequences of that shared motion. For corresponding visible evidence, a simplified latent prediction is:

The provisional blue-stapler identity and known alternatives can therefore make different predictions about which patches should remain the same thing and how their representations should move.

The spatial decoder composes those predicted features into a future image, including where the stapler is expected to appear. That image prediction and its uncertainty are frozen in temporal escrow before the next image exists.

Step 5: act

The camera moves right.

Step 6: acquire independent evidence

A new image arrives and is encoded:

The predictions were fixed before these pixels existed. Compare the decoded prediction directly with the acquired image, and also compare latent predictions where correspondence is valid.

Step 7: resonance or reset

If the provisional identity predicts the new sensor observations and corresponding patch transitions better than the alternatives, it gains evidence.

If it fails badly, it can be revised, merged with a known identity, or reset.

Transformation coherence across several patches can also strengthen the grouping hypothesis that those patches belong to one rigid object.

Step 8: learn widely

Only after scoring, gradient descent may improve:

• the shared equivariant encoder;
• the shared decoder and identity–pose composition/factorization;
• the invariant identity/retrieval readout;
• the action-to-Lie-algebra map;
• the latent group representation;
• residual dynamics for non-group effects;
• vigilance calibration;
• hard-negative separation from the red stapler, tape dispenser, and hole punch;
• grouping behavior;
• eventually higher abstraction machinery.

Replay interleaves older experiences.

Step 9: escrow the encoder update itself

Suppose this episode exposed a genuine weakness in the representation: blue and red staplers were too difficult to distinguish.

MORPH does not immediately replace the deployed encoder.

It trains a candidate \(\,E_1\,\), fits a migration transform \(\,Q\,\), and asks whether old representations satisfy approximately:

It also verifies that old physical transformations remain valid under the changed basis:

Only after old identities, old dynamics, and the new discrimination problem all pass review is the new encoder committed.

Stored prototypes are migrated or lazily version-transformed, ANN keys are refreshed as needed, and the new residual capacity becomes part of the shared substrate.

The new identity itself may have been created in one encounter, while its experience contributes to general learning across the whole system without silently erasing earlier things.

14. Deep abstraction uses the same protocol recursively

MORPH is intended to be recursive.

At the lowest level, nodes represent local sensory states.

Higher levels may represent:

The architectural rule does not have to change.

A higher-level model \(H^{(\ell+1)}\) can be proposed from lower-level states:

But those same child states cannot be counted again as independent validation.

The higher model must earn support by predicting something else:

If the prediction survives new evidence, resonance strengthens the abstraction.

This gives a possible operational criterion for creating abstractions:

A higher-level model earns persistence when it compresses existing structure and predicts held-out or future consequences that its constituent memories do not explain as well individually.

A reusable “door-opening” model, for example, might combine a handle, hinge, panel, grasp action, and predictable transition. It becomes a persistent higher-level model only if the composition consistently earns new predictive evidence.

15. What MORPH is not claiming

This architecture borrows ideas from several established traditions but should not be confused with any one of them.

• Adaptive Resonance Theory motivates vigilance, reset, resonance, and dynamic category creation.
• Metric and prototype learning motivate separating a shared representation from an open-ended collection of identities.
• Group-equivariant representation learning motivates representations whose changes under transformations are structured rather than arbitrary.
• Lie groups and Lie algebras in robotics motivate representing continuous rigid motion through generators, exponential maps, twists, and compositional transformations such as \(\,SE(3)\,\).
• Backward-compatible representation learning motivates explicitly preserving interoperability between old stored embeddings and newer encoders.
• Belief propagation motivates careful treatment of recurrent messages and the danger of double-counting in loops.
• Complementary Learning Systems motivates separating rapid item memory from slower distributed structure learning.
• Active inference and information-seeking control motivate choosing actions that discriminate among competing hypotheses.

MORPH’s specific combination is a working research proposal:

temporal evidence escrow + open-ended identity memory + a shared equivariant sensorimotor substrate + explicit representation migration under continual learning.

The architecture does not assume that every sensory change is a Lie-group action. Rigid motion is the cleanest case. Occlusion, contact, topology changes, deformation, lighting, articulation, and independent agents may require local groups, piecewise models, or residual dynamics.

It also does not assume that the physical motion group and the encoder-migration group are the same object. They are deliberately separated.

The proposal should be judged experimentally.

Comparison with unsupervised re-identification

Representative unsupervised re-ID systems such as Cluster Contrast alternate feature extraction, clustering into pseudo-identities, and contrastive encoder learning. Their memory dictionaries make clustering and feature learning mutually dependent. “Unsupervised” means no target identity labels; it does not automatically mean a learner starts without pretraining.

MORPH shares the retrieval, prototype, and pseudo-association problem. Its proposed emphasis is a continuous embodied stream: allocate provisional identities immediately, predict action consequences, and test associations before they support persistent memory or candidate shared updates. Batch clustering can still be a useful consolidation step. The distinction is a lifecycle and validation protocol, not a claim that prior re-ID is only clustering or lacks temporal methods.

These are experimental comparisons, not demonstrated advantages. Evaluate both on the same stream and report initialization, information access, latency, and identity errors.

Comparison with Homomorphism AutoEncoders

Keurti et al.’s Homomorphism AutoEncoder (HAE, ICML 2023) is a direct precedent. It jointly learns observation encoding, decoding, and action matrices using reconstruction and multi-step latent prediction. The learned action maps seek to preserve composition: the matrix for a composed transition should agree with sequential matrix application.

HAE also discusses separating object identity as an orbit from pose along that orbit. MORPH should credit that overlap explicitly. Learning group-structured dynamics or describing identity through transformation orbits is not a new contribution here.

The proposed additions concern deployment over time: open-ended nonparametric identity memory, joint multi-object grouping, sparse retrieval, uncertain hypotheses challenged by later evidence, active disambiguation, and reviewed encoder migration. Both architectures include a decoder. MORPH explicitly decodes the transformed, recomposed embedding into sensor space and learns from real observation errors alongside latent consistency. The proposed distinction is continual memory and update governance, not decoder removal. An HAE-style model could supply MORPH’s shared predictive substrate.

HAE’s setting assumes group-compatible transitions with informative action signals. That does not establish that arbitrary fixed patches under occlusion, independently moving objects, or contact obey one invertible group action. MORPH must test those extensions and use explicit uncertainty and residual modeling where the assumptions fail. This comparison is conceptual; no performance advantage has been demonstrated.

16. Minimal prototype

A useful first experiment should be smaller than the full architecture.

Environment

Use a simulated camera observing a small world containing reusable rigid objects.

Requirements:

• objects can leave and later return;
• new identities can be introduced continuously;
• multiple similar objects can exist;
• the agent can translate and rotate its camera;
• selected objects can be translated or rotated independently;
• ground-truth identity and pose are available only for evaluation, not identity training.

Model

Use:

1. fixed image patches;
2. one shared equivariant predictive encoder with identity–pose factorization and composition;
3. a shared spatial decoder, sensor-space prediction/reconstruction losses, and an approximately invariant identity/retrieval readout;
4. ANN index over persistent local prototypes;
5. ART-like vigilance threshold;
6. UNKNOWN as a legal candidate;
7. provisional object memories with latent prototypes and orbit anchors;
8. action-to-Lie-algebra mapping;
9. a simple \(\,SE(2)\,\) or \(\,SE(3)\,\) latent group-action model;
10. optional residual dynamics for effects the group model cannot explain;
11. one-step temporal escrow predictions;
12. replay buffer;
13. simple spatial grouping plus transformation-consistency grouping;
14. candidate encoder updates trained with migration compatibility;
15. versioned memory migration after accepted encoder changes.

Add a fully unsupervised clustering/contrastive re-ID baseline and an HAE-style encoder/action predictor/decoder with a simple prototype gallery. Use the same input stream and pretraining policy, and record any extra segmentation or motion supervision. Compare joint grouping–pose inference against a diagnostic oracle-grouping condition to reveal whether failures originate in binding, dynamics, or retrieval.

The first prototype should prefer a low-dimensional known physical group over trying to discover every symmetry from scratch. Once the control loop is working, the harder experiment is to learn some generators from sensorimotor trajectories.

Encoder-update protocol

Three quantities must be kept separate:

The dynamics residual is not itself evidence of unmigratable memory. The migration residual measures the coordinate change’s unexplained error, not a fraction of records that are permanently unmigratable. Some memories may require backfill even when average error is low. Both the residual predictor and its outputs must remain compatible with the accepted encoder coordinates, through retraining/distillation or validated transport.

When a sustained prediction or re-identification failure triggers plasticity:

1. freeze the deployed encoder \(\,E_0\,\);
2. clone a candidate encoder \(\,E_1\,\);
3. train on current + replay data;
4. jointly fit a migration transform \(\,Q\,\);
5. measure migration residual on held-out old anchors, separate from the anchors used to fit the migration;
6. test old identity retrieval after migration;
7. test old action transitions after conjugating protected-block operators, and validate the residual head and any new capacity separately;
8. test the new failure case;
9. commit only if the candidate improves the target problem without unacceptable historical degradation, with a backfill policy for memories lacking new features.

Baselines

Compare against:

• fixed softmax classification with periodic retraining;
• nearest-prototype memory without vigilance;
• prototype memory + vigilance without temporal escrow;
• temporal escrow with a generic unconstrained next-latent predictor;
• equivariant action prediction without encoder migration constraints;
• migration-compatible learning without replay;
• replay without migration-compatible learning;
• full re-encoding of the historical gallery after every encoder update;
• versioned \(\,Q\,\)-based migration of stored representations.

Main measurements

Measure:

• new-identity creation precision and recall;
• false merges of distinct objects;
• false splits of one object into multiple identities;
• re-identification after absence;
• cross-view and cross-pose re-identification;
• action-conditioned latent and decoded sensor-space prediction accuracy;
• group composition error;
• inverse-consistency error where applicable;
• residual-dynamics magnitude;
• transformation-coherence quality as a grouping cue;
• catastrophic forgetting on old objects;
• cross-version retrieval compatibility;
• migration residual after encoder updates;
• historical identity geometry distortion;
• historical dynamics error before and after migration;
• sample efficiency for new objects;
• ANN retrieval quality as the identity library grows;
• calibration of resonance confidence;
• performance when visually similar objects require action to disambiguate.

Also measure grouping/correspondence quality, identity–pose leakage, pose uncertainty calibration, candidate-budget saturation, residual takeover, and identity stability under symmetric or partially occluded views. Test a rotating object crossing patch boundaries with its center hidden, two similar independently moving instances, and a deforming object. These expose assumptions that a centered single-object rotation demo would miss.

Three ablations are especially important.

First:

Does Lie-structured action prediction generalize to unseen action compositions better than an unconstrained next-state predictor?

Second:

Does migration-compatible encoder training preserve historical identities and dynamics better than replay alone without requiring a complete gallery backfill?

Third:

Does temporal escrow reduce self-confirming identity errors when identity, action dynamics, and encoder plasticity are all recurrently coupled?

17. Failure modes to expect

MORPH has obvious ways to fail.

Provisional identity explosion

If vigilance is too strict, normal viewpoint variation may create a new object every few frames.

The system then repeats an old failure mode: representational novelty becomes ontological novelty.

Equivariant orbit modeling may reduce this failure if viewpoint changes are explainable as lawful transformations of one identity.

Candidate count, unresolved-association entropy, memory-allocation rate, and inference latency can trigger a budget response. The robot can pause new memory allocation, narrow attention, slow motion, seek a simpler viewpoint, or retreat to a previously observed scene. An illustrative policy enters this mode after several consecutive budget violations and exits below a lower threshold, avoiding oscillation. Adjusting vigilance or filtering thresholds trades false splits against false merges; it should not force acceptance of an identity merely to reduce load. Preserve UNKNOWN and log the trade-off.

Identity collapse

If vigilance is too permissive, distinct similar objects merge.

Wrong group assumption

A rigid transformation model can be confidently wrong when the actual event involves deformation, articulation, occlusion, contact, illumination, or independent motion.

The residual model must be allowed to explain non-group effects without becoming an unrestricted escape hatch. The staged training procedure in section 8 freezes the group path while fitting that residual, then evaluates both paths on held-out transitions. Persistent structured errors should trigger a different motion model or richer context, rather than unlimited residual growth.

Residual takeover

If \(\,r_\omega\,\) is too expressive or weakly regularized, the system may ignore the structured group path and learn every transition in the residual.

The prototype should track the fraction of prediction improvement attributable to the equivariant path versus the residual.

Patchwise gauge drift

If every patch is allowed an arbitrary independent migration transform, local memories may remain individually compatible while cross-patch geometry becomes incoherent.

The first implementation should therefore prefer a shared migration transform with only limited local correction.

Overconstrained migration

If the encoder update is required to be almost exactly a global orthogonal transform, the system cannot repair genuinely poor historical geometry.

MORPH needs a protected migratable subspace plus controlled new capacity, not perfect rigidity.

Migration-chain accumulation

Versioned lazy migrations can accumulate numerical or modeling error after many encoder revisions.

Periodic consolidation may still need to re-encode high-value sensory anchors or collapse a chain of transforms into a fresh canonical version.

Correlated “new” evidence

The next frame is technically new but may contain almost the same information as the current frame.

Temporal separation is a causal provenance boundary, not a guarantee of statistical independence.

Longer temporal holds, withheld patches, new viewpoints, touch, or active experiments may provide stronger validation.

Shared-model contamination

Temporal escrow protects inference ordering, but aggressive SGD can still overfit recent experience or distort the representation.

Replay, migration review, and consolidation remain necessary.

Memory-key drift

Even if full latent prototypes migrate cleanly, a separately trained invariant retrieval head can drift.

ANN keys must therefore be regenerated, migrated through their own compatibility map, or produced by a sufficiently stable readout.

Confirmation through action policy

If the current hypothesis determines actions, it can choose observations that are easy for itself to predict.

An information-seeking policy should therefore prefer actions that discriminate competing hypotheses rather than merely maximize expected fit to the leading one.

Dynamics-preserving but identity-damaging updates

A coordinate migration may preserve the algebra of old transformations while still damaging the invariant identity readout.

MORPH must test both. Neither is sufficient alone.

Deep abstraction explosion

If every coincident group of lower-level models can create a higher-level memory, the hierarchy can grow combinatorially.

Higher abstractions will need stronger persistence criteria than one successful prediction.

18. Research principle

MORPH can now be compressed into three coupled rules.

Rule 1: evidence cannot validate itself

A model may use prior evidence to decide what to predict next. It may not count the success of a prediction unless the validating evidence was unavailable when that prediction was committed.

That is temporal escrow:

Rule 2: actions should move representations lawfully

A shared latent should preserve the distinction between identity and transformation.

Equivalence groups views judged to represent the same thing under allowed transformations. Invariance means a readout stays unchanged under those transformations. Equivariance means the full code changes in the corresponding predictable way. A front and side view can be equivalent for identity, share an invariant retrieval descriptor, and still have different equivariant pose codes.

The full representation is equivariant:

while identity is read out invariantly:

Motor action should map into generators of predictable latent motion:

The system therefore learns not only what state it is in, but also the lawful directions in which that state can move.

Rule 3: plasticity must carry old knowledge forward

When SGD improves the shared substrate, old memories should not simply become stale.

A candidate encoder revision should try to decompose its change into migratable old structure plus new capacity:

The update remains in escrow until historical identities and historical action transitions survive the change.

If committed, both memory and dynamics move with the coordinate system:

Around these rules, three complementary functions emerge:

The first lets an embodied agent remember a new thing immediately.

The second lets actions predict how that same thing should appear as the robot and world move.

The third lets the common representation improve without silently invalidating the things and transformations already learned.

This also suggests a different interpretation of re-identification. A persistent thing need not be represented only as one fixed point. It can be represented by an anchor together with its reachable orbit under learned transformations:

Re-identification can then ask whether new evidence belongs to a lawful trajectory of a known thing rather than relying only on static nearest-neighbor similarity.

If these rules can recurse through layers of learned models, object identity may be only the first use case.

The larger hypothesis is that an embodied intelligence could build an open-ended hierarchy of persistent models whose internal state is defined partly by what it is and partly by how it can lawfully transform, while retaining the aggressive distributed learning advantage of deep neural networks and without allowing recurrence or plasticity to erase its own history.

Future expansions

These are directions to investigate after the identity, prediction, and migration loop works.

• Egomotion, allomotion, and proprioception: separate the robot’s own movement from independently moving objects, using joint encoders, IMU/odometry, and touch where available. Predict relative sensor–object motion while retaining uncertainty about its cause.
• Object tracking: maintain a time-linked instance hypothesis as an object moves across patches, becomes occluded, and reappears. Tracking asks which current evidence continues the same physical instance; re-identification asks which persistent memory that instance belongs to. Keep competing associations and pose uncertainty through gaps, without treating an unobserved predicted trajectory as fresh evidence.
• Action-transition tracing: retain timestamped records linking the prior hypothesis and pose, commanded action, measured execution where available, predicted transition, observed outcome, and model version. Preserve the distinction between what was expected and what actually happened. These traces can support replay, credit assignment, and diagnosis of delayed effects; repeating or reinterpreting a stored observation must not count it as new independent evidence.
• Compositionality and articulation: bind parts into objects and relations, allowing a hinge or joint to move while the parent identity persists. Multiple interacting objects and changing topology need richer models than one rigid orbit.
• Language integration: attach words and descriptions to already grounded identities, relations, and affordances. Language can suggest hypotheses or tasks, while sensory evidence tests their physical implications.
• Continuous action-control manifolds: learn how a continuous action parameter maps to local latent generators, then use predicted outcomes for control. Motor commands generally include constraints, delays, contact, and noninvertible effects; a Lie group models suitable transformation components rather than the entire controller.
• State estimation and mapping: maintain relative frames, uncertainty, landmarks, and loop closure so a new view can be related to remembered objects during longer navigation episodes. Metric calibration may become necessary even if early pose representations are latent.
• Affordances and contact dynamics: learn what an object permits the robot to do, including grasping, pushing, and manipulating. These transitions require embodiment-specific feedback and often piecewise or hybrid dynamics.
• Planning, retrospection, and goal-directed behavior: express goals as desired object states, relations, or task outcomes, and roll the predictive model forward over candidate action sequences. Compare expected progress, uncertainty, cost, and physical feasibility, then replan as observations arrive. Retrospection revisits recorded transitions to explain failures or revise past associations; reconstructed and counterfactual episodes must remain distinct from observed history. Accurate one-step prediction alone does not establish reliable long-horizon planning.
• Time-based cognition: represent event order, elapsed durations, action-dependent delays, and predictions at multiple time horizons. Connect short sensorimotor traces into longer episodes so the system can anticipate an event, wait for an effect, remember what preceded it, and pursue temporally extended goals. The number of recurrent inference iterations is a compute budget, not a substitute for physical elapsed time.
• Resource allocation: choose which hypotheses deserve experiments, how much recurrence to spend, and when to consolidate, backfill, forget, or suspend new identities. Task cost and physical feasibility must constrain information seeking.

Each expansion introduces assumptions to test; none follows automatically from an equivariant encoder.

References and conceptual precedents

• Carpenter, G. A. & Grossberg, S. Adaptive Resonance Theory. See the Scholarpedia overview.
• McClelland, J. L., McNaughton, B. L., & O’Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex.
• Snell, J., Swersky, K., & Zemel, R. S. (2017). Prototypical Networks for Few-shot Learning.
• Dai, Z., Wang, G., Yuan, W., Zhu, S., & Tan, P. (2022). Cluster Contrast for Unsupervised Person Re-Identification.
• Keurti, H., Pan, H.-R., Besserve, M., Grewe, B. F., & Schölkopf, B. (2023). Homomorphism AutoEncoder — Learning Group Structured Representations from Observed Transitions. See also the full paper for identity/pose and orbit discussions.
• Cohen, T. & Welling, M. (2016). Group Equivariant Convolutional Networks.
• Bronstein, M. M., Bruna, J., Cohen, T., & Veličković, P. (2021). Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges.
• Shen, Y. et al. (2020). Towards Backward-Compatible Representation Learning.
• Backward-compatible and orthogonal feature-alignment methods motivate the idea that an encoder revision can preserve an older feature geometry through an explicit transformation while allocating additional capacity for new information.
• Lie-group state estimation and robotics provide the standard mathematical machinery for \(\,SO(3)\,\), \(\,SE(3)\,\), twists, exponential maps, adjoint transforms, and compositional rigid-body motion.
• Unit dual quaternions provide a compact representation of rigid-body rotation and translation and are a possible implementation choice for the \(\,SE(3)\,\) action path; they are not required by MORPH.
• Active inference and epistemic action provide one family of approaches for choosing actions that reduce uncertainty; MORPH uses that family of ideas only as a starting point for action selection.
• Standard treatments of loopy belief propagation illustrate the double-counting problem when evidence circulates around cycles; MORPH’s first prototype avoids solving general message ancestry by restricting new evidence credit to temporally escrowed observations.

Working research note · architecture under active review. Predictive Sensorimotor Grouping (PSG) is an exploratory design, not an experimentally validated model. The architecture has changed as the research question has become more precise.

The current V1 no longer assumes that Slot Attention is the grouping mechanism. Slot Attention remains an important baseline and source of ideas, but PSG now separates two problems that were previously conflated:

First, a recent evidence trail must be associated with a continuing active track: is this still the same currently observed thing? Second, that active track must be associated with a structured persistent object model: is this a known thing, and where is the current trail within its learned landmark topology?

The recent trail supplies local state. The active track supplies continuity through the current encounter. The persistent object model stores evidence and transition structure that may not have been observed recently at all.

Prediction, action conditioning, tactile sensing, active sensing, hierarchy, and scalable lifelong retrieval remain staged extensions rather than requirements for the first tracking and object-model experiments.

The problem: how does experience become a thing?

A robot does not receive objects. It receives streams.

A camera produces changing pixel values. Touch sensors produce pressure or contact histories. Encoders and inertial sensors estimate how the robot itself moved. Motor commands change what can be sensed next.

Somewhere between those streams and a useful model of the world, the robot needs to form provisional piles of evidence:

• these visual changes may belong together;
• this new feature may support one continuing hypothesis strongly and another weakly;
• this feature disappeared during an occlusion but may still belong to the same continuing source;
• two groups that currently move together may later separate;
• one group may contain evidence from two different things and need to split;
• two provisional groups may later turn out to describe one thing;
• an assignment that looked reasonable a moment ago may need to remain uncertain until later evidence resolves it.

The word object is intentionally loose. A useful persistent group might eventually represent an object, a part, a surface, a place, an articulated component, or another recurring structure.

PSG asks whether object-like organization can emerge from streaming evidence without supplying object identity as an input.

A simple counterexample motivates the broader research direction. Suppose a system successfully predicts a black seam on a basketball. Later it successfully predicts a black court line. Both predictions can be excellent. That does not mean the seam and the court line belong to the same physical entity.

Prediction is evidence about regularity. It is not, by itself, evidence of ownership.

PSG therefore separates three concepts:

1. A motif is a reusable local sensory or sensory-motion pattern.
2. An occurrence is one particular observation of a motif at a place and time.
3. A persistent hypothesis is the system’s revisable claim that some changing set of occurrences shares a continuing source.

The third item is the difficult one. A persistent thing does not need to be represented by the same pixels from frame to frame. Its supporting evidence may change continuously.

Architecture at a glance

The revised PSG architecture has three distinct temporal structures:

1. a recent evidence trail, containing what has just been observed;
2. an active persistent track, representing one currently traceable source through the ongoing encounter;
3. a persistent object model, retaining landmarks and transitions that can survive long absences from the current sensory stream.

The important distinction is:

Tracking asks whether current evidence belongs to the same continuing thing. Recognition asks whether that currently tracked thing corresponds to a durable object model encountered before.

Those are two different association problems.

flowchart TD
    V["streaming video"] --> E["local motif occurrences"]
    E --> R["recent evidence trails"]
    R --> A1["association 1: trail ↔ active track"]
    A1 --> T["active persistent tracks"]
    T --> A2["association 2: track + trail ↔ object model"]
    R --> A2
    A2 --> M["persistent landmark / transition models"]
    M --> L["localize recent trail within object model"]
    R --> L
    L --> P["later: predict next trail / landmark transition"]
    M --> P
    X["known action or self-motion"] --> P

An active track can remain strong even when recognition is unresolved:

active track H7

continuity confidence: high
persistent identity: unknown

That is a feature rather than a failure. A novel object should be trackable before the system knows whether it has seen that object before.

First association: recent trail to active track

Let \(R_i(t)\) denote recent evidence trail \(i\), and let \(H_k(t)\) denote active track \(k\).

Define a soft association

At each time slice, this is a bipartite association problem:

recent trails                         active tracks

R1  --------------------------------> H1
 | \                                  ^
 |  \-------------------------------> H2
 |
R2  --------------------------------> H2
 |  \-------------------------------> H3
 |
R3  --------------------------------> H1
    \-------------------------------> H3

Stack those association slices through physical time and the tracking problem becomes a conceptual volume

A continuing thing is therefore not one fixed set of pixels. It is a coherent path through changing trail-to-track associations.

Second association: active track to persistent object model

Now let \(M_j\) denote persistent object model \(j\).

A second relation asks whether active track \(H_k\), together with its accumulated and current trail evidence, corresponds to a known persistent object:

Failure to recognize an object therefore does not break tracking. A high-confidence active track can remain associated with an UNKNOWN alternative while PSG gradually constructs a new object model.

The persistent object model is structured

The long-term representation should not be only a FIFO of old evidence.

A persistent object model needs to retain evidence that may not have been observed recently and organize that evidence into something that can support relocalization and future transition prediction.

A minimal conceptual model is a set of landmarks plus learned transitions:

persistent object M42

        [L1 rim]
        /      \
       /        \
 [L2 side] ---- [L4 opposite side]
      |
      |
 [L3 handle junction] ---- [L5 handle]

A landmark need not be a named semantic part or an explicit 3-D coordinate. It can be a learned recurring evidence state, a prototype, a local latent state, or another compact representation.

The edges encode observed reachability or transition structure. Later, those edges can be conditioned on known action or self-motion.

The recent trail can act as the local coordinate

PSG does not necessarily need to encode the current location on an object as an explicit Euclidean pose such as

Instead, the recent trail can be matched against the object’s learned landmark topology:

Here \(\ell_t\) may be a landmark, a probability distribution over landmarks, a short landmark path, or a learned local state.

The persistent object model provides global context. The recent trail says where the current encounter appears to be within that structure.

This gives the architecture a route to learned geometry without requiring a pre-specified geometric coordinate system.

1. Visual Motif Encoder: Build local sensorimotor motifs before trying to build objects {:#1-build-local-sensorimotor-motifs-before-trying-to-build-objects}

The first stage deliberately avoids asking which object produced an observation.

Instead, it asks:

What local pattern of visual change occurred here over a short recent history?

V1 visual evidence

A minimal video prototype can begin with a short stack of frames or frame-to-frame differences.

One illustrative design retains nine camera samples and therefore eight differences. At each image location, an encoder can receive channels such as:

• change in intensity;
• local image motion or optical-flow-like x change;
• local image motion or optical-flow-like y change.

For a \(32\times32\) toy image:

The eight entries on the temporal axis are differences, not eight original frames. Producing eight differences requires nine samples unless the preceding difference has already been retained.

One illustrative 3D convolution uses ten filters, each spanning all three input channels with temporal-height-width extent \(3\times3\times3\):

visual_history = [3, 8, 32, 32]

responses = conv3d(
    visual_history,
    out_channels = 10,
    kernel = [3, 3, 3],
    stride = 1,
    padding = 0
)

# responses: [10, 6, 30, 30]

The exact tensor widths are implementation choices, not PSG requirements. A simpler V1 could use a small CNN on individual frames plus a short temporal encoder.

The important requirement is that the grouping stage receives local evidence occurrences with enough spatial and temporal support to remain separable.

A crucial choice is what not to do next. Pooling an entire temporal or spatial region too aggressively can mix evidence from different physical sources before grouping has even begun. A later hypothesis cannot reconstruct information that was already destroyed.

An occurrence might contain:

feature embedding
sensor-space support
short temporal support
timestamp
local motion estimate
uncertainty

The occurrence itself already contains a small amount of micro-time: it summarizes what happened locally over a short window.

The trail-to-track association process then evolves over a longer macro-time. This distinction matters:

short temporal motif
        |
        v
evidence occurrence
        |
        v
recent evidence trail
        |
        v
association through many video steps
        |
        v
active persistent track

Tactile motifs are a later extension

Touch should eventually get its own modality-specific encoder rather than being forced into the visual tensor. A tactile array may need longer temporal support for contact onset, sustained pressure, slip, release, and traversal across neighboring taxels.

That remains important to the broader PSG program, but it is intentionally out of scope for V1.

2. First matching problem: recent trails to active persistent tracks

After local feature extraction, PSG groups temporally adjacent occurrences into short recent trails.

A trail is more informative than one raw pixel or one isolated feature. It can contain local appearance, motion, timing, and the order in which nearby motifs were encountered.

recent trail R_i(t)

e(t-3) -> e(t-2) -> e(t-1) -> e(t)

PSG then maintains a bounded active set of tracks:

\(K\) is a computational budget, not a claim that the world contains exactly \(K\) objects.

The first association matrix \(A_t(i,k)\) asks:

Which active track, if any, best explains the continuity of this recent trail?

Trails can vote upward

A recent trail can support several active tracks while evidence is ambiguous:

trail R29 says:

H1: weak support
H2: strong support
H3: little support
H7: moderate support

Those values do not initially have to sum to one.

Two visually similar objects may remain ambiguous during an overlap or occlusion. PSG should be allowed to defer the decision until later evidence separates their trajectories.

Active tracks can answer downward

An active track can also inspect the current trails:

Which recent trails are consistent with the physical source I have been following?

active track H7 says:

R3:  strong continuity
R8:  weak continuity
R29: strong continuity
R44: moderate continuity

This makes the first matching problem naturally bidirectional.

A V1 update can alternate between:

1. recent trail -> active track
   "which continuing source do I support?"

2. active-track update
   integrate support with recent continuity state

3. active track -> recent trail
   "which trails remain consistent with me?"

4. revise A[t]

5. repeat for a bounded number of inference iterations

Competition is still useful where physical ownership should be exclusive, but it is not assumed to be the universal operation.

Time is the third dimension of tracking

A single \(A_t\) matrix describes only one physical moment.

The history

describes how recent trails remain associated with active tracks over time.

Suppose \(H_7\) is supported by one collection of trails now and a different collection later:

time t          time t+1        time t+2

R2 --\
R3 ----> H7      R8 --\          R14 --\
R4 --/           R9 ----> H7     R15 ----> H7
                  R10 --/         R19 --/

The individual evidence changes. The continuing source hypothesis persists.

This is the first meaning of persistence in PSG:

An active track is a temporally coherent path through changing recent evidence.

The implementation does not need to store the entire \(A(i,k,t)\) volume. It can keep a bounded recent trail history plus recurrent track state.

3. Second matching problem: active tracks to persistent object models

Tracking continuity is not the same as recognizing a known object.

Once an active track has enough evidence, PSG can compare it with a collection of persistent object models:

The second association matrix

asks:

Does active track \(H_k\), together with its current trail and accumulated encounter evidence, correspond to persistent object model \(M_j\)?

This association can also remain uncertain.

active track H7:

M12: 0.08
M42: 0.91
M77: 0.18
NEW: 0.11

The exact values need not be calibrated probabilities.

Unknown but consistently traceable objects

A particularly important state is:

H7

tracking continuity: strong
known object match: none

The system should be able to follow a previously unseen object for an extended encounter without prematurely forcing it into the nearest known identity.

As evidence accumulates, PSG can construct a new persistent object model from the active track.

Later, when the object disappears and reappears, the new active track can be matched back to that stored model.

A persistent object is not a long FIFO

The previous PSG draft treated slower evidence memory as though it might itself become the object representation.

That is insufficient.

An object model needs to preserve evidence that is not currently or recently visible and organize it into a structure that can be revisited.

A candidate model is:

M[j]
├── landmarks / persistent evidence states
│   ├── representative motif evidence
│   ├── uncertainty
│   └── modality-specific evidence later
│
├── transition structure
│   ├── observed landmark-to-landmark transitions
│   ├── transition confidence
│   └── later: action-conditioned transition models
│
└── identity-level state
    ├── accumulated encounters
    ├── model confidence
    └── lifecycle / consolidation state

The representation need not be an explicit metric mesh.

It can instead be a learned topology: a structured memory of which evidence states tend to follow or become reachable from which others.

Object construction and recognition are reciprocal

During a novel encounter:

recent trail
    |
    v
active track H_new
    |
    v
accumulate landmarks / transitions
    |
    v
new persistent model M_new

During a later encounter:

recent trail
    |
    v
active track H7
    |
    v
match H7 + trail against stored models
    |
    v
recognize M_new

This separation prevents representational novelty from automatically becoming ontological novelty. Failure to match a known model can remain uncertain while tracking continues.

4. Recent trails localize within persistent object models {:#4-thinking-harder-means-another-pass-over-the-same-evidence}

Once an active track has a plausible object-model match, the recent trail provides the current local state within that object’s learned topology.

Let the currently favored model be \(M_j\).

PSG can infer

where \(\ell_t\) is the current position in the learned landmark structure.

This position does not have to be a Cartesian coordinate.

It could be:

• one landmark;
• a soft distribution over landmarks;
• a short sequence such as \(L_{12}\rightarrow L_{13}\rightarrow L_{19}\);
• a learned latent state associated with a neighborhood of the object model.

Example: building a mug model

A persistent mug model might eventually contain:

L1: circular rim evidence
L2: smooth side evidence
L3: handle junction evidence
L4: handle outer-curve evidence
L5: bottom-transition evidence

learned topology:

L1 -> L2
L2 -> L3
L3 -> L4
L2 -> L5

Now suppose the current recent trail contains:

circular edge
    ->
smooth vertical surface
    ->
small horizontal protrusion

The trail can match approximately to

L1 -> L2 -> L3

inside the persistent model.

That gives the system both identity context and local state:

active track: H7
recognized model: M42
localized trail: near L3

This is more informative than a long-term bag of historical evidence. Evidence that has not appeared for a long time can still remain part of \(M_{42}\) and become relevant again when the recent trail reaches that region.

Recent memory and persistent memory now have different jobs

The short recent trail answers:

Where am I in the currently experienced evidence topology, and how did I get here?

The persistent object model answers:

What broader landmark and transition structure has been learned for this thing, including evidence not seen recently?

That distinction replaces the earlier fast-buffer / slow-buffer picture.

A small recent FIFO may still be an implementation detail, but the long-lived object representation should be structured rather than merely older evidence.

5. Slot Attention is a baseline, not the ontology

Slot Attention remains relevant because it offers a clear solution to one nearby problem: a fixed number of latent slots can compete to explain features within an observation.

That makes it an important baseline for the first matching problem.

Slot Attention can be summarized roughly as:

features
   |
   v
competitive assignment
   |
   v
fixed slot set
   |
   v
scene decomposition

The PSG tracking candidate is broader:

local occurrences
      |
      v
recent evidence trails
      |
      v
trails <----> active tracks
      |
      v
associations evolving through time

The difference is not that competition is forbidden. It is that competition is no longer assumed to be the only or fundamental interaction, and the active tracking layer is only one stage of the architecture.

Even a successful recurrent Slot Attention baseline would still leave the second problem open:

How does a currently tracked thing map to a durable object model containing landmarks that may not have been visible during the current encounter?

A useful experimental comparison is therefore:

• static Slot Attention;
• a recurrent/video Slot Attention baseline;
• one-way trail-to-track assignment;
• bidirectional trail↔track message passing;
• a persistent landmark store;
• a structured landmark/transition model;
• track→object recognition and relocalization.

If a simpler competitive recurrent-slot model performs equally well for active tracking, PSG should use the simpler tracking mechanism and reserve complexity for the object-model problem that actually requires it.

6. Prediction comes after tracking and object localization

Prediction is deliberately staged after PSG can maintain active tracks and construct or recognize persistent object models.

The architecture now gives prediction a much more specific input.

At time \(t\), PSG may know:

• persistent object model \(M_j\);
• current local state \(\ell_t\), inferred from the recent trail;
• recent trail \(R_t\);
• later, known action or self-motion \(a_t\).

A passive predictive version can begin with:

When new evidence arrives, the predicted trail can provide additional support for both track continuity and localization.

The causal rule remains strict:

A prediction used to support an association at time \(t+1\) must have been generated before the evidence at \(t+1\) was incorporated.

Otherwise a hypothesis can claim evidence, train on it, and then cite its own reconstruction as proof of ownership.

7. Actions condition transitions in the persistent trail

Action fits naturally once the object has a learned landmark or transition structure.

The key prediction becomes:

The terms now have distinct roles:

• \(M_j\): the persistent object model;
• \(\ell_t\): current localization within that model;
• \(R_t\): immediate recent evidence trail;
• \(a_t\): executed action or known self-motion;
• \(\hat \ell_{t+1}\): predicted next local state;
• \(\hat R_{t+1}\): predicted next evidence trail.

persistent object model M42
          +
current local trail near L3
          +
known move-right action
          |
          v
predict transition toward L4
          |
          v
predict handle-like evidence
          |
          v
actual next trail

If the expected transition occurs, several beliefs strengthen together:

• the active track is probably still following the same thing;
• the recognition match to \(M_{42}\) becomes stronger;
• localization within \(M_{42}\) advances toward the predicted region.

This gives PSG a form of sensorimotor geometry.

The model need not explicitly store an object-relative coordinate such as

Instead, the landmark topology plus learned action-conditioned transitions can encode which local evidence states are reachable from which others.

Actions therefore label or condition transitions through the persistent object model rather than becoming another axis of the representation.

8. Predictive Evidence Refinement and active sensing are later stages

Prediction can eventually do more than support temporal association.

A hypothesis might request extra computation on already available evidence or choose an action that obtains new evidence.

Internal refinement

A hypothesis could request a fresh sensory operation:

retained raw history
        |
        v
selected region / kernel / resolution
        |
        v
new local evidence occurrence

Or it could request a composite operation over existing occurrences:

occurrence A
occurrence B
motion evidence
      |
      v
selected relation / composition
      |
      v
higher-order evidence

The earlier Recurrent Predictive Routing (RPR) proposal explored a more specific mechanism involving compatibility-gated pairs and recurrent pair states. PSG should not assume that machinery belongs here unless experiments show that it helps.

External refinement

A robot can also act.

uncertain grouping
      |
      +---- THINK ----> build more evidence from existing observations
      |
      +---- ACT ------> change the world/sensor relation
      |
      +---- WAIT -----> preserve uncertainty

This remains part of the broader PSG direction, but it should not contaminate the first streaming-grouping experiment.

9. A first falsifiable world

The revised architecture suggests that V1 should be split into several experiments rather than jumping directly to motor prediction.

V1a: streaming track formation

Start with controlled video containing:

• two to five moving objects;
• identical or near-identical appearances;
• path crossings;
• brief occlusions;
• temporary common motion followed by separation;
• hidden simulator identities used only for evaluation.

Test whether recent evidence trails can remain associated with stable active tracks.

Relevant metrics include identity switches, fragmentation, inappropriate merging, recovery after occlusion, and uncertainty during genuinely ambiguous intervals.

V1b: persistent object-model construction

Next, expose one tracked object through changing views.

The model should accumulate landmark-like evidence states and transition structure that survives after individual landmarks leave the recent trail.

For example:

encounter over time:

rim
 -> side
 -> handle junction
 -> handle
 -> side
 -> bottom

persistent model after encounter:

L1 rim
L2 side
L3 handle junction
L4 handle
L5 bottom

plus learned transition relationships

The important test is whether old landmarks remain available after they have left recent memory.

V1c: recognition and relocalization

Let the object leave the scene.

Later reintroduce it under a different starting view.

The system forms a new active track first. It should then:

1. match that track to the previously learned object model;
2. localize the recent trail within the stored landmark structure;
3. preserve the possibility of UNKNOWN when evidence is insufficient.

This directly tests the separation between tracking and recognition.

Baselines and ablations

A useful experimental ladder is:

The experiment should not assume the larger model is best.

If a bag of old landmarks performs as well as a structured transition model, the topology may be unnecessary.

If recognition works without a distinct active-track layer, the two-stage association may be unnecessary.

If recurrent Slot Attention matches the full tracking performance with much less machinery, PSG should retain the simpler mechanism.

What would falsify the current formulation?

Useful negative results include:

• active tracks fragment whenever appearance changes;
• track-to-object recognition merely memorizes recent appearance;
• persistent landmark models fail to retain unobserved parts;
• relocalization fails after an object leaves and reappears;
• the structured landmark graph gives no benefit over an unstructured memory bank;
• unknown objects are incorrectly forced into known identities;
• introducing recognition destabilizes otherwise-correct active tracking.

V1 exists to discover which separations are actually necessary.

Persistent object models and scalable lifelong retrieval are separate layers

A persistent object model is now part of the core architecture: it is the durable landmark and transition structure that represents one learned thing across encounters.

But retrieving one object model from a lifetime containing millions of models is still a separate scaling problem.

The distinction is:

core persistent model:
    what is stored for one object?

global retrieval:
    which stored object models should be considered now?

A future sparse associative index, dense approximate-nearest-neighbor system, or hybrid mechanism could nominate candidate persistent models for the second bipartite association.

The important architectural rule remains:

Global retrieval may nominate persistent object models; it should not decide object identity by itself.

An unstable retrieval encoder can create a destructive positive feedback loop: failure to retrieve a known model creates a new object, and the existence of separate objects then teaches the encoder to preserve a distinction that may have been accidental.

The active tracking layer protects against this failure. A novel or poorly recognized object can remain one coherent active track while recognition stays unresolved.

First solve what one persistent model should contain and how an active track maps into it. Then solve retrieval across a very large model library.

Relationship to Thousand Brains / Monty

PSG and the Thousand Brains / Monty direction share several motivations: local sensing, temporal persistence, multiple hypotheses, movement, and eventually compositional behavior.

One conceptual difference is how geometry and persistence are represented.

A reference-frame-centered system can represent features at explicit locations in an object-relative model and infer identity and pose within that coordinate frame.

PSG now proposes a different decomposition:

• a recent evidence trail supplies the current local history;
• an active track preserves continuity through the current encounter;
• a persistent object model stores landmarks and their learned transition structure across encounters.

The recent trail can then be localized within the persistent object model:

Later, an action-conditioned predictor can learn:

If this is sufficient for useful prediction and grouping, PSG may not need to construct an explicit object-relative Cartesian frame. The topology of landmarks and predictable transitions can itself provide the geometry needed for action and persistence.

This is a hypothesis, not yet a result. An explicit reference frame may still prove more efficient or more stable. The proposed experiments should compare those possibilities rather than assume the topological representation is sufficient.

What PSG is claiming—and what it is not

The current research program separates four claims.

V1a tracking claim

A recent evidence trail can be associated through time with an active persistent track, allowing physical continuity to remain stable even while the visible evidence changes.

V1b/V1c object-memory claim

A currently tracked thing can be matched separately to a durable landmark/transition model, allowing recognition and relocalization without making long-term identity a prerequisite for tracking.

Predictive claim

Once a recent trail is localized within a persistent object model, the model and current local state can predict likely future trail transitions.

Sensorimotor claim

Known actions may condition those landmark transitions, allowing the persistent object model to acquire useful sensorimotor geometry without requiring an explicit object-relative Cartesian coordinate system.

Several pieces remain unresolved:

• how local occurrences should be compressed into recent trails;
• whether trail↔track inference should be competitive, collaborative, or hybrid;
• how active tracks should be born, split, merged, retired, and recovered;
• what constitutes a useful landmark;
• whether landmarks should be prototypes, learned latent states, short paths, or something else;
• how transition structure should be represented and consolidated;
• how a novel active track should become a new persistent object model;
• how track→object association should avoid premature recognition;
• what form localization \(\ell_t\) should take;
• how prediction should influence tracking, recognition, and localization without self-confirmation;
• how action-conditioned transitions should be learned;
• how touch should add landmarks or transition evidence;
• how part-whole hierarchy should interact with object identity;
• how global retrieval should nominate persistent models without controlling ontology.

Those are not details to hide. PSG is useful only if they can be converted into small falsifiable experiments rather than protected by adding machinery after failure.

The mental model to keep

The current shortest picture is:

sensory stream
      |
      v
local motif occurrences
      |
      v
recent evidence trail
      |
      v
+-------------------------------+
| MATCH 1                       |
| trail <-> active track        |
| "is this still the same       |
|  currently observed thing?"   |
+-------------------------------+
      |
      v
active persistent track
      |
      v
+-------------------------------+
| MATCH 2                       |
| track + trail <-> object      |
| "is this a known thing?"      |
+-------------------------------+
      |
      v
persistent landmark /
transition model
      |
      v
localize recent trail
within persistent model
      |
      v
later:
model + local trail + action
      |
      v
predicted next landmark /
evidence transition

The architecture therefore separates three kinds of state:

RECENT TRAIL
where the current evidence has just been

ACTIVE TRACK
which continuing physical source is being followed

PERSISTENT OBJECT MODEL
what has been learned about that source across
views and encounters, including landmarks that
have not been seen recently

The time-indexed trail↔track association still matters. It describes continuity through the current encounter.

The persistent object model sits above that tracking process. It supplies durable structure against which the current trail can be recognized and localized.

Action-conditioned prediction then operates on the combination:

The robot does not need to decide immediately what every observation is.

It needs a disciplined way to turn local evidence into recent trails, keep those trails attached to continuing active tracks, preserve uncertainty when identity is unresolved, and build durable object models whose landmarks remain available long after they leave the current view.

Revision 2 · Single-layer design under review. This article separates a concrete, dimensioned implementation sketch from the research questions it leaves open. The diagrams run scripted geometry, not trained RPR. There are no experimental results yet.

The problem: the world does not arrive as a sequence of independent frames

A robot sees a room, turns its camera, moves an arm, feels contact, and senses the consequences. Useful information lives both in individual observations and in how they change together:

• Which earlier signals help predict the next signal?
• Which relationships deserve computation now?
• Which observations belong to the same continuing thing?
• How can actions gather evidence that distinguishes competing explanations?

The first two questions concern predictive routing. The third concerns correspondence: deciding that observations at different times, locations, or sensors refer to the same entity. The fourth connects both to active sensing.

Recurrent Predictive Routing (RPR) explores a sparse network of learned predictive relationships. Its central unfinished question is how to organize those relationships into coherent, persistent explanations of the world. Accurate local prediction does not establish object identity.

A layman’s picture

Imagine watching a basketball, then lowering your camera toward a black line on the court. Orange pixels on the ball help predict nearby orange pixels; its black seams help predict nearby black pixels. After the camera moves, the court line also produces easy-to-predict black pixels.

A system could predict this entire sequence reasonably well while mixing evidence about the ball and the floor into the same memory. “I correctly predicted another black pixel” does not tell it which thing that pixel belongs to.

RPR therefore needs two kinds of progress: learning useful temporal relationships, and discovering where their evidence should accumulate. The first has a concrete working sketch below. The second is reserved for an interactive design session near the end.

Architecture at a glance

A token is one indexed unit of input presented to this layer. In our small example it is a sensor channel’s recent samples. A node is the processing/state record associated with that token.

A relationship pair is an ordered index pair (i, j): use evidence from input i to help predict input j. A route is the gate attached to that pair. The pair identifies the relationship; the gate decides whether its predictive contribution is computed or passed onward. In a graph drawing, that selected relationship is a directed graph edge.

Here is the complete single-layer roadmap. The numbers 1–5 match the explanation below. Stages 1a–1d are deliberately expanded: encoding, learned expansion, and recurrent calculation all happen before the compatibility matrix.

h names the short-history encoding; n names its expanded learned representation; s names recurrent per-token state; e names recurrent pair state. They are separate quantities in this revision.

flowchart TD
    X[/"Sensor samples and applied action"/] --> B["1a · Per-index four-sample buffers"]
    B --> H["1b · Encode each buffer: h"]
    H --> N["1c · Learned expansion: n"]
    N --> S["1d · Recurrent encoder: s"]
    S --> D[("Store s until next tick")]
    D --> S
    S --> C["2 · Compatibility matrix C"]
    C --> G{"2 · Select pair gates"}
    G --> E["3 · Update selected pair states e"]
    N --> E
    E --> ED[("Store pair state and age")]
    ED --> E
    E --> M["4 · Decode and combine predictions"]
    N --> M
    S --> M
    M --> P[("Cache forecasts made at this tick")]
    P --> L["5 · Score when the next sample arrives"]
    X --> L
    L -. training and later gate statistics .-> C

Buffers are drawn as sample storage, cylinders as state retained across ticks, a diamond as gate selection, and rectangles as computations. The detailed figures below open those computations into history cells, feature vectors, a feedback loop, and a matrix.

The fixed widths and GRUs below are proposed implementation choices, not established properties of RPR. The previous draft used n for a GRU state; this revision reserves n for the learned expanded encoding and gives the recurrent state its own symbol, s. That makes the distinction explicit rather than letting one symbol change meaning mid-explanation.

The working scope is one layer. A later hierarchy would reuse this kind of transformation on different inputs and separately maintained states; whether weights should be shared is a separate design choice.

A concrete toy world

Consider a camera at position \((0.5,1.5)\text{ m}\) in a \(4\text{ m}\times3\text{ m}\) room. A green marker sits at \((2.8,2.1)\text{ m}\), and an amber marker at \((3.2,0.8)\text{ m}\). Counterclockwise angles are positive. The green marker’s initial bearing is about \(+14.62^\circ\), at range \(2.38\text{ m}\).

The camera’s illustrative 110-degree field of view has twelve soft sampling locations:

bin_centers_degrees = [-55, -45, -35, -25, -15, -5,
                        5,  15,  25,  35,  45, 55]

To avoid handing object identity to the learner, the revised input sketch combines the two markers into one scalar intensity per visual bin. The green/amber labels are available to the illustration and evaluation code, not as separate “target” and “distractor” input channels.

Thus there are 15 indexed scalar streams, not fifteen known objects. Tokens 12 and 13 together describe one physical angle; splitting them is an explicit input-design choice. The command is an exogenous input: the model receives it but is not rewarded for guessing the next externally selected command.

Interactive world and sensor geometry

Move the slider to step through six scripted camera positions. The world panel shows physical locations; the sensor panel shows how the same geometry changes the sampled intensities.

What runs behind the diagram? The site’s JavaScript file, rpr-visualizer.js, uses Plotly to render a fixed sequence at times 0.0–0.5 seconds. The pan angles are manually specified as [0, 3, 6, 9, 12, 14] degrees. Commands [30, 30, 30, 30, 20, 0] degrees/second are the outgoing commands for those frames; the first five integrate to the next angle with a 0.1-second interval. The external controller commits the outgoing command before the predictor runs, so channel 14 supplies that known action alongside the current measurements. This timing makes the forecast action-conditioned without exposing future measurements.

For marker \(o\), the program calculates its bearing relative to the camera and a Gaussian response at each bin. The scalar input is the clipped sum:

bearing = degrees(atan2(marker.y - camera.y, marker.x - camera.x))
relative_bearing = bearing - pan_degrees
response[bin, marker] = (
    exp(-0.5 * ((bin_degrees - relative_bearing) / 6)**2)
    if abs(relative_bearing) <= 55 else 0
)
sensor_input[bin] = min(1, sum(response[bin, marker] for marker in markers))

All angular quantities must use consistent units in the Gaussian calculation; the code converts atan2 to degrees. Colored response components are diagnostic overlays; the combined bars are the proposed visual inputs. Responses outside the field of view are suppressed.

This is a scripted sensor-geometry illustration. It runs neither an RPR network nor a feedback controller or trained reference baseline. It demonstrates the input transformation a predictor would have to learn. It provides no evidence of learned tracking.

The 3D view uses the same scripted values as coordinates. Its axes are physical quantities, not learned latent dimensions.

1. From sensor samples to an expanded recurrent representation

The first stage runs independently for each sensor index. The diagram shows one such lane; there are fifteen lanes with separate buffers and recurrent states.

1a — Buffer the input. For each index \(i\), collect the last four scalar samples:

for i in 0..14:
    buffer[i] = last_four_samples_of_stream(i)   # [4]
# Whole batch: [15, 4]. Begin forecasting after four real samples.

This buffer is a literal short shift register. At 10 Hz it spans 0.3 seconds from oldest to newest sample. Normalize each channel with fixed training-set scales. A practical stream must additionally specify missing-sample handling and timestamps.

1b — Encode that short pattern. A learned feed-forward encoder expands four samples into sixteen features:

h[i] = tanh(linear_4_to_16(buffer[i]))   # [16]

Sixteen is a chosen feature width, not sixteen timestamps. The encoder can form combinations of level, change, curvature, and other patterns from the four samples. It does not create new observations. The same encoder weights can be shared across normalized channels while each index keeps its own data.

1c — Expand the learned representation. A second learned encoder produces the 32-value representation called \(n_i[t]\):

n[i] = tanh(linear_16_to_32(h[i]))       # [32], recomputed this tick

Here n is an expanded description of the recent sensor pattern, produced by its own learned projection. It is used both to form recurrent context and to update the pairwise relationship model in stage 3. Width 32 is another tunable capacity choice, not an additional 32 samples or a guarantee of richer information.

1d — Integrate with previous recurrent state. A GRU (gated recurrent unit) takes this tick’s 32-value encoding and the previous 32-value state to produce \(s_i[t]\):

s[i] = local_gru(input=n[i], previous_state=s_prev[i])  # [32]
# All 15 lanes update; stage 2 then receives s with shape [15, 32].

A GRU uses learned gates to mix a candidate state with retained previous state. One conventional parameterization is:

update_gate = sigmoid(input_projection(n) + state_projection(old_state))
reset_gate  = sigmoid(other_input_projection(n) + other_state_projection(old_state))
candidate   = tanh(candidate_input(n) + candidate_state(reset_gate * old_state))
new_state   = (1 - update_gate) * old_state + update_gate * candidate

Each gate has 32 components; * in the last two lines is elementwise multiplication. The matrices are learned, with separate weights for each named projection. Gate conventions differ between implementations; this is one explicit convention.

The GRU returns the current state immediately at every tick. An unrolled drawing shows repeated calls to the same cell across time, with shared weights. It is not a pipeline that waits for the sample to travel through 32 delayed stages. Only the four-sample buffer literally shifts samples.

What teaches these encoders? A prediction head reads s to forecast the next raw normalized sensor value. The next actual sample supplies the error, which trains the head, GRU, and n/h encoders together through backpropagation through time. Pairwise losses provide additional training signal in stages 3–5. No individual latent coordinate is preassigned “confidence” or “object identity.” Predicting raw measurements also avoids an unconstrained learned-target collapse in which both a latent target and its forecast become constant.

The input-to-state pipeline is now complete: four samples → 16 features → 32 expanded features → 32 recurrent-state values, per index.

2. Compatibility becomes a matrix of pair gates

Now place the fifteen current recurrent vectors into a matrix \(S[t]\in\mathbb R^{15\times32}\). The next computation compares every source row \(i\) with every prediction destination \(j\).

A source is the channel offering predictive evidence. A destination is the channel whose future value we want to predict. For example, (14, 12) asks whether the applied pan command helps predict the camera’s sine-angle channel. (7, 6) asks whether evidence at the +15° bin helps predict the +5° bin.

One illustrative compatibility function projects each state to two eight-value vectors:

keys    = linear_32_to_8(S)      # [15, 8], potential sources
queries = other_32_to_8(S)      # [15, 8], prediction destinations
C = keys @ transpose(queries) / sqrt(8)  # [15, 15]

The two projection matrices differ, so C[i,j] can differ from C[j,i]. A score is a learned priority proposal, not yet proof that the pair predicts well. Its supervision is discussed in stage 5.

The figure is schematic: shaded cells illustrate a selection, not trained scores. A route record uses (i,j) as its address; its gate controls computation or transmission for that relationship. The token vector is evidence entering the gate, and a learned pairwise prediction is the eventual contribution leaving it.

Keep the diagonal. The complete matrix has \(15\times15=225\) entries. Its fifteen diagonal cells represent self-comparisons. We retain those self pathways explicitly because they provide temporal prediction and novelty signals.

For a concrete budget, reserve one self pathway plus up to three other incoming pairs per destination:

self_records  = 15
cross_records = at_most(15 * 3) = 45
total_records <= 60

There are 210 off-diagonal candidate scores and fifteen diagonal scores; only 45 of those cross-pairs receive expensive pair updates at once. Thus “60 relationships” follows from the budget, not a hidden tensor dimension. In a different experiment, four cross-pairs plus a self record would require 75 slots.

Selection and transmission are separate: the self predictor stays available even when its outgoing novelty gate is closed. Cross-pair gates may start with Top-3 selection per column, then be compared against soft or hybrid selection.

3. Selected pairs update a recurrent relationship record

Take one selected pair, (i,j). It gets a record with a stable pair key, its last-update time, and a 16-value recurrent summary \(e_{ij}\). Think of a small row in a table addressed by (i,j), rather than trying to picture memory living inside a drawn arrow.

The raw samples remain in their four-sample buffers. The pair record stores a compressed learned summary, not a growing list of successful sensor samples. Its contents can reflect co-change, lag, or other useful predictive patterns, but those interpretations require measurement.

Using the two current expanded encodings gives a 65-value GRU input: 32 source values, 32 destination values, and one normalized elapsed-time value:

pair_input = concat(n[i], n[j], elapsed_since_pair_update / time_scale)  # [65]
e[i,j] = pair_gru(input=pair_input, previous_state=e_previous[i,j])      # [16]

For a newly allocated record, initialize the state to zero and mark the current time. Shared GRU weights process every selected pair; each pair keeps its own state. At most sixty records hold 960 pair-state floats. Allocation and expiry policies remain experimental.

Where is the predicted change? The GRU returns e, an intermediate representation. A decoder in stage 4 turns that representation into a proposed change in the destination signal. Prediction error trains both decoder and pair GRU; e itself is not required to equal a change in position or intensity.

We will call increases and decreases in a sensor signal temporal transitions. A graph edge means the indexed relationship (i,j). These two uses of “edge” must not be mixed.

Even a perfectly trained (7,6) record is still organized by sensor indices. It could summarize both a basketball seam and a court line passing those bins. That is a central limitation, returned to in the correspondence section.

4. Decode gated contributions and form destination predictions

The function G is a shared learned feed-forward message decoder. An MLP (multilayer perceptron) here means a learned linear projection, a tanh hidden activation, and a learned linear output. It takes the pair state and source encoding and emits a 16-value predictive contribution:

G = MLP(input_width=48, hidden_width=32, output_width=16)
m[i,j] = G(concat(e[i,j], n[i]))    # 16 pair-state + 32 source values

This message is a learned feature vector derived from the source and its relationship history. A separate output head gives it a directly testable role.

For measured destination \(j\), the self decoder reads local state and its self-pair record. The cross decoder adds a correction:

self_delta = D_self(concat(s[j], e[j,j]))      # 48 -> 1
self_prediction[j] = x[j] + self_delta
pair_correction[i,j] = D_cross(concat(s[j], m[i,j]))  # 48 -> 1
pair_prediction[i,j] = self_prediction[j] + pair_correction[i,j]

Both D functions are shared learned MLPs with 48 inputs, a chosen hidden width of 32, and one scalar output. The self pathway forecasts the destination’s change; the cross pathway asks whether additional evidence improves that forecast.

For active cross-pairs, normalize the selected compatibility scores into weights \(\alpha_{ij}\) and combine their scalar corrections:

weights = softmax(selected_cross_scores_for_destination_j)
combined_prediction[j] = self_prediction[j] + weighted_sum(pair_corrections)
# If all cross gates are closed, use the self forecast alone.

The original heading called this “messages update the destination.” More precisely, in this sketch messages update the destination’s prediction, while its recurrent local encoder continues to ingest its own observations. This keeps “self history” cleanly separated from cross-channel information. Feeding received messages back into node memory is a separate ablation; if adopted, the self baseline needs a separate local-only state to remain a fair comparator.

We cache each forecast now and judge it only after the next measurement arrives. All messages are computed from information available at the current tick.

5. Evaluate cross-prediction gain and self novelty

Stage 5 closes the learning loop at the next sample. Squared error on normalized raw observations is a simple initial loss:

actual = next_sample[j]
self_error = squared_error(actual, cached_self_prediction[j])
pair_error = squared_error(actual, cached_pair_prediction[i,j])
gain[i,j] = self_error - pair_error

A positive gain is evidence that this cross-pair added predictive value. It can train future compatibility scores or affect record retention. Forecasts must be cached before the target arrives; selecting pairs using their already-observed next-step error would leak the answer.

The prediction training objective can average self, pair, and combined errors over the measured channels (0–13). Training those losses through the decoders and recurrent states shapes e, s, n, and h. The external action channel is excluded as a prediction target but remains a source of context. The weights between losses, rollout horizon, and routing surrogate require experiments; hard selection itself does not supply gradients to rejected pairs.

The diagonal has another job. Comparing a self forecast with the same forecast gives zero incremental gain, so cross-gain is the wrong survival rule for self pathways. The local self-prediction error is a temporal novelty signal.

One provisional transmission policy smooths that error and emits a self message on novelty onset, sustained novelty at a limited rate, and recovery. Hysteresis uses a higher threshold to turn the gate on than to turn it off:

novelty[j] = smooth(cached_self_error[j])
if quiet[j] and novelty[j] > high_threshold:
    emit_self_event(j, "novelty onset")
    quiet[j] = false
elif not quiet[j] and novelty[j] < low_threshold:
    emit_self_event(j, "novelty subsided")
    quiet[j] = true
elif not quiet[j] and refresh_due(j):
    emit_self_event(j, "still surprising")

# Regardless of transmission, continue the self predictor's local updates.

These are rising/falling novelty transitions, not graph edges. A reliably predicted intensity increase can have little novelty; a static but unexpected signal can have substantial novelty. Noise and undertrained models also produce error, so novelty needs calibration and is not automatically meaningful evidence.

Cross-gain and novelty should both be retained as potential predictive-signature features. Neither is an object-identity certificate or proof of causality. Several individually useful cross-pairs can also be redundant; test combined forecasts and leave-one-pair-out contributions.

The complete toy forward pass

Returning to the overview, here is the dimension ledger for the same five stages. The formerly ambiguous “directional candidates” are simply the entries of the compatibility matrix.

for each new tick t:
    if cached_forecasts_exist:
        score_forecasts_against_current_measurements()  # stage 5 for t-1

    B = append_samples_to_four_sample_buffers()          # 1a
    h = encode_4_to_16(B)                                # 1b
    n = expand_16_to_32(h)                               # 1c
    s = local_gru(n, previous_s)                         # 1d

    C = compatibility_matrix(s)                         # 2: [15,15]
    pairs = all_self_pairs + up_to_three_cross_pairs_per_destination(C)

    e = update_pair_records(pairs, n, timestamps)        # 3
    forecasts = decode_and_combine(s, e, n, current_x)   # 4
    cache(forecasts, selected_pairs, forecast_timestamp)
    retain(s, e, pair_timestamps, running_statistics)

The capacity is an upper bound. Cross-records targeting the exogenous action channel can be skipped because it is not forecast; then at most 42 cross-records plus fifteen self records are maintained. Keeping the uniform 60-slot allocation makes the bookkeeping simple.

Fifteen local states and sixty pair states require 1,440 FP32 values: 5,760 bytes. Four-sample buffers add 240 bytes. These figures exclude weights, indices, timestamps, cached forecasts, transient tensors, training activations, optimizer state, and allocator overhead. A fixed state size describes bounded storage per slot, not lossless storage of an unlimited history.

The correspondence problem: a design-session placeholder

Status: open theory / interactive design session pending. The single-layer sketch above can learn sensor-level predictive relationships. It does not yet specify how evidence becomes associated with a persistent object.

Return to the basketball and court line. Successfully predicting a change at sensor j from sensor i tells us that a relationship was useful on that transition. Appending or compressing source evidence into j’s state does not say whether it belonged to a basketball, a line, a shadow, or an unrelated object now occupying the same location.

That raises an architectural choice: should persistent evidence be organized by sensor index, by predictive factor, or by a separately maintained entity hypothesis whose participating sensors can change? The first is easy to implement. The other two may be essential to correspondence, but their assignment rules are unresolved. Free association needs structure: allowing arbitrary evidence to mix would simply move the contamination problem.

Theory sketch: sorting unknown parts into provisional piles

Imagine a team sorting parts from five disassembled machines: a robot, a washing machine, a television, a car subsystem, and plumbing. Nobody is given the finished designs.

A person picks up a part, turns it, feels its surfaces, and tries a possible fit. Each small interaction supplies another constraint. The part may join an existing provisional pile, start a new pile, or remain ambiguously associated with several piles. Later evidence can force a split or reassignment.

The useful analogy is incremental evidence assignment under uncertainty. A pile is an evolving hypothesis, not a known object label. Motion, touch, visual structure, and predictive factors may constrain which evidence belongs together. Co-motion alone is insufficient: two separate objects may move together, while one articulated object has parts that move differently.

flowchart TD
    A[/"New sensory evidence and action context"/] --> B{"Which hypothesis fits?"}
    B --> C[("Existing provisional pile")]
    B --> D[("New provisional pile")]
    B --> E["Retain multiple assignments"]
    C --> F["Seek discriminating evidence"]
    D --> F
    E --> F
    F --> A
    F -. contradictions .-> G["Revise, split, or merge"]
    G --> B

This is a question map for the design session, not an implemented sorting algorithm. We still need to specify the message contents, compatibility tests, pile representation, uncertainty, birth/split/merge rules, and how hypotheses communicate across sensory regions.

The Thousand Brains Theory account from Numenta is relevant inspiration: it proposes sensorimotor object models with location context and communication among columns to reach agreement. This is a proposed theoretical framework, not proof that the sketch here solves correspondence. The pertinent lesson for RPR is that agreement about an entity requires a mechanism beyond independent next-signal prediction.

Desired efficiency: a bounded amount of work per arriving observation, even after much evidence has accumulated. Fixed-size recurrent summaries can bound memory per hypothesis, but they compress information and can forget. Constant work also requires bounded candidate retrieval and a policy for hypothesis capacity; comparing every new observation with every accumulated pile grows with the number of piles. The trade-off between bounded compute, ambiguity, and revisiting old evidence is part of the open problem.

The next design session should follow one part—or one moving patch—through several observations and answer: Which record receives this evidence, why that record, and what evidence would make us reverse the assignment?

Recursion is a later question

First establish one layer’s evidence semantics, learning objective, and correspondence mechanism. Adding layers before those are clear risks hiding the same ambiguity inside larger latent vectors.

A future layer could receive summaries of predictive factors or provisional entity hypotheses. It would execute the same kind of operations with its own inputs and persistent state. That does not imply copying activations or sharing network weights. Promotion rules, timescales, and downward control interfaces remain future design work.

PCFH provides a vocabulary for learned factors and composition. Whether those factors supply the right grouping unit is a concrete question to revisit after the single-layer experiment.

What RPR currently specifies

The working sketch specifies local temporal encoding, a compatibility matrix, gates on indexed pairs, recurrent pair summaries, and prediction-based training signals. It proposes distinct policies for cross-prediction gain and self novelty.

The broader ambition is coherent evidence accumulation. That remains an architectural requirement rather than a demonstrated consequence of these components.

What is incomplete

The following ten criteria are intended to remain fixed across iterations. These are editorial maturity ratings, not measured performance or probabilities that the theory is correct. The scale is: 0 = requirement identified but mechanism missing; 1 = a candidate mechanism is described; 2 = implemented with reproducible checks; 3 = supported by controlled baseline comparisons. A changed definition must be versioned rather than silently moving the goalposts.

Evaluation log: 2026-09-22, Revision 2, criterion version 1. This is the first recorded assessment; do not invent numeric ratings for the earlier article. In subsequent revisions keep these IDs, append dates and evidence, and explain score changes. There is intentionally no combined score: improvements in predictive loss cannot compensate for missing identity semantics.

Route learning and credit assignment

Hard Top-K selects a bounded number of pairs but does not differentiate through the selected indices. Selected pairs receive training while rejected pairs may never get a chance to improve. A router could learn from cached gain on explored pairs, but needs exploration and a clearly specified delayed credit rule.

Soft routing permits uncertain mixtures and gradient-based weight updates, yet dense scoring and updates can be expensive. Sparse candidate retrieval followed by soft gates, or a bounded set of competing assignments, are useful alternatives to test. The messy ambiguity in the sorting analogy is a genuine requirement, not merely an inconvenience to remove with a winner-takes-all operation.

Edge lifecycle

A graph-edge record currently stores the (i,j) key, learned pair summary, elapsed time, and utility statistics described in stage 3. The working forward pass keeps it by sensor-pair address; it has no object-ownership field or successful-evidence log.

We have not chosen how to preserve, expire, reset, or reassign a record when a pair becomes inactive or when a new scene uses the same sensors. Holding state indefinitely risks mixing entities; immediate reset loses useful continuity. An entity- or factor-indexed evidence store would need its own assignment mechanism before this lifecycle could be defined coherently.

Fair self-versus-cross comparison

Keep the self state local-only, give both models the same prediction horizon, and compare on held-out sequences. Match or report parameter budgets and inference work. A larger cross decoder can outperform a smaller self decoder for reasons unrelated to the source evidence. Test shuffled-source and unrelated-source controls, and whether individual gains survive joint aggregation.

Higher-level node creation

Deferred until one layer has a tested interpretation. Compression alone does not define which factors belong together or which variables a higher-level interface must preserve.

Control and causality

An action stream can help predict a consequence without identifying the true controllable mechanism. Randomized interventions, delays, hidden disturbances, and system changes are required tests. A controller, safety constraints, and a control objective are additional components; the scripted camera path supplies none of them.

Tracking under occlusion and ambiguity

No current mechanism binds an interrupted stream back to a continuing entity. Identical objects, crossing trajectories, and occlusions should remain explicit failure cases until a correspondence design exists.

Open questions

1. Should n remain a separate expansion, or can one recurrent encoder replace both projections without losing useful structure?
2. What does pair state retain that a matched-capacity local recurrent model cannot?
3. Which record should accumulate evidence when the sensor index stays fixed but the observed entity changes?
4. What evidence justifies starting, splitting, merging, or revisiting a provisional pile?
5. How should visual, tactile, and proprioceptive hypotheses communicate about a common entity?
6. Can cross-gain and self-novelty be calibrated without starving quiet but important relationships?
7. How can candidate retrieval remain bounded as object hypotheses accumulate?
8. Which predictive factors would constrain evidence grouping without incorrectly equating correlation with common identity?

Principal risks

• Identity mixing: ball and court-line evidence remain blended despite low prediction error.
• Graph churn and starvation: selections change too quickly, or early lucky pairs monopolize updates.
• Redundant evidence: several high-gain pairs repeat the same clue rather than provide independent support.
• False novelty: noise, camera motion, or an undertrained model dominate self-event traffic.
• Stale state: old pair summaries contaminate new scenes.
• Compression loss: finite recurrent state forgets distinctions needed for later reassignment.
• Hidden compute growth: dense pair scoring or an expanding pile search overwhelms the sparse update budget.
• Causal confusion: a useful predictor is mistaken for a reliable control pathway.
• Premature hierarchy: more layers obscure single-layer defects.
• Evaluation leakage: giving the learner “target” channels or evaluating on the scripted path falsely suggests correspondence has been learned.

The first falsifiable experiment

First build a prediction-only benchmark with randomized camera trajectories and scene configurations, using the unlabeled sensor inputs above. Split by entire scenes/trajectories, not adjacent frames. Train on more than the six illustrated states.

Compare:

1. A last-value predictor and a feed-forward model over the four-sample windows.
2. A flat GRU with access to the same information.
3. Independent per-sensor recurrent predictors.
4. The single-layer RPR sketch with self and cross records.
5. Ablations removing pair recurrence, cross-gain supervision, or novelty gating.
6. Hard gates versus soft gates under matched or explicitly reported compute budgets.

Measure raw next-step and rollout error, cross-source utility, route churn, novelty false-positive rate, latency, memory, and sensitivity to camera gain/delay changes. Test against stronger baselines, not only a weak self predictor.

Separately test correspondence. Build the basketball-seam → court-line counterexample; use two identical moving objects; hide and reintroduce an object; and later add visual/tactile observations of the same versus different things. Ground-truth identities belong only to evaluation, not to the input channels. Once a model exposes assignment hypotheses, measure identity switches, false merges/splits, recovery after occlusion, and contradictory evidence accumulation alongside prediction loss. Until then, record correspondence as not implemented, rather than treating good pixel forecasts as a passing result.

The decisive outcome could be low prediction error with poor or undefined identity consistency. That would show exactly where predictive routing stops short of the intended architecture.

Current conclusion: RPR has a more explicit single-layer predictive computation to investigate. Coherent evidence assignment—the mechanism that turns streaming observations into provisional, revisable “piles”—is the next design problem, not a benefit already obtained from recurrence.

References and implementation boundaries

• Cho et al., 2014: Learning Phrase Representations using RNN Encoder–Decoder introduces the gated recurrent unit used here as one candidate temporal encoder. This supplies a standard building block, not evidence for RPR.
• Numenta: The Thousand Brains Theory of Intelligence explains its proposed location-based sensorimotor models and voting between models. It motivates the correspondence discussion without settling RPR’s design.
• PCFH: variables to predictive factors is the related factor/composition proposal on this site.

I keep coming back to the same question:

How does that happen, exactly?

Because there seems to be a gigantic gap between the AI we have today and an independent intelligence capable of wiping out humanity, and I think people are hand-waving over a lot of very difficult engineering problems in between.

Current models are incredibly impressive. I use them all the time. They can write code, analyze systems, search through information, use tools, reason through problems, and do things that would have seemed ridiculous a few years ago.

They also drift constantly.

Give an AI agent a long enough task and eventually it starts losing track of what it is doing. Assumptions creep in. Intermediate mistakes propagate. It forgets constraints. Context gets summarized and changed. It starts solving a slightly different problem than the one you originally gave it.

Humans still have to keep pulling it back onto the rails.

That seems like a pretty important limitation for something that is supposedly going to independently execute some incredibly complicated plan to take over civilization.

Language grounded in language

I think the problem goes deeper than context windows or better memory systems.

Large language models mostly learn about the world through things humans have already written about the world.

Their language is grounded in more language.

Eventually that language traces back to physical reality, of course. Somebody touched the hot stove. Somebody watched the object fall. Somebody figured out what friction was. Somebody experienced what “mine,” “yours,” “give,” “take,” and “danger” meant before those concepts got written down.

But the model gets the description.

Humans and animals get the experience.

That distinction feels important to me.

One comparison I keep coming back to is how little language a child needs compared with a large language model. The exact number varies a lot depending on how you measure it and what environment the child grows up in, but by around four a child has heard on the order of tens of millions of words. I usually shorthand it as something like 15 million words. By that age, a child already understands language remarkably well and is becoming fluent.

Modern language models are trained on trillions of tokens. Meta says Llama 3 was pretrained on more than 15 trillion tokens.

That is an enormous difference in learning efficiency.

A child does not have to infer the entire meaning of “hot” from statistical relationships among sentences about hot things. They touch things. They feel temperature. They watch steam rise. They hear an adult say “hot” while something hot is actually in front of them. The same thing happens with weight, distance, ownership, fear, falling, giving, hiding, wanting, and thousands of other concepts.

Language is attached to an enormous stream of physical and social experience. In a sense, it is the human experience encoded into a communication system.

That makes me suspect an embodied intelligence could eventually learn and process language with dramatically less training data and compute than modern language models require. If the concepts are already grounded in perception, action, memory, and consequences, language does not have to reconstruct the world indirectly from correlations in text. The words can point at concepts the intelligence already has.

If I have a bad model of where a wall is, eventually I walk into the wall. Reality corrects me.

If I misunderstand how much force is required to pick something up, I drop it.

If I make a prediction about another person’s behavior and I’m wrong, I get feedback.

Physical reality keeps pulling our internal models back into alignment.

LLMs don’t really have that loop. They can consume sensor data, images, tool outputs, databases, and all kinds of other information, but there still isn’t a persistent intelligence continuously existing inside an environment and having its predictions corrected by that environment.

I suspect that matters a lot for long-term persistence.

What would an actually dangerous AI need?

If we’re talking about an AI capable of independently becoming an existential threat, it seems like it would need quite a few things that current systems aren’t particularly good at.

It would need to maintain an objective over very long periods of time without drifting.

It would need a reliable model of what is actually happening in the world.

It would need to learn from the consequences of its own actions.

It would need to recover from unexpected events.

It would need resources, compute, energy, communications, and some way to physically affect things.

It would need to keep working while people were actively trying to stop it.

And it would have to do all of that with very little human supervision.

That’s a much bigger leap than making the next LLM smarter.

My suspicion is that once you start solving those problems, you end up moving toward embodied intelligence.

And that’s where the argument gets interesting.

Why persistence probably needs reality in the loop

For an intelligence to stay coherent over long periods of time, I think it eventually needs a continuous loop with reality. It predicts something, acts on that prediction, sees what actually happened, and then corrects itself. You can fake pieces of that with databases, memory systems and agent frameworks, and we already do, but at some point there is a difference between remembering a description of the world and actually being in the world while it changes around you.

That does not necessarily mean a humanoid robot. An embodied intelligence could be a car, a factory, a laboratory, a building, maybe even a datacenter. The important part is that it has sensors and actuators and some real environment that keeps pushing back on its internal model. If it thinks a valve opened and the pressure sensor says it did not, that is a correction. If it thinks a robot arm is somewhere it is not, the encoder tells it. Reality keeps re-anchoring the state instead of letting an error get repeated for another thousand tokens.

This is also where embodiment changes the safety question in a way I don’t hear discussed much. Once an intelligence acts through a real system, it has a scope. A robot has motors and whatever tools are attached to it. A car has steering, acceleration and braking. A factory controller has whatever equipment is actually wired into that control system. Even a datacenter agent still has credentials, network boundaries and machines it can or cannot reach.

So suddenly the questions are pretty ordinary engineering questions. What can this thing actually control? What network can it reach? Which credentials does it have? What happens if it gets something wrong? Can another system revoke its access? How large is the failure domain? These are questions we already know how to reason about, even if the answers get harder as the systems get more capable.

Don’t build Skynet

Obviously we could throw all of that away and connect everything together. Give one intelligence access to every robot, every factory, every power station, every military system and half the Internet. I mean, sure, that sounds dangerous. It also sounds like an unbelievably bad system design.

We already spend a huge amount of effort trying not to design normal computer systems that way. We use authentication, permissions, separate administrative domains, least privilege, defense in depth, auditing, physical segmentation, all of that stuff. None of those ideas stop applying because the software got smarter.

If one future AI somehow has root access to civilization, the first catastrophic mistake happened before it decided to do anything. Somebody built a control plane with root access to civilization.

And I don’t really see why advanced AI should naturally converge on one giant central intelligence anyway. We don’t have one human brain controlling the planet. We have governments, companies, communities, machines, protocols, competing interests and lots of overlapping authorities. It is messy, sometimes terribly so, but the mess is also part of what keeps one failure from instantly becoming everyone’s failure. I think advanced AI probably needs that same kind of pluralism.

Robot sociology

This is probably the part of the whole argument that I find most interesting. If embodied intelligence becomes common, there won’t be a robot. There will be millions of machines and systems owned by different people, doing different jobs, sharing spaces and depending on one another. At that point their biggest environmental complication may actually be each other.

Two machines want the same charging station. One robot needs another robot to move first. A warehouse system needs a delivery system to show up when it said it would. Machines owned by completely different organizations need to share a road, a loading dock, radio spectrum, electrical power, compute, whatever. Pretty quickly you need identity and communication, but then probably also reputation, negotiation, agreements and some way to handle conflicts when somebody doesn’t do what they said they would.

Basically, robot sociology.

And humans already have a version of this problem. We are intelligent, autonomous, persistent, frequently selfish, occasionally violent, very good at manipulating our environment, and definitely not all aligned to the same objective function. Civilization works as well as it does because we built a huge pile of social systems around ourselves: contracts, laws, courts, norms, markets, governments, professional standards, reputation, checks and balances. They’re all flawed, and some are a mess, but the basic idea is that nobody has to be perfectly trustworthy for the larger system to function.

I don’t see why artificial agents would be fundamentally different there. In some ways we could even make the machinery more explicit. Machines can have cryptographic identities. Permissions can expire. Actions can be logged. Capabilities can be revoked. Dangerous actions could require agreement from several independent systems. Different vendors and different models could reduce the chance that one bug, one bad update, or one weird behavior propagates everywhere at once.

That seems like a much more realistic safety model to me than putting all of our effort into making one gigantic intelligence perfectly good forever. You assume individual parts can fail, including intelligent parts, and then you design the system so one failure doesn’t own everything.

Does this mean AI can’t be dangerous?

No, and I don’t want to make that claim. AI can already be used for fraud, cyberattacks, propaganda and plenty of other harmful things. Humans can connect models to dangerous systems right now, and future models are obviously going to become more capable.

What I don’t buy is the idea that the path from today’s LLMs to “AI kills every human” is short or automatic. There are a lot of hard things that have to get solved along the way: persistent agency, grounding, long-horizon reliability, physical autonomy, access to resources, and then enough real-world authority to do something at civilization scale. Any one of those is a serious engineering problem.

And the thing I keep coming back to is that solving those problems may change the risk at the same time. If persistence needs better grounding, and better grounding pushes us toward embodied systems, then those systems start having actual boundaries. Once there are lots of bounded intelligent systems, they need ways to coexist, and that creates pressure for some kind of social structure and governance between them. None of that guarantees safety, obviously, but it means the future does not have to look like one giant intelligence suddenly escaping from a chat window and taking over the planet.

Could we still screw this up? Absolutely. We could centralize everything, give one system absurd permissions, connect critical infrastructure together in ways that create giant correlated failure domains, or deliberately build autonomous weapons without enough checks. Humans are very capable of making bad architectural decisions when convenience or money is involved.

I just think the extinction argument often skips over too much of this middle. It takes today’s models, assumes the limitations disappear, assumes autonomy and persistence get solved, assumes intelligence turns into real-world power, and then jumps to the end state. Maybe that chain happens. I don’t think it is impossible. I just don’t think we have enough reason to treat it as the default outcome.

The future I find more plausible, and honestly more interesting, is a huge number of specialized intelligences living and working alongside us. Some will be robots, some will run facilities, some will help people, some will do science or manage infrastructure. They’ll have to cooperate with humans and with each other, and we’ll end up building rules and institutions around that whether we call it “robot sociology” or something less silly.

That sounds difficult, but it doesn’t sound hopeless. I think we have a pretty bright future to look forward to.

Part IV · Experiments, failure modes, and falsification · approximately 10–12 minutes

The previous three parts define the architecture’s contracts but intentionally leave several algorithms open: sparse candidate retrieval, participant grouping, factor retention, composition grouping, reference-frame learning, and inverse-control implementation.

A research architecture is useful only if those choices can be compared experimentally and if failure produces evidence against the central hypothesis rather than another layer of explanation.

11. Why this resembles renormalization

The renormalization analogy is useful only as an intuition for effective descriptions. It is not another mechanism in PCFH.

In statistical physics, coarse-graining eliminates microscopic variables while preserving selected large-scale behavior. The effective description can contain new interactions that summarize the influence of hidden detail.

PCFH proposes a learned analogue with a different selection criterion:

many lower-level dynamical factors
        ↓
find a subgraph with a compact external interface
        ↓
hide its internal realization from the next layer
        ↓
retain a macro-state and effective predictive/control law

The shared idea is that a useful coarse description preserves the behavior that matters while making fewer internal degrees of freedom explicit.

The analogy should stop there. PCFH is not claiming to implement the renormalization group from physics.

12. Developmental learning

A plausible training curriculum begins with safe intervention rather than passive observation.

Phase 1: Safe excitation

Use a conservative baseline controller to generate bounded perturbations, repeated trajectories, load changes, and contacts.

Phase 2: Learn sparse influence proposals

Train compatibility heads to identify which token pairs have predictive or interventional relationships.

Phase 3: Learn participant grouping and local factors

Allocate factor slots to candidate participant sets and test whether joint dynamics are simpler or more useful than alternatives.

Phase 4: Learn reference frames and composition

Search for coordinates that simplify prediction/control, then test which retained factor subgraphs admit compact external interfaces.

Phase 5: Learn inverse control around stable baselines

Use factor sensitivities first for residual corrections, feedforward terms, or target refinement rather than immediately replacing the proven low-level controller.

Phase 6: Add longer temporal abstractions

Repeated transformation sequences become candidates for skill-like macro-factors.

Phase 7: Add task objectives

Task reward selects desired high-level consequences. Physical safety constraints remain a separate hard control plane rather than another reward term that the agent can trade away.

13. Failure modes that could kill the idea

The main risks are architectural rather than merely hyperparameter problems.

• Correlation masquerading as causality. Co-moving variables may share a hidden cause. Randomized interventions are essential.
• Quadratic routing. Dense pair search may dominate compute before sparsity can help.
• Transitive over-grouping. A–B and B–C pairwise compatibility may incorrectly produce one A–B–C factor.
• Factor explosion. A sparse proposal graph can still create too many persistent factor hypotheses.
• Composition over-grouping. A chain of retained factors may be merged into one macro-subsystem merely because the factor graph is connected.
• Premature composition. Hiding state too early can remove information later needed for prediction, control, or safety.
• Hierarchy collapse. Higher layers may simply copy lower-level state.
• Unnecessary hierarchy. Extra levels may add latency without discovering new effective interfaces.
• Gauge ambiguity. Many coordinate systems may describe the same dynamics; equivalent frames should be allowed if behavior is preserved.
• Singular inverse control. Desired changes may be unreachable, underactuated, or ill-conditioned.
• Model exploitation. A planner may exploit model error instead of learning valid behavior.
• Representation drift. Lower-factor changes can invalidate higher controllers.
• Temporal under-modeling. Delay, backlash, hysteresis, discontinuous contact, and long memory may exceed the structured temporal basis.
• Integral windup. Persistent unreachable goals can make high-level accumulated discrepancy pathological.
• Safety-critical weak couplings. Sparsity objectives may prune rare but dangerous interactions.

A successful design therefore needs explicit null hypotheses and ablations, not only end-task reward.

14. The first experiment

The first test should be much smaller than vision.

Use a simulated two- or three-link arm with:

• permuted and differently scaled sensor channels;
• joint encoders;
• motor commands;
• end-effector coordinates;
• contact sensing;
• variable payload;
• one movable object;
• rotated external coordinate frames.

Do not tell the model which channel represents what.

Compare at least:

1. a flat MLP dynamics model;
2. a transformer dynamics model;
3. a neural relational inference model;
4. a compatibility model without structured temporal channels;
5. a temporal-relational model without hierarchy;
6. a two-level PCFH model.

The primary question is:

Can the system discover a compact dynamical factorization whose higher-level interfaces preserve both prediction and controllability while hiding unnecessary lower-level detail?

Experiment A: compatibility and participant grouping

Construct synthetic cases in which pairwise compatibility is deliberately non-transitive.

Example:

A ↔ B     strong under mechanism 1
B ↔ C     strong under mechanism 2
A ↔ C     weak

Compare participant-grouping strategies:

per_head_connected_components
seed_and_grow_with_predictive_gain
learned_factor_slots
hyperedge_proposals
pair_factors_only

Measure:

• one-step and rollout prediction;
• intervention consistency;
• false transitive merges;
• factor persistence;
• active factor count;
• compute and memory.

The architecture is weakened if grouping requires privileged semantic labels or if no grouping strategy improves on simple pair factors at matched compute.

Experiment B: composition grouping

This is a different experiment over retained factors rather than compatibility edges.

Create a system with a known internal subsystem—for example several joint-level factors that together determine hand motion—and external variables that interact only through a small boundary.

Compare composition criteria such as:

• connected factor subgraphs;
• minimum-cut / boundary-size heuristics;
• learned composition slots;
• predictive-information bottlenecks;
• no composition.

Measure whether the proposed macro-token preserves:

• future external-state prediction;
• action-to-effect prediction;
• reachability judgments;
• inverse-control quality;
• uncertainty calibration;
• transfer after changing internal realization.

A strong result would be a hand-level macro-factor that remains useful after changing link masses or sensor gains even though the detailed lower-level realization has changed.

Experiment C: reference-frame transfer

Perturb coordinates without changing the underlying task:

• rotate the camera frame;
• permute encoder channels;
• change sensor scaling;
• change link masses;
• attach a payload.

If higher factors retain their predictive/control meaning while low-level realization adapts, that is evidence for genuine reference-frame-like abstraction rather than trajectory memorization.

Experiment D: scaling

Sweep:

• input-token count \(N\);
• retained-neighbor budget \(k\);
• factor-slot budget \(F_{\max}\);
• factor dimension \(d_f\);
• hierarchy depth \(L\).

Measure wall-clock latency, memory, active factor count, compression ratio, rollout quality, control quality, planning horizon, and transfer at matched compute.

The depth claim is weakened if extra levels do not improve predictive/control efficiency or compositional transfer.

Key ablations

At minimum:

remove_integral_channels
remove_derivative_channels
remove_interventions
replace_participant_grouping
remove_composition
replace_composition_grouping
remove_shared_forward_inverse_structure
remove_reference_frame_objective
remove_hierarchy

The proposal should also be considered weakened if generic recurrent memory performs just as well, learned compatibility edges fail interventional tests, factor grouping is unstable, composition does not preserve control, or the forward model provides no measurable benefit to inverse action selection.

Safety boundary

The hard safety plane should remain outside task reward. Learned actions or targets should pass through verified actuator, state, and recoverability constraints wherever practical.

Work such as Safe exploration with learning-based model predictive control illustrates one family of approaches for learning around a recoverable safe set. PCFH does not depend on that specific method; the important architectural separation is that “unsafe but highly rewarding” should not be a valid trade inside the task objective.

15. The core hypothesis

The entire proposal can now be summarized without introducing another abstraction mechanism.

Within one layer: Relate

Between layers: Compose

Across the hierarchy

The deepest research hypothesis is therefore:

Prediction and control can be dual traversals of a recursively composed hierarchy of learned dynamical factors, where each abstraction is justified by preservation of a smaller predictive-control interface.

If that hypothesis survives the experiments above, the same machinery could potentially discover body structure, object-relative coordinates, manipulable interfaces, skill-like temporal factors, and task consequences without requiring each semantic level to be designed independently.

If it does not survive—if composition fails to preserve control, grouping is unstable, depth provides no matched-compute advantage, or the hierarchy merely copies state—then the architecture should be revised or abandoned rather than explained away.

Part III · Composition, recursive blocks, and scaling · approximately 12–15 minutes

Part I introduced Grouping 1: pairwise compatibility edges are converted into participant-set hypotheses for predictive factors. Part II gave those factors dynamics and a forward/inverse control interpretation.

This part introduces the second, distinct grouping problem:

Given a graph of already-learned predictive factors, which factor subgraphs can be hidden behind a smaller predictive-control interface and replaced by macro-tokens at the next level?

That is Grouping 2, followed by Compose.

The second grouping operation: composition grouping

After factor learning, suppose the current layer contains retained factors such as:

F_motor     motor command ↔ current ↔ joint response
F_link      joint 1 ↔ joint 2 kinematics
F_hand      joint state ↔ hand pose
F_contact   hand pose ↔ object contact
F_object    contact ↔ object motion

These factors are themselves connected because they share participants or exchange predictive messages. That creates a retained factor graph.

Grouping 2 does not ask which raw tokens belong in one local law. It asks whether a subgraph of local laws behaves like one coherent subsystem from the outside.

For example, \(F_{\mathrm{link}}\) and \(F_{\mathrm{hand}}\) might be composable into a hand-motion macro-factor if the upper layer can reason accurately using hand pose, reachable motion, and uncertainty without needing every joint-level interaction.

The special case of a single already-useful factor promoting itself is allowed. But the general operation is factor-subgraph composition.

5. Compose means interface-preserving compression

Compose answers one concrete question:

Can this retained factor subgraph expose a smaller interface upward while preserving the predictions and control effects that matter outside it?

The full pipeline therefore contains four distinct operations that had previously been easy to blur together:

That is the central architecture. There is no separate stack of “factor compression,” “Compose compression,” “interface compression,” and “learned compression.” Those phrases all refer to pieces or views of the same inter-layer composition boundary.

Interface-preserving compression

Imagine drawing a cut around a retained factor subgraph.

• Internal state \(x_A\) is detailed state needed to model what happens inside the cut.
• Boundary state \(x_B\) is the state through which the subsystem interacts with the rest of the factor graph.
• The interface is the predictive and controllable relationship visible across the cut: what the outside can observe, request, influence, and remain uncertain about.

flowchart LR
    subgraph L[Lower level factor subgraph]
        F1["motor / joint factor"]
        F2["link factor"]
        F3["hand factor"]
        F1 --> F2 --> F3
    end
    I["Preserved interface: hand state, reachable motion, uncertainty, action port"]
    M["Next-layer macro-token"]
    F3 --> I --> M
    M -. "desired interface state" .-> I

A higher object-manipulation layer should not need every motor current if the arm subsystem can expose an interface such as hand pose, reachable velocity, force envelope, contact effect, and uncertainty.

The lower-level details are not erased globally. They remain available below the abstraction boundary for local prediction, execution, and downward control. They are merely hidden from the next layer’s default state representation.

A learned composition function

One way to write the macro-state is

The function \(\phi_\theta\) is a learned composition operator. It can be implemented by a graph encoder, attention mechanism, state-space block, or another structured model.

The success criterion is not reconstruction of every hidden variable. It is preservation of external behavior.

Three mathematical views of eliminating internal detail

These are three views of one composition operation, not three architectural stages.

Probabilistic view. If \(\psi(x_A,x_B)\) describes joint compatibility, internal variables can be marginalized:

Optimization view. If \(E(x_A,x_B)\) is local incompatibility, an effective boundary cost can be

Learned predictive-control view. Train the macro-state to preserve quantities such as:

• future boundary state;
• action-to-effect mappings;
• reachable transformations;
• important uncertainty;
• temporal dynamics;
• safety-relevant couplings.

The third view is probably the most directly useful for implementation. The first two explain what “eliminating internal detail” means mathematically.

Sparsity is not abstraction

A graph can be extremely sparse while still containing many state variables. If 10,000 tokens each retain four neighbors, interaction search is sparse, but an upper layer still cannot afford to reason over all 10,000 states indefinitely.

PCFH therefore has three different reductions:

edge sparsity       fewer pairwise proposals
factor sparsity     fewer retained local laws
composition         fewer / smaller states exposed upward

If an early layer already discovers a sufficiently compact state, later layers should be allowed to stop compressing, pass tokens through, or operate at the same effective scale. Hierarchy depth is useful only when another predictive-control abstraction genuinely exists.

How a composed subsystem becomes a factor token

Let \(M\) denote a composable factor subgraph. Its next-layer token can be a fixed-width projection such as

Possible components are:

• \(s_M\): current macro relational state;
• \(D_M\): current macro transformation summary;
• \(I_M^\epsilon\): persistent mismatch summary;
• \(\Sigma_M\): uncertainty summary;
• \(e_M^{\mathrm{frame}}\): local-frame descriptor;
• \(e_M^{\mathrm{interface}}\): exposed action/effect interface;
• \(e_M^{\mathrm{type}}\): optional learned factor-family embedding;
• \(P_\ell\): projection to the fixed token width expected by the next layer.

A small handle can point back to the lower-level composed subgraph when a higher layer requests a rollout, sensitivity query, or downward target. The full participant list, raw source values, learned weights, and dense Jacobians do not need to be embedded in the token.

This makes the hierarchy lossy upward but not destructive overall.

6. Objecthood becomes a dynamical property

This composition criterion suggests an operational notion of objecthood or subsystem coherence.

A factor subgraph deserves promotion when:

1. its internal relationships are stable and strongly predictive;
2. interaction with the outside can be summarized through a smaller interface;
3. the interface preserves useful prediction;
4. it preserves relevant controllability and reachability;
5. it preserves important uncertainty and safety-relevant effects.

Conceptually,

This is not yet an exact loss. It expresses the intended pressure.

A rigid object is a natural example: thousands of pixels can move coherently while a much smaller state—pose, velocity, geometry, material/contact parameters—captures most of what another subsystem needs to predict and control its interaction with the object.

A limb can have similar structure. So can a skill, if many lower-level trajectories can be hidden behind an interface such as “move the hand along this reachable path with this force envelope.”

Objecthood, body-part structure, tools, and temporally extended actions could therefore emerge from one composition criterion rather than independent hand-coded categories.

9. A repeating block

The architecture can now be written as a typed transformation:

where both \(Z^{(\ell)}\) and \(Z^{(\ell+1)}\) are sets of fixed-interface tokens.

One full block is:

flowchart TD
    Z["Input tokens Z(l)"] --> H["Compatibility heads"]
    H --> E["Sparse proposal edges"]
    E --> G1["Grouping 1: participant sets"]
    G1 --> F["Predictive factor slots"]
    F --> V["Predict and validate"]
    V --> R["Retained factor graph"]
    R --> G2["Grouping 2: composable factor subgraphs"]
    G2 --> C["Compose interface"]
    C --> Z2["Output macro-tokens Z(l+1)"]

The tiling rule is therefore simply

Z^0 → Block 0 → Z^1 → Block 1 → Z^2 → Block 2 → Z^3 → ...

The next block does not need a special hidden representation. It receives the same kind of token contract produced by the previous block.

Downward targets travel through the retained handles and factor sensitivities described in Part II.

What scale does one block operate at?

A block does not correspond to a fixed semantic scale like “pixels,” “joints,” or “objects.” It operates at whatever scale its input tokens already represent.

The capacity knobs have different interpretations:

None of these automatically equals “more intelligence.” They are capacity knobs whose value has to be established empirically.

Rough compute scaling

Let:

• \(N\) be input-token count;
• \(H\) compatibility-head count;
• \(d_h\) per-head embedding width;
• \(k\) retained candidate neighbors per token;
• \(F\) retained predictive factors;
• \(m\) average participants per factor;
• \(d_f\) factor-state dimension;
• \(E_f\) retained factor-graph edges;
• \(C_{\mathrm{dyn}}\) cost of one shared dynamics evaluation.

Then the major pressures are roughly:

With \(N=1024\) and \(H=8\), a dense compatibility layer has

head-specific pair scores. Keeping only \(k=16\) edges per token gives an edge budget of about

That is 64 times fewer retained head-edges. It does not solve the candidate-retrieval problem by itself; an implementation still needs a scalable way to avoid materializing the full dense matrix when \(N\) becomes very large.

Pipeline latency

Depth differs from width because hierarchy stages have serial dependencies. If \(T_\ell\) is the forward latency of block \(\ell\), a simple worst-case upward traversal is

Within a layer, factor evaluations can often run in parallel. More factor slots therefore increase throughput demand more than unavoidable serial depth. Additional hierarchy levels, by contrast, add latency unless layers are pipelined, updated asynchronously, or run at different rates.

A practical embodied implementation would likely be multirate: low-level sensorimotor factors can update quickly, while object, skill, and task factors update more slowly and cache their most recent effective state.

What capabilities might scale?

The architecture makes testable scaling hypotheses:

• more width should support more simultaneous entities or sensorimotor variables;
• more factor slots should support more simultaneous and overlapping relationships;
• more factor-state capacity should support richer local laws and interfaces;
• more temporal memory should support delays and temporally extended factors;
• more useful depth should support relations among already-composed relations.

Depth only “wins” if deeper models improve predictive/control efficiency, transfer, planning horizon, or robustness at matched compute. If another level simply copies its input or adds latency, it has failed its architectural purpose.

10. A possible emergent hierarchy

Semantic levels should not be hard-coded, but a successful system might produce something resembling:

Some branches may stop earlier than others. The hierarchy should be driven by whether composition discovers a smaller useful interface, not by a fixed semantic depth target.

What Part III establishes

The full recursive representation path is now explicit:

TOKEN GRAPH
   ↓ compatibility
PAIRWISE PROPOSALS
   ↓ Grouping 1
PARTICIPANT SETS
   ↓ learn / validate
PREDICTIVE FACTOR GRAPH
   ↓ Grouping 2
COMPOSABLE FACTOR SUBGRAPHS
   ↓ Compose
MACRO-TOKEN GRAPH
   ↓ repeat

This is the piece that makes PCFH a hierarchy rather than merely a sparse relational dynamics model.

Part IV turns the remaining open choices into experiments and specifies what evidence should count against the idea.

Next: Part IV · Experiments and falsification

Part II · Dynamics, temporal basis, and control · approximately 10–12 minutes

Part I ended with a retained graph of predictive factors. Each factor has participant references, a current relational state \(r_f\), a learned transition law \(F_\theta\), prediction innovation \(\epsilon_f\), and uncertainty. This part asks how that same factor can support both forward prediction and downward control.

4. PID becomes a temporal basis, not the state representation

The original inspiration used proportional, integral, and derivative quantities heavily. The important refinement is that PID-like channels are temporal views of factor state or factor error, not the underlying representation itself.

For a retained factor \(f\), useful channels include:

Not every factor requires every channel at every timestep. In particular, goal channels exist only when the factor currently has a target.

Relational dynamics

A filtered derivative of relational state can expose ongoing transformation:

Numerical derivatives amplify noise, so \(\operatorname{LPF}_\tau\) denotes low-pass filtering over a characteristic timescale \(\tau\). A first-order filter can be

with

Several \(\tau\) values give the factor both fast and slow views of the same transformation.

Persistent model mismatch

A leaky integral of innovation can expose systematic prediction error:

Persistent innovation might indicate a payload change, friction mismatch, sensor drift, unmodeled force, or a bad factorization. This is not another predictive model; it is memory about how the existing factor model is failing.

Goal-directed control

A desired relational state \(r_f^*\) lives in the same factor-state space as the current relationship. For a hand-object factor, a target could correspond to a latent configuration such as “aligned and in stable contact” even if the coordinates are not explicitly named.

The instantaneous goal discrepancy is

PID-like temporal views can then be defined over that control discrepancy:

The interpretation is:

r       what relationship exists?
dr/dt   what transformation is occurring?

ε       what did the model fail to predict?
∫ε      what mismatch persists?

δ       how far are we from a requested relation?
∫δ      how long has that discrepancy persisted?
dδ/dt   are we making progress?

This creates a structured temporal basis without claiming that real-world dynamics are literally PID systems. Residual recurrent or state-space memory can still represent delays, hysteresis, oscillation, discontinuous contact, and other dynamics outside that basis.

Sensitivity belongs to the transition law

The cleanest forward/inverse connection comes from differentiating the learned transition law itself.

Around the current operating point, define

For small perturbations,

The two Jacobians have different jobs:

• \(A_f\) asks how a state perturbation changes the next factor state;
• \(B_f\) asks how an action perturbation changes the next factor state.

This avoids the ambiguous phrase “Jacobian of the factor structure.” We differentiate a concrete learned function with known input and output spaces.

For a one-step desired state \(r_f^*\), define

A local weighted least-squares action correction can be written

where \(\Lambda_f\) weights factor-state error directions and \(R_f\) regularizes action magnitude or cost.

This is one possible local controller, not a requirement that every factor perform a matrix inverse. In practice, the system may use iterative optimization, learned inverse models, model-predictive control, Jacobian-vector products, or lower-dimensional action ports.

The architectural claim is simpler:

The forward model learns how actions affect relational state; downward control can reuse that learned action sensitivity instead of learning a completely unrelated control representation.

The factor need not store a dense Jacobian

Materializing \(A_f\) and \(B_f\) for every factor at every step could be expensive. Many calculations only require directional products such as \(Bv\) or \(B^Tv\). Automatic differentiation can compute Jacobian-vector products and vector-Jacobian products without storing the entire matrix.

That lets sensitivity be a query against the factor model rather than permanent metadata embedded in every upward token.

7. Reference frames are learned because they simplify factors

A relational state is useful only relative to some coordinate system. PCFH therefore should not merely learn arbitrary embeddings; it should prefer local coordinates in which prediction and inverse control become simpler.

A useful frame should tend to make local factor dynamics:

• lower-dimensional;
• sparse;
• stable across context changes;
• compositional;
• well-conditioned for action inversion.

If a coordinate change is

then action sensitivity transforms as

Different coordinate systems can therefore describe the same underlying physical relationship while changing how easy that relationship is to model.

A candidate training objective might combine rollout accuracy, conditioning, sparsity, and transformation composition:

This is a research objective, not yet a settled loss. The underlying hypothesis is that useful reference frames emerge because they make recurring transformations easier to predict, compose, and invert.

This connects conceptually to group-structured representation learning, including Homomorphism Autoencoders, and to the broader Koopman-style idea of searching for coordinates that simplify nonlinear dynamics.

8. The hierarchy runs in both directions

The same factor hierarchy can be viewed as two traversals.

flowchart TD
    L0["Lower-layer tokens"] --> F["Predictive factor"]
    F --> M["Higher-layer macro-token"]
    M -. "desired macro-state" .-> F
    F -. "lower targets or action corrections" .-> L0

The upward direction asks:

Given the current lower-level state and actions, what higher-level relational consequences follow?

The downward direction asks:

Given a desired higher-level relation, what lower-level target or action change would move the system toward it?

The factor’s learned dynamics and sensitivity provide the bridge between the two.

Control does not have to replace the servo layer

At the physical boundary, conventional bounded control can remain in charge of fast actuator loops. For example,

followed by a hard safety projection

SafeProject is shorthand for a verified constraint layer: saturation, rate limiting, collision constraints, thermal limits, a control-barrier-function filter, constrained MPC, or another bounded mechanism appropriate to the system.

The learned hierarchy can operate at slower rates and provide setpoints, trajectories, feedforward terms, gain schedules, uncertainty estimates, and termination conditions rather than issuing raw high-frequency motor PWM directly.

A multirate hierarchy is therefore natural: fast local factors can update frequently while higher object, skill, or task factors update more slowly.

What Part II establishes

The predictive factor is now more than a latent relationship label:

current relation r_f
      ↓
learned transition F
      ↓
predicted next relation
      ↓
innovation ε_f

optional desired relation r*_f
      ↓
goal discrepancy δ_f
      ↓
action sensitivity B_f
      ↓
lower target / action correction

PID-like channels provide temporal context around those quantities without replacing the relational state. Reference-frame learning searches for coordinates that simplify the same predictive-control law.

What is still missing is abstraction: how a collection of already-learned factors becomes one token that can be handed to another identical block. That is Part III.

Next: Part III · Composition, recursive blocks, and scaling

Part I · From variables to predictive factors · approximately 12–15 minutes

The original version of this article tried to introduce the entire hierarchy, control theory, abstraction, scaling, and experiments in one pass. That made individual ideas understandable in isolation but obscured the architecture as a whole. This revision is a four-part series built around one typed dataflow.

The core proposal is a hierarchy in which each layer receives a set of tokens, discovers sparse dynamical relationships among them, learns predictive factors for those relationships, and then composes selected factor subgraphs into macro-tokens for the next layer. Prediction runs upward through the hierarchy; desired states can condition the same learned factors downward for control.

Architecture at a glance

There are two different grouping operations in PCFH, and keeping them separate makes the rest of the design much easier to follow.

flowchart LR
    Z["Layer l tokens"] --> H["Compatibility heads"]
    H --> E["Sparse pairwise proposal edges"]
    E --> G1["Grouping 1: participant sets"]
    G1 --> F["Predictive factor slots"]
    F --> R["Retained factor graph"]
    R --> G2["Grouping 2: composable factor subgraphs"]
    G2 --> C["Compose predictive-control interface"]
    C --> Z2["Layer l+1 macro-tokens"]

The stages have different meanings:

1. Compatibility asks which pairs of current-layer tokens appear dynamically related.
2. Participant grouping turns pairwise proposals into a hypothesis about which tokens should participate in one predictive factor.
3. Factor learning encodes the current relationship and learns how it changes.
4. Factor retention keeps useful hypotheses as nodes in a factor graph.
5. Composition grouping asks whether several retained factors form a subsystem whose internal detail can be hidden behind a smaller external interface.
6. Compose emits that subsystem as a fixed-interface macro-token to the next layer.

The same block can then repeat because the output is again a set of tokens.

This series treats the interfaces between those stages as the architecture. The exact sparse-routing algorithm, grouping algorithm, neural parameterization, and composition loss are still research choices to compare experimentally.

1. Three quantities that must remain separate

The first design mistake to avoid is treating every kind of error as the representation itself. A factor needs to distinguish the current relationship, failure of the dynamics model, and discrepancy from a desired relationship.

Relational state

A relational state is a learned description of the current configuration or interaction among a factor’s participants.

For a simple pair of current-layer tokens \(z_i\) and \(z_j\), a relation encoder could compute

For a factor \(f\) with participant set \(S_f\), the more general form is

where \(c_{f,t}\) is optional context such as a factor-type embedding or local reference frame.

The participant tokens still contain their live current values. The factor stores references or assignments to those tokens and derives \(r_{f,t}\) from their current contents. In other words, “participating variables” identifies which variables are involved; it does not duplicate their current state inside the factor specification.

The relational state itself is current runtime state. For a hand-object factor, components of \(r_f\) might eventually behave like relative pose, contact mode, slip state, or another latent coordinate that makes the joint dynamics simple. Those meanings are learned rather than hand-labelled.

Model innovation

A factor dynamics model predicts the next relational state:

where \(a_{f,t}\) is the action input visible to the factor.

When the next observation arrives, the relation encoder produces \(r_{f,t+1}\). The transition innovation is

So \(r_f\) answers what relationship exists now?, while \(\epsilon_f\) answers what did the learned law fail to predict?

Goal discrepancy

When a higher level requests a desired relational state \(r_f^*\), define

This is a control discrepancy, not a model error. A factor can be perfectly predicted while still being far from its goal, or exactly at its goal while the dynamics model is inaccurate.

The three streams are therefore:

r        current relational state
ε        prediction innovation
δ        goal discrepancy

Part II will use these three quantities to build the temporal and control side of the architecture.

2. The fundamental primitive: a learned dynamical factor

A predictive factor is a learned local dynamical model attached to a sparse set of participants. It is not an attention head, and it is not necessarily pairwise.

It helps to separate the factor’s relatively persistent specification from its time-varying state:

factor_spec := {
    participant_refs_or_assignments,
    relation_encoder,
    dynamics_model_or_type,
    local_reference_frame,
    exposed_action_ports,
    temporal_constants
}

factor_state[t] := {
    relational_state,
    predicted_next_relational_state,
    innovation,
    temporal_summaries,
    uncertainty
}

participant_refs_or_assignments can be hard indices, sparse soft assignments, or another routing representation. The live token contents remain in the current layer.

The predictive residual is just the transition error

The residual does not need to be an additional mysterious object. Given participant values at the next timestep,

Since \(g_\theta(z_{S_f,t+1})=r_{f,t+1}\),

A Gaussian-style transition likelihood can then be written as

Here \(\Lambda_f\) is an information or precision matrix. It is not the learned relationship and it is not multiplied by a Jacobian because of some special architectural rule. It simply weights residual directions according to uncertainty and scale. If the residual coordinates carry physical units, the precision entries carry the corresponding inverse-squared units so the exponent is dimensionless.

The learned relationship is represented by the current factor state \(r_f\) together with the transition law \(F_\theta\).

A factor slot is an instance, not a new network

A practical implementation would probably use a bounded pool of factor slots. Each active slot contains one factor instance: its participant assignments, current relational state, uncertainty, and persistent bookkeeping. Shared neural weights can implement the relation encoder and dynamics model across many slots.

That distinction matters for scaling. If every pair of tokens instantiated an independent neural network, the design would be dead on arrival. The intended structure is sparse routing into shared learned machinery.

3. Attention discovers participation; factors learn laws

The compatibility mechanism is a proposal system. It does not itself learn the full transformation law.

For candidate current-layer tokens \(i\) and \(j\), head \(h\) can produce a pairwise score such as

The symbols are:

• \(i,j\): current-layer token indices;
• \(h\): compatibility-head index;
• \(q_{i,h}=W_h^Qz_{i,t}\): query representation of token \(i\);
• \(k_{j,h}=W_h^Kz_{j,t}\): key representation of token \(j\);
• \(M_h\): a learned bilinear compatibility matrix;
• \(\rho_{ij,t}\): cheap pairwise features available before a full factor exists;
• \(\dot\rho_{ij,t}\): optional recent change in those pairwise features;
• \(a_t\): relevant action context;
• \(\Delta t\): elapsed time;
• \(m_{ij,t-1}\): optional cached summary if a previous retained factor involved the pair;
• \(b_h\): a learned auxiliary scoring function;
• \(C_{ij,h}\): the final proposal score.

Crucially, the score does not require computing a full predictive factor for every possible pair. Compatibility is the cheaper routing stage that decides which interactions deserve more compute.

What is a sparse compatibility graph?

Treat the current tokens as graph nodes. A dense proposal mechanism can conceptually score all pairs, but only a bounded subset of edges is retained.

flowchart LR
    A["Token A"] --- B["Token B"]
    B --- C["Token C"]
    B --- D["Token D"]
    E["Token E"] --- F["Token F"]

A retained edge means roughly:

These two tokens look worth testing as participants in a common dynamical factor.

It does not yet mean that they belong to one object, one cause, or one permanent factor.

With multiple heads, the graph is really a sparse multi-relational proposal graph: the same pair can receive different scores under different learned notions of compatibility.

How compatibility scores become predictive factors

The first half of the block is:

flowchart TD
    A["Current-layer tokens"] --> B["Compatibility heads"]
    B --> C["Sparse pairwise proposal graph"]
    C --> D["Grouping 1: participant-set hypotheses"]
    A --> E["Shared relation encoder"]
    D --> E
    E --> F["Predictive factor slots"]
    F --> G["Shared or type-conditioned dynamics"]
    G --> H["Prediction, innovation, uncertainty"]
    H --> I["Predictive and interventional validation"]
    I --> J["Retained factor graph"]

This diagram intentionally stops at the retained factor graph. The second grouping operation—composition of already-learned factors into a macro-subsystem—happens later and is the subject of Part III.

A candidate factor can be created in seven conceptual steps:

1. Score edges. Compatibility heads compute \(C_{ij,h}\).
2. Sparsify. Keep a bounded set of strong candidate edges.
3. Participant-group proposal. Turn pairwise edges into one or more candidate participant sets.
4. Allocate factor slots. Each group hypothesis receives an active factor instance.
5. Encode and predict. The factor encoder computes \(r_f\); shared dynamics predict \(\hat r_{f,t+1}\).
6. Validate. Prediction, persistence, intervention response, uncertainty, and control usefulness determine whether the hypothesis is worth retaining.
7. Build the retained factor graph. Useful predictive factors become nodes for later message passing and composition.

The expensive learned dynamics therefore run on sparse factor hypotheses rather than all \(N^2\) token pairs.

Does A–B plus B–C imply one A–B–C factor?

This question refers specifically to pairwise compatibility edges in the first grouping stage.

Suppose a compatibility head gives a high score to A–B and another high score to B–C. Does that mean the participant-grouping algorithm should create one factor over \({A,B,C}\)?

No—not automatically. Pairwise compatibility is not transitive.

A–B and B–C could arise because:

• all three variables really participate in one coherent transformation;
• B participates in two different transformations;
• A and C share only an indirect path through B;
• one edge is causal and the other merely correlated;
• two compatibility heads are expressing different relationship families.

Ordinary connected components therefore make a useful proposal heuristic, but not a sufficient definition of factor membership. A long chain of locally strong edges could otherwise collapse into one enormous factor.

Possible participant-grouping algorithms include:

• per-head connected components followed by a group-level predictive test;
• seed-and-grow grouping where a token is added only when joint prediction improves;
• learned factor slots with sparse token-to-slot assignments, related in spirit to Slot Attention;
• hyperedge proposal networks that score sets directly;
• graph clustering constrained by intervention consistency.

The acceptance criterion should be closer to:

Does this proposed participant set admit a coherent, reusable dynamical law that predicts or controls better than the relevant smaller alternatives?

That question is deliberately left open for experiment rather than smuggled into “connected component” as an architectural assumption.

There is a similar but separate overlap question later when already-learned factor nodes are grouped for composition. Part III treats that as Grouping 2 rather than conflating it with compatibility-edge grouping.

Interactive PCFH block explorer

The two views below are a toy visualization, not a learned model. They are intended to make the data types concrete.

The compatibility heatmap shows the same six tokens under two different heads. Switch heads to see that a pair can be important under one relationship family and unimportant under another.

The second visualization shows the type flow through one upward block. Sankey width is only illustrative; it should not be interpreted as a probability or conserved physical quantity.

The important transition is:

raw/current tokens
    → pairwise proposals
    → participant groups
    → predictive factor slots
    → retained factor graph
    → composition groups
    → macro-token interfaces

That is the complete upward representational path. Part II adds the factor dynamics and downward control path; Part III explains the second grouping stage and why its output can tile recursively.

What Part I establishes

The architecture now has a precise first half:

• token contents are distinct from participant references;
• relational state is current factor state, not prediction error;
• model innovation and goal discrepancy remain separate quantities;
• compatibility heads create pairwise proposals;
• Grouping 1 converts those edges into participant-set hypotheses;
• factor slots learn and validate local dynamical laws;
• validated factors form a retained factor graph;
• the second grouping operation acts on that factor graph, not on raw compatibility edges.

This makes the core proposal close to a typed pipeline rather than a collection of loosely connected metaphors.

Neighboring ideas

The factor-discovery side is related to Neural Relational Inference, which learns interaction graphs and dynamics from trajectories. Learned slot-based grouping is also relevant as a candidate implementation family; Slot Attention is one representative example. PCFH’s additional hypothesis is that retained dynamical factors can be recursively composed into predictive-control interfaces and traversed in both directions.

Next: Part II · Dynamics, temporal basis, and control