Diagnosing and Fixing Encoder Collapse in Tiny JEPA Code Transition Models

Tiny JEPA-style code transition models trained on chains of Git edits exhibit a mysterious ${\sim}700$-step training ceiling. We identify the failure mode — target encoder collapse under fast EMA tracking — and diagnose it with a cheap copy-baseline indicator, $\cos\!\bigl(f_{\bar\theta}(s_t),\,f_{\bar\theta}(s_{t+1})\bigr)$, that climbs to ${\sim}0.999$ well before standard loss curves degrade.

The fix is to hold the target encoder approximately static; we verify this with two equivalent settings — near-frozen EMA decay $0.99999$ (default, three seeds) and fully frozen decay $1.0$ (a random-initialised target that never updates, two seeds) — that agree on validation transition cosine to within $\pm 0.002$, so target staticity is the lever, not a specific EMA decay value. With the fix in place, a $1.1$ M-parameter $128$-dimensional model trains stably for $15{,}000$ steps to a mean-of-last-five validation transition cosine of $0.9898 \pm 0.0006$ and performs compositional three-step prediction with per-step delta cosines ${\approx}\,0.98$, more than $0.9$ above every delta-space null we evaluate; the same property transfers to edit chains from nine held-out Python repositories.

A secondary retrieval experiment shows the model is competitive at $112\times$ fewer parameters on in-distribution CommitPackFT edit retrieval and on a $3{,}998$-pair twenty-repo leave-one-repo-out sweep, while losing cleanly to modern dense encoders out-of-distribution. A zero-dependency Rust inference engine with four-level weight quantisation compresses the deployed model to $0.6$ MB and a $2.6$ MB in-browser WebAssembly bundle.