The Affine Ceiling: Sharp on Affine Motion, Blind at Depth
Formal motion estimation is dismissed as good only for small motion. A benchmark against pixel-exact ground truth shows that is false: large scaling and rotation are solved to sub-pixel accuracy. The real ceiling is the affine assumption, and it collapses at exactly the depth boundary that 3D reconstruction is made of.
Motion estimation is the front end of 3D reconstruction. Everything downstream inherits its errors, which is why it is worth knowing exactly where it breaks rather than roughly how well it scores.
There are two ways to estimate motion today, and neither is currently usable for safety-critical 3D.
Learned monocular models are advancing fast. Robbyant's LingBot-Map reconstructs 3D scenes from a single RGB camera in real time, streaming and end to end at around 20 FPS, and it is open source. That is impressive engineering, and formal hand-derived algorithms do not currently keep up on raw capability. But it is a trained neural network, probabilistic by construction: no error bound, no repeatable output, no provable worst case. For a photo app that is fine. Under ISO 26262, DO-178C or IEC 62304 it is disqualifying.
So I benchmarked the other side properly. Formal, non-probabilistic, phase-based estimation, derived from signal theory rather than trained from data. Every output reproducible and analyzable. The open question there is not whether you can trust it. It is whether it is accurate enough for 3D.
The folklore is wrong
Phase-based estimation reads displacement from the phase shift of a band-pass filter. Phase is unique only within one turn, so beyond half a filter wavelength it wraps and the answer is silently wrong. That is a real ceiling, and on a single scale it is small.
It is also not fundamental. Downsampling halves the motion. At a coarse enough pyramid level even a large displacement drops under the wall and can be read without wrapping. Warp the finer level by that estimate and only a small residual remains, read the same way, down to full resolution. The wall never disappears, every level simply stays below it.
I benchmarked three implementations against pixel-exact ground truth: single-scale Fleet-Jepson, my own multi-scale reimplementation, and a faithful port of the 2026 affine multi-scale method of Li et al. On a known scaling, which is exactly what a fronto-parallel plane does under a Z-dolly, the median endpoint error looks like this:
| Max displacement | Fleet (single-scale) | Reimplementation | Affine M-PME |
|---|---|---|---|
| 4 px | 0.41 | 0.11 | 0.03 |
| 11 px | 1.91 | 0.20 | 0.12 |
| 16 px | 8.93 | 0.27 | 0.25 |
| 29 px | 16.82 | 1.57 | 1.14 |
Single-scale degrades linearly with displacement, exactly as the theory predicts. Both multi-scale methods stay sub-pixel across the whole sweep. The claim that formal methods only handle small motion has not been true for years.
The real ceiling is the assumption, not the magnitude
Look at why the affine method wins: scale and rotation are affine. A six-parameter affine model represents them with no residual, so the method is not approximating anything. It is solving the exact problem it was given.
A real scene is not a single plane. Move the camera along Z over a near object in front of a far background and the optical flow becomes depth-dependent, with a discontinuity at the object's edge. That violates the locally-affine assumption every one of these methods rests on.
On the smooth, locally-affine regions all three do fine, with a median endpoint error between 0.13 and 0.70 px. At the depth edge the error jumps by a factor of four to nine. And there is the result I did not expect: the faithful affine method is worse at the edge, at 1.17 px, than my simpler per-pixel reimplementation at 0.52 px. Its window averages across the discontinuity, which is precisely where the 3D structure lives. The more principled model loses at the only place that matters.
A blurred edge is not a small error
Invert the recovered flow back into depth and the failure becomes obvious. Fleet returns noise. The reimplementation frays. The affine method returns a soft, blurred step where 3D needs a crisp one.
A depth map with soft, wandering boundaries produces a 3D model with soft, wandering surfaces. Walls bleed into floors. Objects fuse with their background. For a robot deciding where the table ends, or a vehicle deciding where the curb is, a blurred boundary is not a small error. It is the wrong answer at the one place where accuracy is non-negotiable.
The gap
The two halves of this benchmark bracket the state of formal motion estimation. Sharp on affine motion, at large magnitude, genuinely solved. Blind at the depth boundary, which for 3D reconstruction is the whole point.
That is the gap. The probabilistic models that handle real scenes cannot be certified. The formal methods that can be certified cannot yet resolve depth. Neither is usable for safety-critical 3D as it stands, and pretending otherwise is how programs discover the problem during assessment instead of during design.
It is the same structural pattern I keep finding in production perception stacks: output that looks plausible in aggregate and fails exactly where the decision gets made. A snapshot machine has the same shape of problem in the time dimension that this has in the spatial one.
We are building a formal, non-probabilistic method designed to stay sharp exactly where these blur: crisp at the depth boundary, accurate enough per pixel to drive 3D reconstruction. This benchmark is the baseline it will be measured against, which is the point of publishing it before there is anything to claim.
The working note, the benchmark code and every figure are reproducible from source: github.com/XIXUM/implicit-kalman
