Digital Twin Perception

A perception stack that runs a fresh classification on every frame is a snapshot machine. It has no memory of what it saw a moment ago, so identity breaks when one object passes behind another, static scene and moving traffic are guessed apart frame by frame, and a single sensor's blind spot becomes the whole system's blind spot. This project treats perception as the opposite: not a stream of independent snapshots, but one driving context that persists and accumulates evidence over time.

It is the temporal, three-dimensional extension of the single-frame idea in Hierarchical Perception: keep the taxonomy and the never-drop-a-novel-object principle, but ground every detection in metric geometry and carry it across time.

One context from many sensors

The core is a hierarchical sensor fusion. LiDAR contributes metric 3D geometry and occupied volume, a monocular camera contributes depth and a 2D detector contributes class and image box. Rather than trusting any one of them, the fusion associates the 2D and 3D cues per object and lifts them into a single 3D hypothesis, which a 3D-consistency classifier then validates against physical size and volume.

That validation is where redundancy pays off. A pedestrian roughly 1.8 m tall with real 3D volume is accepted. An oversized statue is rejected as too large to be a person. A flat billboard is rejected because it has no volume at all, the classic flying-car false positive stated in metric terms. Probabilistic paths (the learned detectors) and deterministic paths (the geometric and semantic constraints) verify each other, so agreement between two independent model families becomes genuine evidence rather than one confident guess.

Static scene, moving objects

The component in focus right now is LiDAR point cloud consolidation: fusing returns over time into a coherent scene, then separating the static environment from the things that move through it, using the Doppler signal to tell them apart. Each moving object is carried as a 3D bounding box with a velocity vector, an observed past trajectory, a predicted future path, and an explicit uncertainty around that prediction. A vehicle and a pedestrian are the same problem at different scales, and both are tracked past to future rather than re-guessed each frame.

The consolidated result feeds a digital twin: a virtual 3D environment that runs as a co-simulation of the real scene, holding every validated traffic participant in one place. Perception stops being a per-frame label and becomes a running model of the world the vehicle is driving through.

The through-line

The design goal is a Physical AI that builds a driving context out of several sensor sources, some probabilistic and some deterministic, that all cross-check each other through redundancy. It expresses that context as formal and semantic constraints, so an implausible detection is not just improbable but provably out of context.

The intellectual origin is the essay Your perception stack is a snapshot machine, and the perception principle underneath it runs through F. Schaller, "The Role of Semantic Models in Constraining Pattern Recognition in Modern AI Systems" (Intelligent Environments 2025, doi:10.3233/AISE250023), "Hierarchical Taxonomic Abstraction for the Safe Handling of Novel Objects in Autonomous Driving Perception" (Zenodo, 2026, doi:10.5281/zenodo.21593472), and its open-world successor "Open-World Hierarchical Perception: Taxonomic Abstraction over Class-Agnostic Proposals for the Safe Handling of Out-of-Vocabulary Road Objects" (arXiv:2608.07577, 2026), published as the HOWC model at huggingface.co/freshNfunky/howc.

This is an active research build, not a finished product, and the source is kept private. It is shown here as a direction of work, so the honest framing is a capability being built rather than a benchmark being claimed.