VIO from First Principles: Failure Modes and the Regression Suite

This section tells how cortex got its visual-inertial odometry working: every defect, investigation and fix along the way, as one story. It is written for an engineer who has to build a VIO pipeline from first principles, or certify a new one. Most of what went wrong here goes wrong in any MSCKF-style pipeline, and a unit-test suite that passes doesn’t catch it.

The storyline

What broke, in the order we found it: what each failure looked like, the measurement that decided its cause, and how it was fixed.

Diagnostic signatures

A symptom → cause table and twenty-two pitfalls, so the next person recognizes a failure on sight.

Aligning sensor samples

A failure mode of every fusion pipeline: integrating a sample over an interval it doesn’t represent, how it hid here for five months, and a checklist.

The regression suite

Every functional test a VIO pipeline must pass, with its pass criterion, the test that implements it here, and whether it is locked in CI.

The main lesson

Over about six months, five results that looked solid turned out to be artifacts of the test setup: a compiler “bug” that was an unsequenced random draw, a filter “inconsistency” that was a velocity step in the synthetic world, a “consistent” control that fed the filter exact sensors, a consistency verdict whose “true noise” left out the ground truth’s own error, and an S2 closure test whose threshold was fitted to the very defect it should have caught. Each time, the filter or the test was trusted first. The storyline states every revision next to the claim it revises, because how a wrong conclusion was reached is as useful as the right one.

The other lessons, in one line each:

Synthetic unit tests are necessary and nowhere near sufficient. When the VIO epic closed, every part was unit-tested and nothing had run on real data. The first real run was off by 26 km.
ATE hides over-confidence. A filter can be accurate to 30 cm with a covariance a hundred times too small. Report NEES and NIS beside ATE, always.
Measure before fixing; ship candidates default-off. Several textbook fixes made things worse once they were measured end to end.
Know what time each sample stands for. Holding an IMU sample over an interval it doesn’t represent skewed the IMU 2.5 ms against the camera in both backends, and it passed for ground-truth error and over-confidence until a continuous truth exposed it (Aligning Sensor Samples).

Vocabulary

The stages follow the canonical pipeline, whose contracts are described one page per stage in the VIO Pipeline section.

Stage	Role	Stage	Role
S0	sensor models	S6a	Jacobians
S1	initialization	S6b	null-space projection
S2	IMU propagation	S6c	compression
S3	clone augmentation	S6d	gating
S4	visual front end	S6e	EKF update
S5	triangulation	S7/S8	SLAM features, zero-velocity update (not implemented)
S9	marginalization	S10	online calibration

There are two backends:

Body-frame: MsckfBackend, an MSCKF whose error state is expressed in the body frame.
R-IEKF: MsckfInvariantBackend / InvariantVioBackend, a right-invariant EKF on SE₂(3).

The metrics (ATE, RPE, NEES, NIS) are defined on Reading the Metrics.

Where it stands (2026-11-02)

Area	State
Body-frame filter	Accurate on EuRoC (ATE 0.18–0.73 m) but over-confident there (NEES/dof 43–436, measured before the #481 gate) and on the synthetic world under matched noise (median pose NEES 34 against 6). The mechanism is the yaw gauge leak in S2/S3; its share of the EuRoC NEES isn’t attributed yet. It stays the EuRoC accuracy reference.
R-IEKF	Consistent on the synthetic world under noise: normalized full NEES 0.56–1.84 over 21 seeds at the default onset ( #547 ), and 0.39–1.88 over 21 seeds with a slow 2 s start ( #548 ). Less accurate on EuRoC than the body-frame filter (V1_01 1.31 m, #429 , against 0.22 m under the χ² gate, #572 ), and its EuRoC NEES verdict is open. Closing that gap is the central open problem.
Initialization	Robust with static init. Dynamic init is limited by excitation and has open SfM defects.
Real time	Real-time factor about 0.10×.