Visual-Inertial Odometry Overview

Epic E3 (Phase 3) delivered a complete Multi-State Constraint Kalman Filter (MSCKF) visual-inertial odometry pipeline — clean-room from the papers (Mourikis & Roumeliotis 2007, Forster et al., VINS-Mono), no GPL source contact.

The pipeline

  images ─▶ Front End ──┐
                        │ FrontendObservation (feature id, pixel u,v)
  IMU ────────────────┐ │
                      ▼ ▼
              ┌──────────────────────┐
              │     VioEstimator     │  lifecycle-gated façade
              └───────────┬──────────┘
                          │ pixels → bearings (camera model)
                          ▼
              ┌──────────────────────┐
              │   MsckfBackend       │
              │  • IMU init          │  static / dynamic
              │  • propagate         │  per-sample mean + covariance
              │  • augment / margin. │  sliding window of cloned poses
              │  • camera update     │  null-space + Mahalanobis gate
              └───────────┬──────────┘
                          ▼
                   NavState (T_world_imu, velocity, stamp)

The components

Component	Page	Issue
Visual front end (FAST + KLT + pyramid)	Front End	#37, #38, #49, #48
IMU preintegration + initialization	IMU	#40, #42
MSCKF state / propagator / clones	MSCKF State	#43
Camera updaters (null-space)	Camera Updaters	#44
MSCKF backend (the wiring)	MSCKF Backend	#35
Sliding-window backend skeleton	MSCKF Backend	#36
Top-level estimator API	VioEstimator	#45

Why MSCKF

MSCKF keeps a sliding window of past camera poses in the filter state and uses each tracked feature as a multi-view geometric constraint that is marginalized (removed from the state) the moment it is consumed — via a left-null-space projection of its measurement Jacobian. The feature is never added to the state vector, so the filter cost stays bounded and roughly linear in the window size, which is exactly the property a real-time embedded estimator needs.

How it was validated

• Stability: the camera-update path holds the covariance positive-semidefinite and the innovation well-conditioned over 1000 consecutive updates.
• Accuracy: ATE/RPE against ground truth via an EuRoC replay harness.
• Latency: a per-frame budget on feed_image.

See Benchmarks & Validation for the numbers and methodology.

What’s deferred

A square-root covariance variant of the backend (numerically superior conditioning via a QR-form update) was split out as a post-MVP follow-up (#187) rather than shipping a dead static_assert-guarded template knob. The current backend is the full-covariance form.