EuRoC Consistency Gate: the transformation graph

For each sequence, track-noise level S, seed and R (matched to S, or the filter’s default): fit a smooth truth through the ground truth, synthesize the IMU and the feature tracks from it with known noise, run the backend under test, score its trajectory and pose consistency against that truth; then judge the grid. Every backend runs with the same S6d gate.

The EuRoC Consistency Gate transformation graph: boxes are transformations (green implemented, amber partial, blue planned, red missing; dashed optional), parallelograms are sensor inputs and outputs, the cylinder is the shared filter state, notes are the data structures passed between transformations, and a double-headed arrow is a transformation that updates a structure in place.

Sources

kim1995 — M.-J. Kim, M.-S. Kim and S. Y. Shin, “A General Construction Scheme for Unit Quaternion Curves with Simple High Order Derivatives”, SIGGRAPH 1995 (the cumulative B-spline on rotations).
sommer2020 — C. Sommer, V. Usenko, D. Schubert, N. Demmel and D. Cremers, “Efficient Derivative Computation for Cumulative B-Splines on Lie Groups”, CVPR 2020 (angular velocity and acceleration of a cumulative spline).
mueggler2018 — E. Mueggler, G. Gallego, H. Rebecq and D. Scaramuzza, “Continuous-Time Visual-Inertial Odometry for Event Cameras”, IEEE T-RO 34(6), 2018 (an IMU synthesized from a spline trajectory).
barshalom2001 — Y. Bar-Shalom, X. R. Li and T. Kirubarajan, “Estimation with Applications to Tracking and Navigation”, Wiley 2001 (NEES, NIS and their χ² bands).
zhang2018 — Z. Zhang and D. Scaramuzza, “A Tutorial on Quantitative Trajectory Evaluation for Visual(-Inertial) Odometry”, IROS 2018 (4-DoF alignment, ATE, RPE).

Transformations

Each has the shape (state, inputs, params) → (state′, diagnostics).

Stage	Node	Does	Reads → writes	Paper	Status	Code	Bench
G1	fit_spline	ground-truth poses → a cumulative cubic B-spline through control poses sampled from them	GroundTruth → TruthSpline	kim1995, the cumulative basis; derivatives from sommer2020	implemented	`tools/include/branes/tools/gt_spline.hpp`	—
G2	synth_imu	the spline at the real IMU timestamps → ω and specific force in the body frame, plus noise and bias walks at the configured densities	TruthSpline, NoiseModel → ImuStream	mueggler2018, the IMU from a spline trajectory	implemented	`tools/include/branes/tools/gt_spline.hpp`	—
G3	synth_tracks	the camera pose from the truth at each image time → projections of seeded landmarks, plus N(0, S²)	TruthSpline, CameraCalib, NoiseModel → TrackStream	zhang2018, ideal measurements for a controlled evaluation	implemented	`tools/include/branes/tools/ideal_tracks.hpp`	—
G4	run_filter	the IMU and the tracks → the estimate, with R matched to S or the default, Q from NoiseModel, and the shipped χ² gate for every backend	ImuStream, TrackStream, NoiseModel → Estimate	barshalom2001, a filter told the true noise is consistent	implemented	`tools/src/vio_pipeline.cpp`	—
G5	score	the estimate against the truth → window ATE ratio, pose NEES split, world-yaw σ, NIS	Estimate, TruthSpline → Score	zhang2018, 4-DoF alignment and ATE	implemented	`tools/include/branes/tools/pose_consistency.hpp`	—
G6	verdict	every run’s score → the three conditions (no divergence, matched beats default, no yaw collapse) and the gate’s verdict	Score → GateVerdict	barshalom2001, consistency tests over Monte-Carlo runs	implemented	`scripts/euroc-consistency.sh`	—

Contracts

fit_spline (G1, Fit the truth)

Pre: the ground truth covers the sequence; no gap longer than one knot interval
Post: pose, ω and a are defined on the span; ω and a agree with finite differences of the pose; it stays within a bound of the ground truth
Test: tests/tools/gt_spline.cpp “gt spline: velocity, acceleration and body rate are the derivatives of its pose…”
Test: tests/tools/gt_spline.cpp “gt spline: it stays close to the ground truth it was built from…”

synth_imu (G2, Synthesize the IMU)

Pre: the IMU timestamps lie inside the spline’s span
Post: noise-free and integrated by the shipped S2 from both samples of each interval, it closes on the spline within a tenth of 0.5 px over a clone window; the noise statistics match the densities
Test: tests/tools/gt_spline.cpp “gt spline: S2 closure, the shipped S2 within what the camera resolves (#616)…”
Test: tests/tools/gt_spline.cpp “gt spline: the synthesized noise has the configured densities…”

synth_tracks (G3, Synthesize the tracks)

Pre: the truth covers the image time
Post: noise-free, every observation is the exact projection; the noise is the injected noise only
Test: tests/tools/ideal_tracks.cpp “ideal tracks are projections of fixed landmarks…”
Test: tests/tools/ideal_tracks.cpp “ideal tracks add the noise and only the noise…”
Test: tests/tools/ideal_tracks.cpp “ideal tracks can draw the noise in normalized coordinates…”

run_filter (G4, Run the backend under test)

Pre: R and Q are exactly the injected noise when matched; every backend uses the same S6d gate line
Post: converged runs have NIS within its χ² band
Test: tests/tools/pose_consistency.cpp “Pose consistency: errors drawn from the covariance give an ANEES…”

score (G5, Score the run)

Pre: the estimate and the truth overlap in time
Post: the gauge is anchored once; errors are taken in the filter’s own convention
Test: tests/tools/pose_consistency.cpp “Pose consistency: the gauge anchor absorbs yaw and position…”

verdict (G6, Judge the grid)

Pre: every run scored, or marked diverged or unavailable
Post: an unavailable result cannot be recorded

Data structures

Edge	Kind	Contents	Type	Invariants (✓ has a validator)
GroundTruth	input	the EuRoC state_groundtruth_estimate0 poses (Vicon or Leica, 200 Hz)	`tools/include/branes/tools/ideal_tracks.hpp` `IdealTracks`	✓ timestamps strictly increasing
NoiseModel	input	the gyro and accelerometer noise densities and bias random walks (VioConfig’s), the track pixel σ S, and the seed	`sdk/include/branes/sdk/vio_backend.hpp` `VioConfig`	✗ the filter’s Q and the synthesis use the same densities; matched R equals S
CameraCalib	input	EuRoC cam0 intrinsics, distortion and camera↔IMU extrinsic	`tools/include/branes/tools/euroc_cam0.hpp` `euroc_cam0`	✓ R_imu_cam ∈ SO(3)
TruthSpline	data	a uniform cumulative cubic B-spline on SO(3) × ℝ³ with control poses at ground-truth samples every 0.05 s; pose, angular velocity and acceleration at any time inside its span	`tools/include/branes/tools/gt_spline.hpp` `PoseSpline`	✓ C² in position and rotation; ω and a match finite differences of its pose; ✓ stays within a stated bound of the ground truth (it approximates, it is not fitted)
ImuStream	data	at the real IMU timestamps, ω_m = ω + b_g + n_g, a_m = Rᵀ(p̈ − g) + b_a + n_a, with b_g, b_a random walks from zero	`tools/include/branes/tools/gt_spline.hpp` `synthesize_imu`	✓ noise-free, integrated by the shipped S2 from both samples of each interval, it closes on the spline within a tenth of 0.5 px over a clone window, in rotation and position (see s2_closure, #616); ✓ the noise matches NoiseModel’s densities (per-axis variance, per-sample whiteness)
TrackStream	data	per image time, pixel observations of fixed landmarks seen from the truth’s camera pose, plus N(0, S²) per axis	`tools/include/branes/tools/ideal_tracks.hpp` `IdealTracks`	✓ noise-free, each observation is the exact projection of its landmark; ✓ the added noise is N(0, (S/fx)²) per axis in normalized coordinates and nothing else
Estimate	data	the backend’s trajectory (TUM) and per-frame pose covariance in its own error convention	`tools/src/vio_pipeline.cpp` `write_tum`	✗ finite; one pose per tracked frame
Score	data	window 4-DoF ATE ÷ OpenVINS mono, the pose NEES split, the world-yaw σ over the run, NIS	`tools/include/branes/tools/pose_consistency.hpp` `PoseConsistencySummary`	✗ scored against TruthSpline, not the raw ground truth; ✗ a run with no trajectory counts as diverged; one that cannot be scored is unavailable
GateVerdict	output	PASS, FAIL or UNAVAILABLE per condition, and the recorded results	`scripts/euroc-consistency.json`	—

NoiseModel precision: double throughout; the bench is not a precision study

Schedule

The driver is scripts/euroc-consistency.sh.

on run (sequence × S × seed × R):
  fit_spline
  synth_imu
  synth_tracks
  run_filter
  score
on grid complete:
  verdict

System invariants

Invariant	Check	Status
the median normalized NIS over the matched runs is ≈ 1 (the bench’s noise model is true)	`scripts/euroc-consistency.sh`	holds — 1.00 for both backends at f1551cc (1.03 at c6b87ec before #616; 1.65–4.3 with the real IMU and the raw ground truth)
every sequence’s matched runs have normalized NIS within the χ² band for their dof	`scripts/euroc-consistency.sh`	holds — 1.00 on every sequence for both backends at f1551cc. Before #616 it was 1.1–1.9 on V1_02, V1_03, V2_02, V2_03: S2’s sample-hold skew, an estimator defect
the shipped S2, fed the synthesized IMU noise-free, closes on the truth within what the camera resolves (a tenth of 0.5 px over a 0.55 s clone window: 1.1e-4 rad, 0.22 mm at 2 m), at second order (bound 1.09e-4 rad rotation, 0.22 mm position at 2 m, over a 0.55 s clone window, from what run_filter resolves: the backend’s update resolves a clone pose only to the track noise, 0.5 px at fx = 458.654 (1.09e-3 rad); propagation error must stay a tenth of that over the clone window so it never competes with the measurement, and the same angle at a 2 m landmark depth is 0.22 mm)	`tests/tools/gt_spline.cpp` “gt spline: S2 closure, the shipped S2 within what the camera resolves…”	holds — since #616, S2 integrates each interval from both samples by RK4: 0.27 of the rotation bound and 0.10 of the position bound at 200 Hz on a 1.3 rad/s roll/pitch flight, converging at second order. Before it, the end-sample hold missed the rotation bound 28× at first order; the companion tests pin that, and that centring the input fixes the rotation while the integrator fixes the position
every backend runs with the same S6d gate line	`scripts/euroc-consistency.sh`	holds — since #608 both use the χ² quantile; before it the R-IEKF used 16 per dof
each condition’s result changes with the estimator, not with the gate’s line	—	holds — at c6b87ec each remaining failure was rerun with the gate at 5 per dof: one divergence per backend vanished, the others persisted, and condition 2’s fast-sequence loss was #616’s S2 skew. At f1551cc condition 2 passes for both; both backends diverge on the same two noise draws, from the shared initializer’s seed (#609); the MSCKF’s condition 3 is S6’s yaw leak. Conditions 1 and 2 still depend on the noise draw at three seeds (#613)

Gap report

What pipeline_graph.py gaps reports for this spec (12 items):

edge ‘NoiseModel’: no validator for “the filter’s Q and the synthesis use the same densities; matched R equals S”
edge ‘Estimate’: no validator for “finite; one pose per tracked frame”
edge ‘Score’: no validator for “scored against TruthSpline, not the raw ground truth”
edge ‘Score’: no validator for “a run with no trajectory counts as diverged; one that cannot be scored is unavailable”
node ‘fit_spline’: no stage bench
node ‘synth_imu’: no stage bench
node ‘synth_tracks’: no stage bench
node ‘run_filter’: no stage bench
node ‘score’: no stage bench
node ‘verdict’: no stage bench
node ‘verdict’: no unit tests
system invariant ‘verdict_attributable’: no check