EuRoC Baseline vs OpenVINS

The goal for the VIO pipeline is accuracy competitive with OpenVINS on EuRoC. Published tables use their own sequence subsets, start offsets, alignments and evaluators, so a gap between our number and a paper’s mixes the estimator with the protocol. This baseline runs both estimators on the same data, through the same evaluator, so the gap is the estimator’s.

The numbers come from scripts/euroc-baseline.json, which scripts/euroc-baseline.sh reruns and compares against (see the regression benchmark).

Protocol

• Data: the 11 EuRoC MAV sequences, cam0 and imu0, ground truth from state_groundtruth_estimate0.

• cortex (commit f1551cc, 2026-11-09, with S2 integrating each IMU interval from both of its samples, #616; every sequence changed from the first baseline at 348c774, see below): vio_pipeline --source euroc with its defaults, monocular, the MSCKF with double stages and a full covariance. Every run starts at the first frame.

• OpenVINS v2.7: the unmodified upstream binary, run outside every Branes repository with its upstream defaults, as ADR 0002’s 2026-11-06 amendment allows. Three run modes: monocular and stereo from the upstream start offsets, and monocular from the first frame. Only the aggregate metrics below are recorded. No test or CI job runs it or reads its output.

• Evaluator: tools/traj_eval scores every trajectory the same way. It matches each estimated pose to the nearest ground-truth sample within 20 ms and reports:

  • ATE 4-DoF: translational RMSE after a position + yaw alignment, the four directions a VIO cannot observe. This is the headline number.
  • ATE SE3: the same after a rigid Horn alignment (vio_pipeline’s ATE). It also forgives roll and pitch errors, which a VIO can observe.
  • RPE: the error of the relative motion over sub-trajectories of 8, 16, 24, 32 and 40 m of ground-truth travel. It is reported as the mean translation (m) and rotation (deg) error per length, then averaged over the lengths.

  The metrics are implemented in branes::sdk::eval from their definitions (Horn 1987 for the rigid alignment, and the closed-form yaw angle derived in align_yaw), and are unit-tested in tests/sdk/trajectory_metrics.cpp.

• Window: OpenVINS’s upstream start offsets skip the opening seconds of the machine-hall sequences, so the window columns score cortex only from OpenVINS mono’s first pose. Both estimators are then scored over the same span.

ATE, 4-DoF (m)

Sequence	cortex, whole run	cortex, window	OpenVINS mono	OpenVINS stereo	cortex ÷ OV mono
MH_01_easy	0.331	0.333	0.096	0.072	3.5×
MH_02_easy	0.456	0.393	0.168	0.117	2.3×
MH_03_medium	0.500	0.472	0.147	0.096	3.2×
MH_04_difficult	0.375	0.379	0.147	0.127	2.6×
MH_05_difficult	0.671	0.629	0.458	0.281	1.4×
V1_01_easy	0.200	0.197	0.069	0.063	2.8×
V1_02_medium	0.163	0.162	0.058	0.048	2.8×
V1_03_difficult	0.158	0.161	0.067	0.059	2.4×
V2_01_easy	0.223	0.215	0.127	0.060	1.7×
V2_02_medium	0.338	0.334	0.073	0.044	4.6×
V2_03_difficult	0.173	0.175	0.181	0.087	1.0×

The ratio compares cortex’s window with OpenVINS mono; both are monocular.

RPE, mean over 8–40 m (translation / rotation)

Over the window:

Sequence	cortex	OpenVINS mono	OpenVINS stereo
MH_01_easy	0.370 m / 1.57°	0.137 m / 1.55°	0.141 m / 1.67°
MH_02_easy	0.478 m / 1.35°	0.173 m / 1.07°	0.132 m / 0.86°
MH_03_medium	0.386 m / 0.75°	0.178 m / 0.61°	0.159 m / 0.62°
MH_04_difficult	0.548 m / 1.03°	0.206 m / 0.83°	0.213 m / 0.50°
MH_05_difficult	0.701 m / 0.95°	0.506 m / 1.06°	0.300 m / 0.85°
V1_01_easy	0.330 m / 4.04°	0.262 m / 2.84°	0.229 m / 2.83°
V1_02_medium	0.209 m / 2.31°	0.110 m / 0.57°	0.106 m / 0.73°
V1_03_difficult	0.235 m / 2.56°	0.125 m / 1.24°	0.125 m / 1.02°
V2_01_easy	0.447 m / 2.41°	0.220 m / 1.06°	0.102 m / 0.89°
V2_02_medium	0.301 m / 4.09°	0.090 m / 1.55°	0.061 m / 1.27°
V2_03_difficult	0.225 m / 1.74°	0.201 m / 1.56°	0.108 m / 1.38°

What the baseline says

• The typical gap is 2–3.5× in 4-DoF ATE: the median over the 11 sequences is 2.6× (2.8× before #616). V2_03, MH_05 and V2_01 are within 2×.
• Initialization costs the most.
  • MH_03 diverged in the first baseline (348c774: 29,804 m). Rotor vibration kept its still start out of static init, and the gravity-align fallback seeded a zero gyro bias, 0.08 rad/s off. Since #591 the fallback seeds the bias of the stillest vibrating rest window (error 0.004 rad/s), and MH_03 scores 0.484 m.
  • MH_01 starts moving, so static init cannot fire, and the first baseline seeded the filter from a 5-frame (0.25 s) dynamic alignment: 3.336 m over the whole run, 0.869 m over the window. Since #592 dynamic init waits for 10 frames and a settled scale (the window’s first 2/3 must agree with the whole within 5%), and a moving start waits up to 6 s for it rather than fall back at 2 s to gravity alignment with zero velocity. MH_01 then initialized 3.8 s after its first frame: 2.003 m over the whole run, 0.580 m over the window.
  • The dynamic alignment’s scale was 14% off at the median, mostly too small (97 accepted 1 s windows with prefer_dynamic over the first 30 s of the 11 sequences, against the ground-truth displacement). Fed ground-truth poses, the same alignment was 5–18% off, and about 2% once it also estimated the accelerometer bias. The error was the SfM track’s positions: their noise attenuates an alignment that fits their second differences. Since #500 and #501 the window’s gates come from the track noise, the window is bundle-adjusted, and the alignment estimates the accelerometer bias. On the same sweep 133 windows are accepted, and on the 125 with ground truth the scale is 3.6% off at the median (12.8% at the 90th percentile; 60% under 5%). On the 32 windows both versions accept (3 sequences), it goes from 13.5% to 2.8% (90th percentile 21.7% to 9.9%).
  • In the default run that changes only MH_01 and MH_02, the two sequences that initialize dynamically: both now initialize at 0.5 s. MH_01 scores 2.305 m over the whole run (2.19–2.41 m for a track noise of 0.8–1.2 px), MH_02 0.817 m. Neither is limited by its init: scored over their first 20 s alone, both are within 2 cm of the truth from either init. MH_01’s error grows while the drone rests on the ground from 20 s to 42 s: its ATE over the first 36 s is 1.27 m from the old init and 1.41 m from the new, and over the first 42 s 3.00 m and 3.35 m.
• Rests drifted. At rest a monocular MSCKF has no parallax to hold the velocity; on MH_01 the estimated speed grew from 0.06 to 1.6 m/s while the drone rested on the ground. Since #600 a zero-velocity update (S8) holds the velocity at zero and the gyro bias at the mean rate while both the IMU and the image are still. On these 11 sequences its detector fired at none of the 24,353 frames whose ground truth moves faster than 0.05 m/s (a measurement on these runs, not a guarantee). Over the whole run every sequence improves or holds: MH_01 2.305 → 0.322 m, MH_02 0.817 → 0.420 m, MH_04 0.481 → 0.376 m, V2_01 0.318 → 0.224 m, V2_02 0.410 → 0.331 m. Over the window MH_03, MH_04, MH_05 and V1_03 are 6–8% worse (MH_04 0.366 → 0.396 m); the median ratio stays 2.8×. MH_05’s rotation RPE over the whole run goes from 1.07° to 1.89°.
• Rotation drift is the next gap. In the Vicon rooms cortex’s rotation RPE is 1.1–4.1× OpenVINS mono’s (V1_02: 2.31° against 0.57°; before #616 1.4–4.4×, V1_02 2.52°). That is consistent with the body-frame MSCKF’s yaw observability leak (#212, #480). The R-IEKF’s is 0.88° at the median (below).
• Starting at the first frame matters for OpenVINS too. Run monocular from the first frame, it diverges on MH_04 (4-DoF ATE 9,889 m) and is otherwise within 0.035 m of its start-offset score. Its start offsets hide the hard openings that cortex is scored on in the whole-run column.

Front end or estimator?

From #595 to #601 the window’s median ratio stayed at 2.7–2.8×, while the whole-run error fell with each initialization and rest fix. To find which half of the pipeline owns the remaining gap, vio_pipeline --tracks gt (#602) keeps the real IMU and timestamps but replaces the KLT front end with ideal tracks:

• landmarks spawned along random pixel rays at 2–8 m (log-uniform) whenever fewer than 150 are in view, 15 px apart;
• projected through the ground-truth pose and the same cam0 calibration the filter uses, with Gaussian pixel noise of σ px;
• each track ends when its landmark leaves the image.

These are the tracks a perfect front end would deliver, with no outliers, drift or calibration error. --camera-sigma-px sets the filter’s visual noise R (default 0.01 normalized, 4.6 px). The table gives window ATE 4-DoF divided by OpenVINS mono’s, with track seed 1:

Sequence	Real KLT, R 4.6 px	0 px, R 0.5 px	1 px, R 4.6 px	1 px, R 1 px
MH_01_easy	3.5×	1.3×	7.9×	11.3×
MH_02_easy	2.2×	0.6×	2.8×	5.9×
MH_03_medium	3.0×	2.1×	2.4×	2.7×
MH_04_difficult	2.7×	1.4×	1.8×	15.7×
MH_05_difficult	1.4×	0.7×	1.1×	0.8×
V1_01_easy	3.1×	2.3×	4.5×	3.7×
V1_02_medium	3.1×	1.7×	4.3×	3.0×
V1_03_difficult	2.8×	2.0×	3.1×	4.3×
V2_01_easy	1.7×	0.6×	1.4×	1.0×
V2_02_medium	4.5×	1.6×	3.2×	2.8×
V2_03_difficult	1.0×	8.2×	1.6×	19.7×
Median (seeds 1–3)	2.8×	1.5–1.6×	2.4–2.8×	3.8–4.4×

• The estimator owns the gap. The comparison that isolates the front end keeps the filter’s R at its default: with 1 px ideal tracks the median is 2.4–2.8× over three seeds, against 2.8× for the real front end. A perfect front end at that noise would buy little. The noise-free column also changes R, so it is not a front-end comparison: it shows what the estimator reaches with perfect measurements.
• The filter does not survive being told the truth. With R matched to 1 px ideal tracks, 3–5 of the 11 sequences diverge (worse than 10×), depending on the seed; with 0.5 px tracks and a matching R, 3–4 do (V1_01 reaches 59–326×). A consistent filter improves when its measurements and its model of them both improve. This one gets worse: it takes in more information than the tracks carry and becomes over-confident. The default R, about 4.6× the KLT noise, is what keeps it stable. That points at the over-confidence described in #212 and in the rotation-drift finding above, not at the front end.
• Even noise-free tracks leave 1.5–1.6×. MH_02, MH_05 and V2_01 then beat OpenVINS mono’s real-image score, but V1_01 stays at 2.3× and V2_03 rises to 6–9×.
• Checked against artifacts:
  • Ground truth and IMU agree in time to within 1 ms on V1_01, MH_01 and V2_01: the lag that best matches the gyro to the ground truth’s differentiated rotation is 0.
  • Two real-image runs of V1_01 are bit-identical, so repeating a run measures no variation at all.
  • The spread across the three track seeds (medians of 2.4–2.8× at 1 px and the default R) shows how large a change a small perturbation alone produces.

The consistency gate

The ideal-track split above showed that the body-frame filter gets worse when it is told the truth about its measurements. scripts/euroc-consistency.sh (#605) turns that into a pass/fail check that any backend can be run through:

• One truth (#613). Every input comes from a cubic B-spline through the ground truth (vio_pipeline --imu gt): the IMU is synthesized from it at the real IMU timestamps with the filter’s own noise densities, the ideal tracks are seen from it with noise drawn in normalized coordinates (σ = S / fx, what a matched R says), and the run is scored against it (traj_eval --truth). R and Q are then the true noise. The bench is itself a transformation graph.

• Runs: ideal tracks at S = 1 and 0.5 px, track seeds 1–3, on all 11 sequences. Each runs twice: with the filter’s R matched to S, and with its default R (4.6 px). Every backend uses the same, shipped S6d gate. That is 132 runs of about 11 s each, since ideal-track runs read no images.

• Pass condition (bounds in scripts/euroc-consistency.json):

  1. no matched run is worse than 10× OpenVINS mono’s window 4-DoF ATE (a run that writes no trajectory counts as diverged);
  2. at each S, the median matched ratio is below the median default ratio, and every default run was scored;
  3. no yaw collapse: nothing observes yaw, so the filter’s world-yaw σ should not fall below its value at the gauge anchor. The median over the matched runs of (smallest yaw σ ÷ yaw σ at the anchor) must be at least 0.5.

  A run that writes a trajectory that traj_eval cannot score is an evaluation fault, not a divergence. The gate then reports UNAVAILABLE, and --record refuses to store the result.

• Pose NEES: vio_pipeline scores every EuRoC run against the ground truth with tools/pose_consistency.hpp. It anchors the gauge (position and yaw) at the first scored frame, takes the error in the filter’s own convention so its own covariance applies, and splits the attitude in the world frame. It writes the split and the world-yaw σ to consistency.json.

• Why the yaw ANEES is not gated. The first version of condition 3 was a median yaw ANEES of at most 3. Review of #607 showed that test is unsound. The yaw error is measured from the anchor, but the denominator is the current yaw variance alone. The variance of the change since the anchor also needs the anchor’s variance and the covariance between the two times, which neither filter exposes. With a yaw prior the filter never reduces, the current variance mostly reflects the prior, so the ANEES reads low. The yaw σ ratio needs no cross-time term.

  • This replacement was made after the first results. The 10× and ANEES bounds had been set before any estimator ran.
  • The 0.5 bound lets the marginal yaw σ fall somewhat through its correlation with observable states (the R-IEKF reaches 0.52 on MH_03), but not collapse.
  • Every verdict below is the same at any bound between 0.13 and 0.9.

Results with one truth, and S2 integrating both samples (#616)

Recorded at f1551cc for both backends, median over the three seeds. #616 made S2 integrate each IMU interval from both of its samples (RK4 on the line between them) instead of holding one, and made Φ that mean’s exact Jacobian (Aligning Sensor Samples). The matched-run NIS is now 1.00 on every sequence for both backends (1.1–1.9 on the fast ones before):

Gate condition	MSCKF	R-IEKF
1. No divergence (66 matched runs)	fail: 2 diverge (MH_03 1 px seed 2, 26.8 m; MH_04 1 px seed 3, 8.8 km)	fail: the same 2 (MH_03 2.3 m; MH_04 9.0 km)
2. Matched beats default (median, 1 px / 0.5 px)	pass: 0.45× vs 0.52×; 0.33× vs 0.53×	pass: 0.40× vs 0.41×; 0.18× vs 0.36×
3. No yaw collapse (median yaw σ min ÷ anchor ≥ 0.5)	fail: 0.179	pass: 0.984
Median NIS, matched runs (range)	1.00 (1.00–1.11)	1.00 (1.00–1.02)
Roll/pitch NEES, matched runs	1.1–24	0.9–6.8

At 0.5 px, matched R, per sequence (window ATE ratio to OpenVINS mono, yaw σ min ÷ anchor, NIS):

Sequence	MSCKF ratio	MSCKF yaw σ	MSCKF NIS	R-IEKF ratio	R-IEKF yaw σ	R-IEKF NIS
MH_01_easy	0.4×	0.09	1.00	0.4×	0.84	1.00
MH_02_easy	0.4×	0.17	1.00	0.2×	0.90	1.00
MH_03_medium	0.3×	0.17	1.00	0.2×	0.89	1.00
MH_04_difficult	0.2×	0.16	1.00	0.2×	0.91	1.00
MH_05_difficult	0.1×	0.21	1.00	0.0×	0.93	1.00
V1_01_easy	0.4×	0.23	1.00	0.3×	0.88	1.00
V1_02_medium	0.5×	0.28	1.00	0.2×	0.94	1.00
V1_03_difficult	0.4×	0.23	1.00	0.2×	0.94	1.00
V2_01_easy	0.2×	0.23	1.00	0.1×	0.95	1.00
V2_02_medium	0.4×	0.21	1.00	0.1×	0.94	1.00
V2_03_difficult	0.1×	0.21	1.00	0.1×	0.97	1.00

• The fast sequences recovered. At 0.5 px the MSCKF’s V2_03, V2_02 and V1_03 went from 0.9×, 2.1× and 1.7× OpenVINS mono to 0.1×, 0.4× and 0.4×; the R-IEKF’s from 1.4×, 2.4× and 2.0× to 0.1×, 0.1× and 0.2×. Condition 2, which both failed at 1 px, now passes for both.
• The MSCKF’s yaw still collapses, to 9–28% of its anchor σ (0.107 → 0.179 at the median). The observability ledger books none of it to S2 any more: on MH_04 it puts 5e-21 rad⁻² of the spurious yaw information on S2 and 100% on S6’s mean shift, against 0.6–5.3% on S2 before. What is left is #480’s S6 leak, which the body-frame error state can’t avoid.
• Both backends diverge on the same two noise draws. On MH_04 (1 px, seed 3) both start from the same dynamic alignment, at the same instant (the R-IEKF runs the MSCKF’s initializer since #608), and both reach a roll/pitch NEES of 34,000–40,000 and about 9 km: the seed, not either filter. MH_03 (1 px, seed 2) starts by gravity alignment; the MSCKF reaches 26.8 m, the R-IEKF 2.3 m. Both point at the tilt and bias seed of #609.

Results with one truth, before #616

Recorded at c6b87ec (#613) for both backends, median over the three seeds. With the noise model true, the median NIS of the matched runs is 1.03 for both (it was 1.65–4.3 before), so the gate now measures the estimators:

Gate condition	MSCKF	R-IEKF
1. No divergence (66 matched runs)	fail: 2 diverge (MH_03, 1 px, seeds 2 and 3)	fail: 3 diverge (MH_01, MH_03, MH_04, all at 1 px)
2. Matched beats default (median, 1 px / 0.5 px)	fail: 1.09× vs 0.97×; 0.84× vs 0.69×	fail at 1 px: 1.04× vs 0.84×; pass at 0.5 px: 0.46× vs 0.60×
3. No yaw collapse (median yaw σ min ÷ anchor ≥ 0.5)	fail: 0.107	pass: 0.985
Median NIS, matched runs	1.03	1.03
Roll/pitch NEES, matched runs	2–20	2–18

At 0.5 px, matched R, per sequence (window ATE ratio to OpenVINS mono, yaw σ min ÷ anchor, NIS):

Sequence	MSCKF ratio	MSCKF yaw σ	MSCKF NIS	R-IEKF ratio	R-IEKF yaw σ	R-IEKF NIS
MH_01_easy	0.5×	0.08	1.03	0.5×	0.83	1.03
MH_02_easy	0.3×	0.07	1.03	0.2×	0.90	1.03
MH_03_medium	0.5×	0.09	1.04	0.4×	0.80	1.04
MH_04_difficult	0.9×	0.11	1.03	0.4×	0.90	1.03
MH_05_difficult	0.2×	0.15	1.02	0.1×	0.93	1.02
V1_01_easy	1.0×	0.16	1.02	0.4×	0.88	1.02
V1_02_medium	1.5×	0.13	1.10	0.7×	0.94	1.12
V1_03_difficult	1.7×	0.10	1.23	2.0×	0.95	1.30
V2_01_easy	0.4×	0.16	1.04	0.2×	0.95	1.03
V2_02_medium	2.1×	0.10	1.39	2.4×	0.95	1.44
V2_03_difficult	0.9×	0.11	1.80	1.4×	0.97	1.87

Each remaining failure was rerun with the gate loosened to 5 per dof, to check that it belongs to the estimator:

• The MSCKF’s yaw collapses to 7–16% of its anchor σ on every sequence while the R-IEKF holds 80–97%. This is the yaw leak #480 attributed to S6 moving the mean, now measured with a true noise model. It is the robust result: it held through a change of every noise draw (below).
• Divergence is mostly the estimators’, and partly the gate tipping them over. One divergence per backend vanishes with the looser gate (MSCKF MH_03 seed 2: 12.0 m → 0.79 m; R-IEKF MH_04 seed 3: 1423 m → 0.16 m). With a true noise model a consistent filter loses only 5% of its inliers to the gate, so both have moments of over-confidence the gate turns fatal. The other three stay with either gate (MSCKF MH_03 seed 3: 5.3 m; R-IEKF MH_01: 2.5 m, MH_03: 8.2 m). MH_03 at 1 px is hard for both.
• Matched R loses at 1 px on the fast sequences, for both. There the NIS is 1.1–1.9, against 1.01–1.04 on the Machine Hall. The cause is #616: S2 holds the sample that ends each interval (the R-IEKF the one that starts it), a 2.5 ms IMU–camera skew. Averaging the two samples gives NIS 1.00 everywhere and cuts V2_03’s ATE 10–15×. Fixed in #616 (above).
• Conditions 1 and 2 depend on the noise draw at three seeds. The first recording drew the IMU noise and the track noise from the same random stream (a seeding bug found in review); drawing them independently moved which runs diverge (the MSCKF from MH_04 to MH_03, the R-IEKF from one run to three) and turned the MSCKF’s condition 2 from a narrow pass into a fail. Condition 3 and the NIS did not move. More seeds would make 1 and 2 decisive.
• Roll/pitch NEES is now 2–20 for both, against 17–3161 before: the real IMU’s accelerometer bias and its disagreement with the ground truth (#609) are no longer in the bench. They remain on real data.

The #605 results (superseded)

Recorded in #605 (commit db7ccdc), median over the three seeds (window ratio to OpenVINS mono). These verdicts were largely an artifact of the S6d gate: see the correction below.

Sequence	MSCKF 1 px matched	MSCKF 1 px default	R-IEKF 1 px matched	R-IEKF 1 px default	R-IEKF 0.5 px matched
MH_01_easy	11.9×	15.1×	9.1×	9.0×	5.1×
MH_02_easy	3.0×	2.7×	0.8×	3.4×	0.6×
MH_03_medium	2.7×	1.8×	1.4×	2.5×	1.1×
MH_04_difficult	29.8×	2.3×	1.5×	1.8×	1.3×
MH_05_difficult	0.8×	1.1×	0.9×	1.1×	0.8×
V1_01_easy	4.4×	5.3×	3.1×	4.9×	1.9×
V1_02_medium	3.0×	2.9×	1.7×	3.6×	1.4×
V1_03_difficult	4.3×	3.1×	2.6×	4.1×	1.8×
V2_01_easy	1.0×	1.3×	0.7×	0.9×	0.5×
V2_02_medium	3.7×	3.6×	2.4×	4.0×	1.6×
V2_03_difficult	24.3×	1.5×	1.0×	1.3×	0.9×

Gate condition	MSCKF	R-IEKF
1. No divergence (66 matched runs)	fail: 22 diverge	fail: 1 diverges (MH_01, 1 px, seed 2)
2. Matched beats default (median, 1 px / 0.5 px)	fail: 4.10× vs 2.62×; 6.42× vs 2.03×	pass: 1.47× vs 3.43×; 1.25× vs 3.39×
3. No yaw collapse (median yaw σ min ÷ anchor ≥ 0.5)	fail: 0.076 (every run 0.03–0.13)	pass: 0.902 (every run 0.52–0.99)
Yaw ANEES (informational)	47.8	1.26

• The body-frame MSCKF fails all three. Told the truth, it gets worse at both noise levels, and 22 of its 66 matched runs diverge. Its world-yaw σ falls to 3–13% of its value at the anchor on every run (to 0.004–0.013 rad), and its yaw error does not shrink with it. This is the leak #480 attributed to S6 moving the mean.
• The R-IEKF passes the two consistency conditions. Its world-yaw σ holds at a median 90% of its anchor value and never falls below its 0.05 rad prior. At 1 px, matched R helps on 10 of the 11 sequences (the MSCKF: 4), and at 0.5 px the median is 1.25× OpenVINS mono’s real-image score.
  • Its one divergence is MH_01, whose start moves: the adapter seeds a zero velocity at the origin (see below).
  • Its weakest sequences on condition 3 are MH_03 (yaw σ falls to 0.52 of the anchor) and V1_03 (0.68). There its anchor σ sits above the prior and returns to it.
• On real KLT tracks the order reverses. With the default front end the R-IEKF scores 0.563 m on V1_01 against the MSCKF’s 0.217 m, and 0.503 m on V2_03 against 0.178 m. With ideal tracks it is the better estimator. So what holds it back on real data is outside the estimator’s consistency: its adapter still gates at 16 per dof instead of the χ² quantile (#481), seeds from a static window at the origin with zero velocity, and has none of the dynamic-init (#592, #597, #599) or zero-velocity (#600) work.
• Roll/pitch NEES is recorded, not gated. It is 17–3161 for both backends, and it is not a clean measure on EuRoC. The ground truth carries an accelerometer bias of 0.09–0.55 m/s² (V1_01: 0.548 m/s² on y). Both filters seed their tilt from the raw accelerometer, so they start 0.5–2.7° tilted and never learn the bias (V1_01: 2–3° throughout, with a filter tilt σ near 0.06°). Even after the ground truth’s own bias is removed, its body frame disagrees with the accelerometer’s gravity by 0.1–0.9° at rest. That caps the roll/pitch NEES any filter can reach against it.

Correction: the verdict was mostly the S6d gate

Running the R-IEKF through the shipped real-data path (#608) put it on the same χ² quantile gate as the MSCKF, and it then failed conditions 1 and 2 as well: 9 of 66 matched runs diverge, and matched R scores 3.62× OpenVINS mono against 2.45× for the default at 1 px (2.59× vs 2.18× at 0.5 px). Its yaw σ ratio is 0.73, still a pass. Reverting one piece at a time on V1_01 at 0.5 px puts the whole difference on the gate. The R-IEKF converges whichever init and S8 it runs, as long as the gate is the old 16-per-dof line, and diverges with the χ² quantile whichever init and S8 it runs.

The reason is that “matched R” is not the noise the filter sees. The ideal tracks are synthesized from the ground-truth trajectory but the IMU is real, so every observation also carries the ground truth’s error relative to the IMU. Even converged runs have NIS well above their degrees of freedom. A gate at α = 0.05 assumes NIS ≈ dof, so it rejects a large share of valid long tracks, and the filter’s heading flips on what is left:

Run (0.5 px, matched R), 4-DoF ATE (m), seeds 1 / 2 / 3	χ² quantile, α = 0.05	α = 10⁻³	α = 10⁻⁶	5 per dof	NIS (5 per dof)
R-IEKF, V1_01	99.4 / 4.7 / 83.4	0.14 / 0.16 / 0.16	0.13 / 0.14 / 0.13	0.12 / 0.13 / 0.13	1.65–1.67
MSCKF, V1_01	9.0 / 22.7 / 4.0			0.14 / 0.16 / 0.16	1.65–1.67
R-IEKF, V2_03	30.3 / 8.8 / 7.8	1.91 / 0.84 / 0.59	0.34 / 0.46 / 0.43	0.30 / 0.26 / 0.26	3.8–3.9
MSCKF, V2_03	16.5 / 8.3 / 19.2			0.68 / 1.25 / 1.27	7.5–10.5

So conditions 1 and 2 above mostly compared gate lines. The pre-#608 R-IEKF adapter gated at 16 per dof and the MSCKF at the χ² quantile. With the same line the two converge to the same accuracy on V1_01. A difference survives on V2_03, where the R-IEKF reaches 0.26–0.30 m against the MSCKF’s 0.68–1.27 m with half its NIS. The gate stays as recorded until #613 measures the ground truth’s residual floor, folds it into matched R, runs every backend with the same gate line and re-records both. Condition 3 does not depend on the gate’s line in the same way, but it is re-measured there too. #613 did this by removing the floor rather than modelling it: see Results with one truth, before #616, and after it.

This is the fourth result in the VIO work that came from the test setup rather than the estimator (the storyline).

scripts/euroc-consistency.sh                    # MSCKF; ~8 min at 3 jobs
scripts/euroc-consistency.sh --backend riekf    # the R-IEKF (--invariant)
scripts/euroc-consistency.sh --record           # record this backend's results

The script exits non-zero while the gate fails. Like the baseline, it needs the dataset on disk and is not in CI.

The R-IEKF on the shipped real-data path

Before #608 the R-IEKF reached EuRoC through a harness that gated at 16 per dof, seeded a zero velocity from its own 150-sample static window, and had no S8. It now runs the MSCKF’s own initializer (static, vibrating-rest gyro bias, gravity-align fallback, dynamic VI alignment), seeds its error state from that seed’s prior mapped into the right-invariant error, gates with the same χ² quantile, and has the zero-velocity update. The change was planned as a delta on the OpenVINS transformation graph. --riekf-legacy gate,init,zupt puts any piece back, so each was measured on its own, cumulatively, on the real KLT front end:

R-IEKF configuration	Median window ATE ÷ OpenVINS mono	Sequences better than the MSCKF	Median rotation RPE
Before #608 (--riekf-legacy gate,init,zupt)	7.76× (V1_02 diverges, 1729 m)	0 / 11	4.18°
+ χ² gate (--riekf-legacy init,zupt)	2.71×	5 / 11	1.65°
+ the shipped initializer (--riekf-legacy zupt)	3.22× (MH_05 diverges, 6390 m)	2 / 11	2.46°
+ S8: all pieces (--invariant)	2.62×	7 / 11	0.90°
The MSCKF (reference)	2.83×	—	2.21°

Per sequence, window 4-DoF ATE (m) and window rotation RPE (°):

Sequence	R-IEKF before #608	MSCKF	R-IEKF	OpenVINS mono	R-IEKF ÷ mono	MSCKF rot	R-IEKF rot	OpenVINS mono rot
MH_01_easy	0.952	0.332	0.315	0.096	3.3×	1.62	0.80	1.55
MH_02_easy	0.586	0.369	0.384	0.168	2.3×	1.12	1.51	1.07
MH_03_medium	1.455	0.439	0.392	0.147	2.7×	0.65	0.67	0.61
MH_04_difficult	0.883	0.396	0.377	0.147	2.6×	1.19	0.42	0.83
MH_05_difficult	0.944	0.645	0.619	0.458	1.4×	0.81	0.70	1.06
V1_01_easy	0.565	0.213	0.181	0.069	2.6×	4.03	3.81	2.84
V1_02_medium	1728.9	0.183	0.159	0.058	2.7×	2.52	0.89	0.57
V1_03_difficult	1.281	0.189	0.240	0.067	3.6×	2.63	0.90	1.24
V2_01_easy	0.822	0.215	0.221	0.127	1.7×	2.43	1.26	1.06
V2_02_medium	0.565	0.330	0.274	0.073	3.8×	4.55	1.35	1.55
V2_03_difficult	0.497	0.181	0.210	0.181	1.2×	2.21	1.81	1.56

• The gate is the largest single piece. At 16 per dof the harness admitted tracks the χ² quantile rejects, and V1_02 diverged. With the shipped gate the R-IEKF reaches the MSCKF’s accuracy.
• The shipped initializer needs S8. Alone it is worse than the harness’s window: MH_05 diverges, and V1_03 doubles. With S8 holding the rest periods it is the best configuration. On MH_01 and MH_02 it starts by dynamic alignment, on MH_03 by the gravity fallback, elsewhere statically.
• With every piece, the R-IEKF is slightly ahead of the MSCKF: 2.62× against 2.83× OpenVINS mono, better on 7 of 11 sequences, and with less than half the rotation drift (median 0.90° against 2.21°; OpenVINS mono 1.07°). It is still 2.62× OpenVINS mono’s ATE.
• Its NIS is now reported. It averages 0.97–2.55 on the Machine Hall and on V1_01 and V2_01, and 6.8–15.4 on V1_02, V1_03, V2_02 and V2_03, where the filter is over-confident or the gate admits outliers.
• Roll/pitch is still wrong at the start: V1_01 has a roll/pitch NEES near 3000 in every configuration, the tilt and accelerometer-bias seed of #609.

After #616: S2 integrates both samples

#616 changed S2 in both backends (each IMU interval integrated from both of its samples by RK4, and Φ the exact Jacobian of that mean). On the real KLT front end, window 4-DoF ATE (m) and window rotation RPE (°), before → after:

Sequence	MSCKF ATE	R-IEKF ATE	OpenVINS mono	MSCKF ÷ mono	R-IEKF ÷ mono	MSCKF rot	R-IEKF rot
MH_01_easy	0.332 → 0.333	0.315 → 0.312	0.096	3.5×	3.3×	1.62 → 1.57	0.80 → 0.82
MH_02_easy	0.369 → 0.393	0.384 → 0.370	0.168	2.3×	2.2×	1.12 → 1.35	1.51 → 1.37
MH_03_medium	0.439 → 0.472	0.392 → 0.402	0.147	3.2×	2.7×	0.65 → 0.75	0.67 → 0.66
MH_04_difficult	0.396 → 0.379	0.377 → 0.359	0.147	2.6×	2.4×	1.19 → 1.03	0.42 → 0.46
MH_05_difficult	0.645 → 0.629	0.619 → 0.612	0.458	1.4×	1.3×	0.81 → 0.95	0.70 → 0.68
V1_01_easy	0.213 → 0.197	0.181 → 0.182	0.069	2.8×	2.6×	4.03 → 4.04	3.81 → 3.74
V1_02_medium	0.183 → 0.162	0.159 → 0.146	0.058	2.8×	2.5×	2.52 → 2.31	0.89 → 0.81
V1_03_difficult	0.189 → 0.161	0.240 → 0.131	0.067	2.4×	2.0×	2.63 → 2.56	0.90 → 0.88
V2_01_easy	0.215 → 0.215	0.221 → 0.219	0.127	1.7×	1.7×	2.43 → 2.41	1.26 → 1.33
V2_02_medium	0.330 → 0.334	0.274 → 0.261	0.073	4.6×	3.6×	4.55 → 4.09	1.35 → 1.37
V2_03_difficult	0.181 → 0.175	0.210 → 0.177	0.181	1.0×	1.0×	2.21 → 1.74	1.81 → 1.80
median				2.83× → 2.57×	2.62× → 2.44×	2.21 → 1.74	0.90 → 0.88

• On real data the gain is real but small. On the spline truth #616 cut the fast sequences’ ATE 4–24×; here the medians move 9% (MSCKF) and 7% (R-IEKF). The R-IEKF’s V1_03 halves (0.240 → 0.131 m) and V2_03 gains 16%; the MSCKF’s rotation drift falls 21% at the median (V2_03 2.21° → 1.74°). The MSCKF’s MH_02 and MH_03 are 7% worse. The R-IEKF is now ahead of the MSCKF on 9 of 11 sequences (7 before).
• The real-data NIS did not move. Both backends stay at 0.93–2.55 on the Machine Hall, V1_01 and V2_01, and at 6.7–15.5 on V1_02, V1_03, V2_02 and V2_03. On the spline truth the same sequences went from 1.1–1.9 to 1.00, so the half-sample skew is not what makes the filters over-confident on the real IMU and the real tracks. What does is still open: the front end’s outliers, R, the ADIS16448’s own filter delay, the camera–IMU time offset (#332).

CORTEX_VIO_ARGS="--invariant" scripts/euroc-baseline.sh --no-build                       # all pieces
CORTEX_VIO_ARGS="--invariant --riekf-legacy gate,init,zupt" scripts/euroc-baseline.sh   # before #608

The baseline compares each sequence with the recorded MSCKF numbers, so it exits non-zero whenever the R-IEKF is worse on any of them.

The regression benchmark

scripts/euroc-baseline.sh reruns the baseline on the development station. It runs vio_pipeline on every sequence, scores each trajectory with traj_eval over the whole run and over the window, and compares the whole-run scores with the recorded cortex numbers:

scripts/euroc-baseline.sh                    # all 11 sequences, ~25 min serial
scripts/euroc-baseline.sh V1_01_easy MH_05_difficult
CORTEX_BASELINE_JOBS=4 scripts/euroc-baseline.sh
scripts/euroc-baseline.sh --update           # record a new baseline

• Verdict: a sequence is worse when its 4-DoF ATE, RPE translation or RPE rotation exceeds the recorded value by more than 10% (CORTEX_BASELINE_TOL) plus a noise floor (5 mm, 5 mm, 0.05°). It is better when it falls below by the same margin.
• Exit status: the script exits non-zero on any worse or failed sequence.
• Updating: when a change improves the pipeline, --update records the new numbers and the commit; update the tables on this page to match.
• Not in CI: the script needs the EuRoC dataset on disk. The OpenVINS columns it prints are the recorded targets, not inputs; nothing here runs OpenVINS.

CORTEX_VIO_ARGS passes extra vio_pipeline flags to every run, for example the ideal-tracks experiment above:

CORTEX_VIO_ARGS="--tracks gt --track-sigma 1 --track-seed 2" \
  CORTEX_BASELINE_OUT=build/ideal scripts/euroc-baseline.sh --no-build

--update refuses to record a run that sets it.

To score another estimator’s trajectory on the same terms, write it as TUM (t x y z qx qy qz qw, IMU pose in the world frame) and run traj_eval --dataset <mav0> --est <file> [--from <s>] [--json].