The Storyline

The chapters are in the order the problems were found. Each one names what the failure looked like, the measurement that decided its cause, and the fix. Where a later chapter revised an earlier claim, the revision is stated where the claim is.

Chapter 1. Build it from parts

Epic #33 built the pipeline bottom-up, each part unit-tested against synthetic known answers:

Interfaces and backends: the VioBackend interface, the MSCKF backend, a sliding-window skeleton.
Front end: FAST-9 with non-maximum suppression, pyramidal KLT, images and pyramids (S4).
Models: camera models with analytic Jacobians, IMU preintegration, feature representations with the MSCKF null-space types (S0).
Estimator: static and dynamic IMU initialization, the state and propagator, the camera updater with null-space projection and a Mahalanobis gate, and the VioEstimator facade.
Evaluation: EuRoC replay with ATE/RPE and a latency budget. A square-root covariance backend came later ( #187 ).

When the epic closed, nothing had run on real data. Its EuRoC acceptance tests skip in CI, which has no dataset. Every later chapter is about what synthetic unit tests can’t see.

Pitfall: FAST-9 finds no corners on the X-junctions of an aligned checkerboard ( #37 ). Choose test images for the detector you have.
Pitfall: the preintegrated Δv and Δp are gravity-free, with R_i·Δv = v_j − v_i − g·Δt ( #40 ). That convention became the seam of the worst bug in Chapter 2.

Chapter 2. First contact with real data: 26 km of error

Wrong extrinsics. The first EuRoC V1_01 run gave an ATE of 26,574 m against a 0.5 m gate, after a 13-minute run. There were two faults:

The camera↔IMU extrinsics were left at identity. EuRoC cam0 is rotated about 90° and offset about 7 cm.
Initialization fell back to an identity attitude when the start was never static.

The fix loaded the real T_BS, added try_gravity_align (roll and pitch from the mean specific force), InitDiagnostics, a per-frame divergence guard and --fail-fast. V1_01 dropped to 0.305 m. The replay tests had the same identity-extrinsic fault: MH_05 was at 21,405 m and V2_03 at 6,680 m until it was fixed (MH_05 0.785 m, #245 ).

The initialization race. With real extrinsics, V2_03 still diverged to 12,456 m with init=dynamic ( #244 ). A degenerate dynamic init, whose scale wasn’t observable, won the race against static init. A scale-observability gate (min_dynamic_motion = 5 cm) made dynamic init decline; static init then won, and the error fell to 0.293 m.

Forced dynamic init: a chain of faults ( #247 ). Forcing the dynamic path on MH_05 gave 41,910 m, against 0.79 m with static init.

Fault	Problem	Fix
F2	Near-hover windows are rank-deficient, and the scale collapses to about 1e-8	The observability gate; after it, dynamic init never fired
F4	Bootstrapping SfM on adjacent 20 Hz frames gave almost no parallax: a best metric motion of 0.14 mm	Use the widest-parallax pair: 56 mm, still 12.9 km of error
F5a, the root cause	The alignment solve used +g·Δt where the preintegrator uses −g·Δt, so it recovered g = −g_true. The seed was tilted 177.7° from the accelerometer, injecting about 2 g of phantom acceleration	Fix the sign: tilt 177.7° → 2.33°, error 48 km → 19.3 m

The synthetic fixtures carried the same sign error, so CI stayed green. Scoring ATE only after initialization (19.9 m) confirmed the remaining 19 m is real post-init error.

A negative result: seeding roll and pitch from the accelerometer on a moving start made it worse (37.3 m), because motion contaminates the specific force. It was reverted.

Still open in initialization:

Forced dynamic init on MH_05 is still about 19 m against a 1.5 m gate ( #254 ). The start moves at about 0.1 m/s, and a 20% scale error over about 100 m explains it. This is an excitation floor, not a defect.
A metric lever arm is added to up-to-scale SfM positions ( #499 ). Even on exact data the scale error is 0.88–5.49%, against 0.017–2.40% with the lever arm zeroed.
The RANSAC inlier gates are about 0.9 px (two-view) and 2.3 px (PnP) ( #500 ). At 2 px of noise no window ever builds. On MH_05, 82 of 170 moving attempts fail despite more than 100 px of disparity. At rest, about 0.05 px of pseudo-parallax is accepted, because there is no minimum-parallax gate.
Alignments with 21–90% scale error are accepted, and up to 257% with 4 Hz keyframes ( #501 ). The gate checks observability, not consistency.
Dynamic alignment declines in float and posit32, because it solves normal equations with a fixed ridge ( #468 ).
The diagnostics log a 177° tilt but don’t refuse the seed ( #267 ).
Fixed: jacobi_eigh stopped only at an exactly zero off-diagonal and always ran 100 sweeps ( #497 ).

The full contract of this stage is on S1 — Initialization.

Chapter 3. Accurate, but over-confident

With initialization fixed, V1_01 met its ATE gate (0.305 m), but the relative error was about 18% (rotation 3.1°/m) against a 5% target ( #212 ). Once there were instruments to measure it (NEES and NIS, gauge-anchored and per block, #264 ), the real problem showed: the filter was over-confident.

Measured in double (long double gives the same, so precision is not the cause; #532 ):

Sequence	ATE	NEES/dof	NIS
V1_01	0.289 m	436 (attitude 993)	1.5
MH_05	0.755 m	42.8	—
V2_03	0.269 m	140	14.7

Per-block NEES put the problem in attitude (V2_03: attitude 169, position 3.5). NIS stayed healthy while NEES was in the hundreds: the innovations looked fine while the state covariance was wrong. That combination is the signature of information leaking along an unobservable direction.

We built instruments to find the cause:

Stage probes, synthetic and per stage, one per pipeline stage.
Real-data stage inspectors, tapped from a live EuRoC replay ( #371 ).
An observability null-space audit ( #337 ).
A per-stage ground-truth-injection harness ( #266 ), later superseded by the probes.

Each candidate cause got a measurement. The method is described on Diagnostic Methodology, and the worked chain on Consistency Analysis.

Candidate	Deciding measurement	Verdict
Q/R mistuned	Best NEES at Q×10. R×4 gives NIS 1.08 but ATE 2.07 m; R×10 gives 278 m. No (Q, R) pair is both consistent and accurate ( #280 )	Refuted as the sole cause; tuning only compensates
Gross calibration or time-sync bias	Innovation mean −0.013σ	Refuted
Non-white innovations	After a deterministic update order, MH_05 innovations are white	Partly an artifact
Yaw null-space leak (body-frame EKF)	‖H·N‖ is 2e-16 at a consistent point and 0.147 at σ = 2 cm, growing with σ (0.028 → 0.462); the translation leak is exactly 0 ( #337 ). On real V2_03 the mean leak is 0.39, nonzero on every update. S6’s projection adds none at its own linearization point ( #457 ); S2 leaks 2.4e-6 per 5 ms step	Confirmed as a mechanism. Attributed in the row below: the S6 update moving the mean carries 95–99%, S2 1–5%, S3 none
FEJ, measurement-only	Attitude NEES 993 → 1336 (V1_01); MH_05 → 107,544 and 20.9 m ( #339 )	Refuted
FEJ, full	V1_01 → 2281 m; MH_05 → 11 km	Refuted as implemented: partial FEJ mixes a first-estimate H with a current-estimate P
Unmodeled calibration, as an isotropic R term	NIS 14.7 → 1.41, but NEES 140 → 3074 and ATE → 4.74 m	Refuted as the fix: it cures the symptom and worsens the state
Online spatial extrinsics (S10)	V2_03 NEES 140 → 110; NIS unchanged; ATE never better ( #332 )	Cleared as the driver
Time offset t_d	not implemented	Open
Parallax gate (S5)	V2_03 ATE 0.27 → 1.99 m, NEES 140 → 322	Refuted for the body-frame filter on EuRoC
Diagonal Q_d (S2)	Drops 0.35% of Q; propagation-only NEES 6.4 against 6	Cleared for the body-frame filter (but see Chapter 5 for the R-IEKF)
Update algebra (S6)	NIS/dof 1.002 at matched (P, R); null-space residual 2.8e-16	Cleared
Augmentation and marginalization (S3, S9)	Machine-zero residuals	Cleared
Linearization points drifting apart across stages	Observability ledger: the update moving the mean carries 95–99% of the spurious yaw information on EuRoC, S2 1–5%, S3 and S9 none; no position leak ( #480 )	Confirmed for the body-frame filter; the R-IEKF removes it by construction
Arithmetic precision	Long double = double	Refuted
χ² gate too loose	5·dof admitted 59% (dof 1) and 70.5% (dof 3) of 2σ outliers; the χ²₀.₉₅ quantile admits 46% and 22.5%. End to end it lowers EuRoC ATE (V1_01 0.29 → 0.22 m, MH_05 0.75 → 0.73 m, V2_03 0.27 → 0.18 m) and leaves NIS unchanged (1.5, 2.2, 14.7) (finding #481 ; measured in #572 )	Fixed (the shipped gate); an accuracy lever, not the over-confidence driver

The structural cure: the R-IEKF ( #347 ). A right-invariant error makes the four-DoF gauge (translation and yaw) independent of the state, so the update cannot leak information along it.

Proven at the unit level: H·N = 0 and Φ·N = N at any state ( #348 ). On real V2_03 perturbations the yaw leak is 2.3e-16, against 0.39 for the body-frame filter.
Assembled, it failed at first. The backend diverged, and with updates it did worse than without them (162.9 m against 14.6 m, #365 ). There were four bugs ( #366 ): the sign of H_φ, a retraction mismatch, a first-order Φ that needed to be second order, and a wrong NEES metric in the test. Only a multi-cycle finite-difference check of the joint step caught them.
Unit-proven is not system-proven. On real EuRoC the R-IEKF still loses on accuracy: V1_01 ATE 1.31 m against 0.22 m, V2_03 1.36 m against 0.18 m (R-IEKF figures from #429 ; body-frame figures under the χ² gate from #572 ). Its attitude NEES on EuRoC has never been reported.

Chapter 4. Divide and conquer, and numerics

Some numerical failures (a square-root ATE of 6e+40 m, a non-finite δx) could not be told apart from algorithmic ones. That motivated the infrastructure of epic #444 , described in its own section:

Stages S0–S10 exposed as transformations, so the backend is a call sequence.
Stage benches with known-answer, ground-truth-injected and captured fixtures, checked by an invariant library with tolerances scaled to the arithmetic type.
Stream capture from real runs with a first-violation trigger, and composition benches.

We then made the pipeline generic over arithmetic types:

Generic over types: StrictDouble has explicit-only conversions, so an implicit narrowing is a compile error, and the result must match double bit for bit.
Precision measurement: a precision mode measures the bits lost per type, and each stage can run in its own type (Mixed Precision).

What we learned:

Stages that write P (S2, S3, S9, S6e) must stay double, unless the covariance is square-root ( #532 ).
- Mechanism: in float, at takeoff (V1_01 frame 105), one update cuts the largest variance from 1.49 to 0.135. The Joseph form’s cancellation error (about ε·‖P_prior‖) then makes P indefinite (λ_min −4e-5 of the largest), and the run ends at 128,911 m.
- A diagonal “undo negative variance” guard rejected 25,777 updates and still ended at 10,147 m: P is indefinite before any diagonal goes negative.
- Square-root all-float holds all three sequences (V1_01 0.295, MH_05 0.784, V2_03 0.282 m).
- Stages that only read P tolerate float.
Divergence follows conditioning, not the number type. The S6e update loses log₂κ(S) to log₂κ(S)+3 bits in every type; data-movement stages lose none. posit16 is too narrow: triangulation breaks at 1° of parallax or less.
The square-root factor is tall after augmentation ( #533 , #461 ). The update assumed a square factor and read past the end of H·S. That gave NaN on V1_01 in double, at a frame that varied from run to run (heap garbage), while synthetic tests passed bit for bit. After the fix, ASan and UBSan are clean on the real replay. The worked example tells how it was found.
Never re-factor P into S in a narrow type ( #482 ). Clone/pose correlation puts pivots below float resolution: 36 of 51 were zeroed. Carry the factor instead.
Some tolerances should scale with ε and some must not (Numerical Thresholds):
- Scale with ε: the small-angle hand-off at max(1e-5, ε^(1/3)), and the PSD slack, measured against the largest P_ii rather than per entry.
- Never scale: regularizing ridges stay at a fixed 1e-9. An ε-scaled ridge biased S1’s float scale from 0.88% to 17.2% error, even in exact arithmetic.
Keep timestamps double. Summing dt in float drifts about 6e-11 s per sample, so reset the filter clock from the sensor time after each propagation.

Chapter 5. When the test lies

Four times, a “filter defect” turned out to be a defect in the test.

The compiler was not the cause ( #437 ). The R-IEKF end-to-end test passed on GCC (3.44 m) and failed on Clang (58.48 m), with each compiler deterministic and the sanitizers clean.

Cause: the test built the gyro and accelerometer samples from random draws inside two arguments of one call. Argument evaluation order is unspecified, so each compiler saw a different noise realization.
Fix: sequence the draws. Clang CI now runs the whole suite.
The real lesson: GCC’s seed passed by luck. Sweeping 21 seeds showed 5 diverged (21–120 m).

The world was wrong ( #475 , #537 ). The R-IEKF looked inconsistent and diverged on 5 of 21 seeds, and changing the χ² gate rescued some seeds while breaking others. There were two causes:

The world: the synthetic trajectory left its warm-up with a velocity step of about 1 m/s that no accelerometer sample recorded, while the ground truth reported rest. With noise-free sensors, roll/pitch NEES reached about 3000.
The adapter: setting the state at t = 1.5 s still integrated the warm-up IMU samples from t = 0, a constant offset of about 3 m.

Smoothing the onset and dropping the stale samples fixed both. All 21 seeds then gave 1.8–7.6 cm RMS at a normalized NEES of 0.56–1.84. The work also found one real filter gap: the process noise wasn’t right-invariant. A Monte-Carlo test 20 m from the origin showed the diagonal Q under-predicted position variance 135×.

A noise-free control proves nothing ( #551 ). After the world was fixed, the S10 control read NEES 0.036, and the docs called the body-frame filter “not over-confident”. But that control fed exact observations. With noise matched to what the filter assumes (1 px, the datasheet IMU densities) and perfect calibration:

10 seeds	consistent	measured
pose NEES	6	10.6–112, median 34
roll/pitch NEES	2	3.9–33.5, median 11.5
yaw NEES	1	0.3–100 (7 of 10 above 6)
position NEES	3	1.4–12.8, median 2.0
yaw σ, first → last frame	never falls	0.16 → 0.010–0.025 rad

These numbers are from after the synthetic camera was corrected ( #467 , below) and the χ² gate became a quantile ( #481 , measured in #572 ). It first read 9.0–5088, median 26, then 11.5–288, median 47; each change moved the magnitudes but not the signature.

Global yaw is unobservable, so its σ can only grow; here it falls about tenfold within seconds. With IMU noise alone and exact observations, the yaw error reaches 0.23 rad on one seed, about 400× what gyro noise integrates to. This is the Chapter 3 yaw leak, reproduced for the first time where the truth is known, and it needs no calibration error. The R-IEKF under the same noise gives 0.55–1.83. The body-frame filter was kept, characterized by a test, as the EuRoC accuracy reference until the R-IEKF matches it there.

Where the leak comes from ( #480 ). The S6 benches had shown that one update can’t add information along the unobservable directions: it evaluates every Jacobian at one estimate, so H·N(x̂) = 0. The issue suspected the stages around it, S2’s Φ and S3’s clones. An observability ledger settled it. It carries the unobservable basis through the run with the filter’s own Jacobians, and splits the basis’s drift away from N(x̂) into the defect each stage introduces. Every camera update’s information along the carried basis is then booked to the stage whose defect put it there.

	V1_01	MH_05	V2_03	synthetic, 1× noise
S6: the update moves the mean	99.4%	99.1%	94.7%	99.9%
S2: Φ against the mean’s Euler step	0.6%	0.9%	5.3%	0.1%
S3, S9, untapped changes	0	0	0	0
information along global translation	0 (1e-22)	0	0	0
filter yaw σ, start → minimum	0.1 → 0.027 rad	0.1 → 0.024	0.1 → 0.019	0.1 → 0.043

So the cause is the textbook EKF inconsistency. An update corrects the mean, the next Jacobians are evaluated at the corrected mean, and the basis they annihilate is no longer the one the earlier Jacobians carried. S2’s share, Φ linearized against the mean’s Euler step (a higher-order Φ leaks more, not less), grows with rotation rate (V2_03); it isn’t the driver. Since #616 Φ is the exact Jacobian of the RK4 mean, and S2’s share is 0.0% on MH_05 and V2_03. Three remedies act on that mechanism:

First-estimates Jacobians, which were tried and refuted on ATE ( #280 , #339 ).
Projecting each H off the carried basis, the observability-constrained EKF, which hasn’t been tried.
An error state whose basis doesn’t depend on the estimate: the R-IEKF, the chosen path ( #365 ).

The truth wasn’t true, and then the test blessed the defect ( #605 , #613 , #616 ). The consistency gate feeds a filter ideal tracks made from the ground truth and tells it the true noise. Its first verdict was that the R-IEKF passed and the MSCKF failed.

The truth: with the real IMU, the “true noise” wasn’t true. EuRoC’s ground truth disagrees with the gyro by 0.25–0.45 px over 0.25 s in the Vicon rooms, so matched R and Q were wrong by an unmodelled, time-correlated error, and the χ² gate starved both filters.
The fix: one truth for everything ( #613 ). A B-spline through the ground truth feeds the synthesized IMU, the tracks and the score. Median NIS became 1.03 for both backends.
What that exposed: NIS was 1.00 on the slow sequences and 1.1–1.9 on the three fast ones. Both backends held one IMU sample over each 5 ms interval, the MSCKF the end sample and the R-IEKF the start sample: a ±2.5 ms IMU–camera skew. Averaging the two samples gave NIS 1.00 and cut V2_03’s ATE 10–15× ( #616 ).
The test that blessed it: the S2 closure test written for #613 measured the first-order error, and its threshold was set just above it. Its bound now comes from what the camera resolves, a tenth of 0.5 px over a clone window. The S2 probe had also recorded the first-order error in June, as a metric with no bound.
The fix and its limit: both backends now integrate each interval from both samples by RK4, with Φ that mean’s exact Jacobian. On the spline truth NIS is 1.00 on every sequence and the fast sequences’ ATE drops 4–24×. On the real IMU and tracks the median ATE improves only 7–9%, and the real NIS doesn’t move (6.7–15.5 on the fast sequences), so the skew isn’t what makes the filters over-confident on real data.

The whole account, with a checklist for any fusion pipeline, is Aligning Sensor Samples.

Smaller ones:

The synthetic camera’s tangential distortion p2 was 1000× EuRoC’s ( #467 ): a divisor lost three zeros. The world is self-consistent, so nothing failed; the S0 bench found it by comparing the world’s pixels with the real EuRoC calibration. Fixing it moved the synthetic numbers that depend on pixels: the body-frame matched-noise NEES above, and the demo’s noise cliff, which moved from between 2× and 4× the assumed noise to between 1× and 2×. A test now holds every camera parameter to the published calibration.
The S5 bench counts a correct triangulation rejection as a violation, so it becomes the first violation in every capture ( #487 ).
The S6 inspector’s “18% non-PSD” was its own positive-definite check, not a PSD check.
Innovation non-whiteness came from the update order.
Flight-data Allan variance overstates the noise density about 30×.
The tools silently wrote nothing when --out didn’t exist.

Chapter 6. A slow start

With the world fixed, a slow motion onset (1 s or more) broke the R-IEKF again ( #548 ): 14 of 21 seeds were out of band at 1 s and all 21 at 2 s, with roll/pitch NEES up to 624.

Cause: the R-IEKF triangulated with a linear solve only. At the 0.93° parallax of the first window (pixel noise 0.29°), all 234 features came out more than 1 m off (mean 5.6 m).
How it was confirmed: substituting the true landmarks for the first second’s updates brought roll/pitch NEES from 294 to 0.79.
Fix: Gauss-Newton refinement of the reprojection error, which the body-frame path already had, with a backtracking line search that never accepts a step raising the cost. Onsets up to 2 s are now in band on all 21 seeds; at 3 s one seed is marginal. Five iterations weren’t enough from a seed that bad; 20 converge.
Why not a gate: a parallax gate worked only in a narrow threshold window and threw away hundreds of tracks. On EuRoC V2_03 it had already regressed the body-frame filter.

The stage contract is on S5 — Feature Triangulation.

What finishes the story

The R-IEKF’s NEES on real EuRoC ( #365 ) and its accuracy gap ( #429 ).
The time-offset state t_d, the one untested over-confidence candidate ( #332 ).
The dynamic-initialization defects of Chapter 2.
Real-time latency ( #213 ).
Why a front end without border churn costs about 10% EuRoC ATE ( #574 ): the detection margin that fixes it ships off until that is understood.
Turning the real-data consistency checks from warnings into asserted bounds once a backend is consistent on EuRoC, and adding the missing tests.