S10 — Online Calibration

The camera and IMU are two rigidly-attached sensors. Fusing them requires knowing exactly how they’re mounted relative to each other (the extrinsic transform T_CI), how the lens maps rays to pixels (intrinsics + distortion), and how their clocks line up (the time offset t_d). S10 is about whether the filter treats those numbers as estimated state with uncertainty — or as perfectly known constants. The cortex filter does the latter by default. This page measures what that costs, and shows with a matched-noise control that the body-frame filter is over-confident even when the calibration is perfect.

What it does

OpenVINS carries the calibration as state, alongside the pose and biases:

• Extrinsics T_CI ∈ SE(3) — 6 parameters per camera.
• Intrinsics + distortion — 8 parameters per camera (radtan).
• Time offset t_d — 1 parameter per camera.

Their Jacobians enter the measurement Jacobian H_x of the MSCKF update, so the filter both corrects them online and, more importantly, carries their uncertainty. The principle is unconditional: even a filter that does not estimate calibration online must still account for its uncertainty — by inflating the measurement noise R, or carrying a fixed prior block — or it treats imperfect calibration as perfect and becomes over-confident.

Why it matters

This is the crux of #212. By default the cortex MSCKF has no calibration states and no calibration term in the noise budget — the EuRoC T_CI is taken as truth. Any real mounting error then has nowhere to go: the filter can’t estimate it away and doesn’t widen its covariance for it, so the resulting reprojection error is silently absorbed into the state while the covariance stays tight. That is the textbook recipe for over-confidence.

The contract

Signature	calibration as state: T_CI (6/cam), intrinsics+distortion (8/cam), t_d (1/cam)
What it does	estimates calibration online (its Jacobians enter H_x); or, if fixed, accounts for its uncertainty in R
Post / invariants	if estimated: calibration states observable only under sufficient excitation (like accel bias) · if fixed: the assumed-perfect calibration’s true uncertainty must appear somewhere in the noise budget
cortex	extrinsics only, off by default — VioConfig::estimate_extrinsics (default false, #332) adds one 6-DoF CalibState block (R_imu_cam, p_imu_cam) per camera with a prior; calib_ext_rot_sigma_deg (default 0) is the alternative R term. No intrinsics or t_d states. With both off (the default), T_CI/t_d/intrinsics are treated as perfectly known

Metrics

The S10 probe measures the hypothesis two ways — an analytic budget and an end-to-end sweep on the real backend.

1. Analytic noise budget — what unmodeled calibration costs in pixels

Given a realistic calibration uncertainty, compute the reprojection error it induces and the equivalent R-inflation. The filter assumes ~4.6 px of image noise; a single degree of extrinsic error already exceeds that:

Extrinsic σ (gentle motion)	induced reprojection	equivalent R inflation
0.5°	4.1 px	1.8×
1.0°	8.1 px	4.1×
2.0°	16 px	13.2×

A 1° extrinsic error (entirely realistic for a hand-measured rig) induces ~8 px — nearly double the assumed noise — for an R-inflation of ~4×. That number is not a coincidence: it’s the same R×4 the EuRoC troubleshooting found restores consistency.

2. End-to-end R-inflation sweep — the exact-sensor control

On the synthetic world with perfect calibration and exact sensors, pose NEES sits below dof = 6 at R×1, and inflating the measurement-noise variance R only makes it more pessimistic:

R scale	pose NEES (target 6)	ATE
×1	0.036	2.3 cm
×4	0.029	1.6 cm
×16	0.024	1.8 cm
×64	0.016	2.5 cm

This sweep once showed NEES ≈ 43 at R×1 (ATE 0.63 m), almost exactly the EuRoC MH_05 value, and was read as the clean setup reproducing the real over-confidence. It was a synthetic artifact: the world left its warm-up with a ≈1 m/s velocity step that no accelerometer sample recorded (#475, fixed in #537). With the world corrected, NEES falls below dof.

That low NEES says the world and the plumbing are self-consistent, and nothing more. The observations and the IMU are exact, while the filter assumes 4.6 px of pixel noise, so it is pessimistic by construction. An earlier version of this page read it as “the clean control is not over-confident”, and concluded that the EuRoC over-confidence needs something the clean world lacks. What the clean world lacks is noise (#551).

3. Matched-noise control — the body-frame yaw leak (#551)

matched_noise_control(seed) runs the same backend on the default synthetic world with perfect calibration and noise at exactly the σ’s the filter assumes: 1 px on every observation (camera_noise_normalized = 1/f), and white IMU noise at VioConfig’s gyro_noise_density and accel_noise_density. It splits the pose NEES by direction, with attitude in the world frame so that yaw (about gravity) separates from roll/pitch, and records the yaw σ at the first and last frame.

seeds 1–10	consistent	measured
pose NEES (6-DoF)	6	10.6–112, median 34
roll/pitch NEES	2	3.9–33.5, median 11.5
yaw NEES	1	0.3–100 (7 of 10 seeds above 6)
position NEES	3	1.4–12.8, median 2.0
yaw σ, first → last frame	never falls	0.16 → 0.010–0.025 rad
ATE	—	13–40 cm

These are measured on the corrected synthetic camera (#467: its p2 was 1000× EuRoC cam0’s) with the χ²-quantile gate (#481). On the old camera the same seeds gave pose NEES 9.0–5088 (median 26), and seed 7 diverged to 126 cm ATE; on the corrected camera with the old 5-per-dof gate, 11.5–288 (median 47) and ATE 12–22 cm. The gate lowered the median and the worst seed but raised the mean ATE (17.5 → 20 cm): this world has no outliers, so the gate can only discard true inliers. The leak signature, yaw σ falling about tenfold on every seed, is unchanged.

Splitting the noise sources (seeds 1–5, matched_noise_control(seed, 1, 0.25, imu_noise, camera_noise)) isolates yaw:

noise	roll/pitch NEES	yaw NEES	position NEES
IMU only (exact camera)	3.8–6.4	50–86	1.8–2.6
camera only	3.9–33.9	2.6–116	1.6–13.1
both	3.9–33.5	4.4–100	1.9–12.8

The body-frame MSCKF is over-confident under matched noise, with perfect calibration, and yaw carries most of it. Global yaw is unobservable in VIO, so its σ can only grow, yet on every seed it falls about tenfold within the first few seconds. With IMU noise alone and exact observations, seed 3’s yaw error grows to 0.23 rad, about 400× what matched gyro noise integrates to over 14 s (≈ 6·10⁻⁴ rad), so the updates themselves pull yaw. That is the signature of the observability leak of #212, reproduced here on a world with known truth for the first time. The magnitude is larger than the textbook linearization effect, so a body-frame Jacobian defect that adds to it is not ruled out. Roll/pitch are over-confident as well (3.9–33.5 against 2, median 11.5; under the 5-per-dof gate before #481 it was 3–7 on 8 of 10 seeds, up to 19 on the other two). A 2° parallax gate did not help (pose NEES 66–91 on seeds 1, 3, 5, measured on the camera before #467).

The R-IEKF on the same kind of noise is consistent (full NEES 0.55–1.83 over 21 seeds, #548), which is what it was built for (#347). FEJ, the classical body-frame fix, was refuted earlier (#280). The body-frame filter stays the EuRoC accuracy reference until the R-IEKF runs on EuRoC (#365) and matches its ATE (#429).

For this stage the control settles one thing: calibration uncertainty is not needed to explain over-confidence. A filter with perfect calibration is already over-confident under noise. The budget above remains a real, additional term on real data. The test S10 matched-noise control: the body-frame MSCKF leaks information into yaw (tests/sdk/calibration_probe.cpp) pins the signature on three seeds: yaw σ ends below a third of its first value, and the median pose NEES is above 9, 1.5× the dof (measured 23.5 on seeds 1–3). The median position NEES stays below 6, 2× its dof (measured 5.1). A fix flips the first two checks. The noise is drawn by Box–Muller from raw mt19937 output, because std::normal_distribution is implementation-defined and MSVC would otherwise draw a different realization for the same seed.

Run the probe

Both measurements come from s10_online_calibration (tools/src/s10_online_calibration.cpp), which drives sdk/include/branes/sdk/eval/calibration_probe.hpp:

cmake --build build -j$(nproc) --target s10_online_calibration
./build/tools/s10_online_calibration --help    # print the S10 contract and exit
./build/tools/s10_online_calibration --no-out  # compute and print only, no files
./build/tools/s10_online_calibration --out DIR # write the CSVs to DIR

R-inflation @ 1deg ext, slow          4.1  x   induced 8.1px vs assumed 4.6px
matched-noise pose NEES, seed 3       112      rp 3.91 (2), yaw 100 (1), pos 12.8 (3)
matched-noise yaw sigma, seed 3       0.025 rad  from 0.16 rad: unobservable, must not fall

Tool	Location	Purpose
Calibration probe	tools/src/s10_online_calibration.cpp + eval/calibration_probe.hpp	noise_budget(...) (pixels → R inflation), r_inflation_sweep() (the exact-sensor NEES sweep) and matched_noise_control(seed) (the consistency control)
Jacobian test	tests/sdk/calib_jacobians.cpp	finite-difference check of the online-calibration Jacobians
Recovery test	tests/sdk/calib_recovery.cpp	online calibration drives a deliberately wrong extrinsic back to truth

The online estimator is VioConfig::estimate_extrinsics (sdk/include/branes/sdk/vio_backend.hpp), with the priors calib_ext_rot_prior_deg (1.0) and calib_ext_trans_prior_mm (10.0); the state block is CalibState and enable_calibration(init, rot_sigma, trans_sigma) in sdk/include/branes/sdk/msckf/state.hpp. Its measurement Jacobians, $\mathbf{H}_{\theta,ic} = [\mathbf{p}_c]_\times$ and $\mathbf{H}_{p,ic} = -\mathbf{R}_{ic}^\top$, are in camera_updater.hpp::project().

There is no YAML or daemon config yet: calibration values are hardcoded in the tools and tests, or set by environment variables in the EuRoC test (tests/sdk/vio_euroc.cpp): CORTEX_ESTIMATE_EXTRINSICS, CORTEX_CALIB_ROT_PRIOR_DEG, CORTEX_CALIB_ROT_SIGMA_DEG. The unified config path is a separate epic (#69). The stage bench s10_online_calibration_bench compares the three treatments over one window.

Real-data inspector (s10_inspect)

The probe above measures the cost of treating calibration as perfectly known. The real-data companion, s10_inspect, exercises the other S10 path — the shipped online extrinsic estimator (estimate_extrinsics) that carries the camera↔IMU extrinsic as state and refines it. It runs the real estimator over a real EuRoC sequence seeded from a deliberately perturbed extrinsic, and tracks the in-state estimate converging toward the dataset’s true calibration. No SDK change — backend().state() exposes the CalibState block.

s10_inspect --dataset /path/to/V1_01_easy/mav0 --out build/s10 --perturb-rot-deg 2 --perturb-trans-mm 30
node docs-site/scripts/gen-calibration-figures.mjs build/s10   # → calib_convergence.svg

It writes calibration.json and renders the convergence figure: the extrinsic-rotation error (deg) and translation error (mm) over time, each inside the filter’s own ±3σ band, decaying toward the reference (0 error). Seeded a couple of degrees / centimetres off, a healthy filter pulls the estimate back to truth while the band shrinks — the in-state remedy for the unmodeled-calibration over-confidence the probe quantifies. The per-frame error/σ read is unit-tested, and calib_recovery proves the operator converges on physically-consistent data.

Interactive

The covariance ellipsoid in the 3-D viewer is this stage made visible: on the sample run the reported position σ collapses to ~13 cm while the true error grows past 2 m. An ellipsoid the ground-truth marker escapes from is the missing calibration uncertainty.

Status

A contributor on real data, not the cause. The analytic budget says calibration uncertainty accounts for an R×4 component on real data (1° extrinsic ≈ 8 px ≈ R×4). But the matched-noise control shows the body-frame filter over-confident with perfect calibration: the cause it exposes is the yaw observability leak (#212, #551), which the R-IEKF (#347) removes by construction. The principled fix is to model the uncertainty — estimate T_CI/t_d online (OpenVINS/MINS-style) or add a calibration term to R — not a blunt global R-scale. Tracked, with the full decomposition, on Engineering Status.