Skip to content

Stage Benches

A stage bench is the focused debug, test, characterization and research environment for one transformation. It runs the stage on fixtures, in several arithmetic types and implementation variants, and prints the stage’s invariants in native units. A bench is one small executable, so an edit to a stage costs seconds to re-check, not a EuRoC replay.

  • Debugging: a failing invariant points at the transformation and names the size of the violation.
  • Testing: fixtures are regression tests, and a captured fixture must replay bit for bit.
  • Characterization: declarative sweeps over noise, Δt, parallax, conditioning or window size, written to CSV for the figure pipeline.
  • Research: alternative implementations of the stage (variants) run on identical fixtures, side by side.
  • Mixed precision: the same stage runs in double, float, posit<32,2>, posit<16,1>, long double and strict_double, each compared with the double run. strict_double is double with only explicit conversions out (#449). Any implicit narrowing of T to double is a compile error in it, and its results must equal double’s bit for bit.
KindInputJudged againstMade by
known_answersynthetican analytic expected output, computed independently of the stagehand, in the bench
ground_truthbuilt with truth injected upstream (#266)the true values the output must reproducea synthetic-world run with the mean reset to truth
captureda recorded stage boundarythe output the stage produced in that runcapture(...); from any run with vio_pipeline --capture (#446)

The runner adds two checks to every stage’s own invariants:

  • replay.bit_identical: a captured fixture, run in the type it was captured in, must reproduce its recorded output exactly.
  • known_answer.residual: a known-answer fixture must match within the type’s arithmetic tolerance.

Fixtures are JSON (branes.vio.fixture/1) with an exact codec:

  • numbers go through double, written in shortest round-trip form;
  • non-finite values are written as strings;
  • rotations are restored without renormalizing.

A file reads back bit-identical; the tests prove it for −0, denormals, the largest double, ±∞, NaN and a full captured state.

{"schema": "branes.vio.fixture/1", "stage": "S9_marginalization", "kind": "captured",
"arithmetic": "double", "source": "synthetic_world, S2/S3/S9 stage sequence, frame 6",
"input": {"state": {"R": [...], "p": [...], "clones": [...], "P": {"rows": 39, ...}}, "clone_index": 0},
"expected": {"state": {...}, "dim_before": 39, "dim_after": 33, "removed_time": ...}}
Terminal window
cmake --build build --target s9_marginalization_bench
./build/tools/s9_marginalization_bench # built-in fixtures; double, float, posit32
./build/tools/s9_marginalization_bench --types double,posit16 # choose number systems
./build/tools/s9_marginalization_bench --variant all # shipped vs research variants
./build/tools/s9_marginalization_bench --fixture F.json # e.g. a fixture captured from EuRoC
./build/tools/s9_marginalization_bench --sweep clones=2,4,8 --csv out/
./build/tools/s9_marginalization_bench --precision --csv out/ # bits lost per type (#450)

--precision is the per-stage precision characterization; it is described under Mixed precision. In that mode a failed invariant in another type is reported, not fatal: the exit status is non-zero only when the double reference fails or nothing ran.

The bench prints the stage’s contract from the registry, then one report per fixture, variant and type. This is real output:

captured_stage_sequence [captured] variant=shipped type=double
------------------------------------------------------------------------------
invariant value threshold unit verdict
------------------------------------------------------------------------------
marginalization.dimension 33 33 count PASS
marginalization.principal_submatrix 0 1.386e-13 dimensionless PASS
covariance.symmetric 0 6.541e-13 matrix units PASS
covariance.psd 0 2.829e-12 matrix units PASS
window.clones_minus_1 4 4 count PASS
window.removed_the_requested_clone 0 0 s PASS
mean.untouched 0 0 count PASS
replay.bit_identical 0 0 count PASS
------------------------------------------------------------------------------
max |output - double output| = 0 stage run time (best of 5) = 7.993 us

The same fixture in float reports max |output − double output| ≈ 2e-7, and every invariant still passes, because each threshold scaled to float’s ε. A normal run exits non-zero on any failure, so it can gate CI. A run that executes nothing, for example an unknown --types, also fails rather than reporting a vacuous pass.

The S9 bench sweeps window size × covariance conditioning and reports, per type, the principal- submatrix residual, the smallest eigenvalue after removal, and the run time. A sweep makes a precision limit visible before it bites in a full run:

cloneslog₁₀ κλ_min after (double)λ_min after (posit32)λ_min after (float)
201.0e-21.0e-21.0e-2
241.0e-61.0e-69.5e-7
281.0e-10−4.1e-9−5.2e-8

At κ ≈ 1e8, neither posit<32,2> nor float can resolve the smallest eigenvalue. The matrix is still positive-definite in double, but not measurably so in the narrower types, and float is an order of magnitude further off than posit32. That is exactly the kind of per-stage precision finding the mixed-precision work builds on.

The S0 and S1 benches (#454) turned up three things on their first run, none of which an end-to-end metric had shown:

  • The synthetic world’s camera wasn’t quite EuRoC cam0. Its tangential coefficient p2 was 1000× the EuRoC value (1.76e-2 vs 1.76e-5). The ground-truth fixture caught it as a 0.1 disagreement in normalized coordinates. The world is self-consistent, so nothing downstream failed. Fixed in #467, which re-baselined the synthetic fixtures; the S0 bench now uses the EuRoC intrinsics directly.
  • The S1 seeding stage didn’t compile for posits: an implicit T → double narrowing, now explicit.
  • The dynamic visual-inertial alignment is precision-fragile. It declines entirely in float and posit<32,2> on inputs where it resolves in double. Its solve forms normal equations (squaring κ), adds a fixed 1e-9 ridge, and needs a Cholesky factorization, which breaks down once κ² exceeds 1/ε. That is exactly the kind of per-stage precision finding the mixed-precision work is built to act on.

The S2 and S3 benches (#455) added two more:

  • The propagator didn’t compile for posits either: timestamp += dt mixed T into a double (now explicit, bit-identical for float and double).
  • Propagation leaks the yaw direction, as the body-frame filter is known to do. Φ preserves global translation exactly, but its first-order form, linearized at the current estimate, leaks yaw by ≈2.4e-6 per 5 ms step on a tumbling trajectory. The bench now measures this directly, per step and across Δt and dynamics: the propagation half of the observability question behind the over-confidence (#212).

Two later variants (#471) answer that half:

  • The R-IEKF’s propagation keeps yaw exactly. The invariant variant runs InvariantPropagator and checks it in its own error coordinates, against an unobservable basis that does not depend on the state (translation ξp = eᵢ, yaw ξθ = ẑ). Its yaw leak is 0 on every fixture and at every sweep point. So where the R-IEKF loses to the body-frame filter end to end, the cause is not its propagation.
  • A higher-order body-frame Φ leaks more. With Φ = I + AΔt + ½(AΔt)² the yaw leak grows 3.5–9× on tumbling dynamics (2.2e-5 against 2.4e-6 per 5 ms step), and exp(AΔt) is the same to three digits. The leak comes from linearizing against the mean’s Euler step, not from truncating Φ: extra terms in Φ that the mean doesn’t integrate make it worse. Removing it takes a different error state, not a better Φ.

The S4 and S5 benches (#456) found four more:

  • Border churn in the front end (opt-in fix). KLT drops features whose window doesn’t fit at the coarsest pyramid level, a ≈24 px border band at the defaults, but FAST detects up to the edge. On the bench’s frames over half of the new detections die on their first tracked frame and are re-detected under new ids; on EuRoC V1_01, 81%. FrontendParams::detect_trackable_only seeds only points cv::klt_trackable accepts (the tracker’s own window test on every level): the churn falls to 0 on every fixture (the trackable_detect variant, which enforces at most 5%) and to 2.9% on V1_01. It is off by default because it costs about 10% EuRoC ATE on V1_01 and V2_03 with the body-frame filter, a tie on MH_05, measured over several target_features settings to rule out noise; the mechanism is open (#474, #574).
  • The forward-backward gate is off by default, so drifted tracks reach the backend. The bench measures them, and its fb_gate_1px variant enforces the contract.
  • The camera updater didn’t compile for posits: qualified std::acos/std::sqrt calls, now ADL.
  • The synthetic world steps from rest to ≈1 m/s at the end of its warm-up, a velocity jump no IMU stream can produce. Pure IMU propagation from rest is off by ≈1 m/s after one frame (#475).

Dynamic initialization from images: S1’s SfM front end

Section titled “Dynamic initialization from images: S1’s SfM front end”

The backend’s moving-start init builds its keyframes from images: a two-view bootstrap on the frame with the most parallax, PnP for the rest, then the visual-inertial alignment. The S1 bench now runs that front end on SfM windows (#472): synthetic windows with known truth, and windows captured from EuRoC MH_05 by the S1 tap. Its sweep crosses observation noise, window length and camera speed. It found five things:

  • The eigensolver never converged. It ran 100 sweeps per call; a window build went from 0.55 s to 28 ms once it stopped at convergence (#497).
  • The metric lever arm is added to up-to-scale positions. Even exact data comes out 0.2–5.5 % off in scale (#499).
  • The RANSAC gates are tighter than pixel noise. The two-view gate is ≈0.9 px. At 2 px no window builds, and on EuRoC 82 of 170 attempts fail the two-view while the drone moves. At rest the bootstrap accepts 0.05 px of “parallax” (#500).
  • The alignment accepts badly scaled windows. A track only ~1 % off aligns with a 21–34 % scale error, because try_dynamic checks observability, not consistency (#501).
  • Spacing the keyframes 0.25 s apart helps only in places; the short-baseline SfM track, with no refinement, is the cause.

Since #500 and #501 the backend builds the window with gates from the track noise (track_sigma in the fixture, absent from older ones, which keep the fixed gates) and bundle-adjusts it. On the 0.1 px window that halves the track error (0.77 % → 0.32 %). Its scale is then 8 % off with the accelerometer bias held at zero (was 21 %). Estimating the bias, as the backend does, this window cannot tell a bias along gravity from vertical acceleration. The scale is pulled 66 % low, and the settling gate declines the window. PnP’s gate does not model the landmarks’ triangulation error: on synthetic windows it needs 2–4× the true noise to place every frame.

The MSCKF update, one sub-step at a time: S6

Section titled “The MSCKF update, one sub-step at a time: S6”

S6 is five transformations, and each has its own bench, input and contract: the Jacobians (S6a) against finite differences, the null-space projection (S6b) against NᵀN = I and Nᵀ H_f = 0, the compression (S6c) against the normal equations, the gate (S6d) against χ², and the update (S6e) against K·r and P − K S Kᵀ. Each sub-step’s fixtures come from running the ones before it, so a failure points at one sub-step. The S6 benches (#457) found:

  • The update adds no information along the unobservable directions. At the estimated state as much as at the true one, the projected Jacobian annihilates them (global position, yaw) to ≈1e-16. That rules out this mechanism inside S6; it doesn’t rule out other sources of the #212 over-confidence. The leak happens between updates: each update moves the linearization point the next one is evaluated at. The observability ledger books 95–99% of the spurious yaw information on EuRoC to that, and 1–5% to S2’s Φ (#480; see Capture and composition).
  • The χ² gate was loose at low dof. Inliers are χ²-consistent, but with 1–3 dof the 5-per-dof gate admitted 59–70% of measurements biased by 2σ. The shipped gate is now the 95% χ² quantile, which admits 46% and 22.5% of them and 95–97.5% of inliers at every dof (#481).
  • A square-root filter must carry its factor. The clone blocks are almost perfectly correlated with the IMU pose, so their Schur-complement pivots are below float’s (and posit32’s) resolution of P. Re-factoring the covariance in those types zeroes 36 of 51 pivots, where double zeroes only the 6 exact ones cloning makes, and represents a different covariance; measured that way the square-root update looked ≈330× less precise than Joseph in posit32. The sqrt_array variant therefore factors the fixture’s covariance in double and rounds the factor to T, as a filter that carried its factor would hold it. Then it lands within 1.3–2.3× of Joseph in every type (#482). The S2 and S3 benches’ square-root variants still re-factor P in T.
  • Posits are precise near 1, not near 1e-17. Residuals at the true state and covariance entries near σ² sit where a posit has few fraction bits, so checks on them use T’s precision at that magnitude (safety_at), not its ε at 1.

Calibration as known, as noise, or as state: S10

Section titled “Calibration as known, as noise, or as state: S10”

The S10 bench applies the online-calibration stage to a fresh state with a believed extrinsic off from the world’s true one, runs the filter, and then updates a window of tracks three ways: with the extrinsic estimated (shipped), trusted as fixed, or trusted with its prior folded into the measurement noise. Over one 6-clone window (16 tracks), a constant camera↔IMU rotation error raises NIS only to ≈0.002 per dof at 5° and ≈8e-5 at 1°: the null-space projection absorbs almost all of it into the feature positions, so a wrong calibration is invisible to NIS and to the gate. Folding the prior into R lowers NIS further. Estimated, the extrinsic moves toward the truth, but by under 30% per window (1° → 0.955° with a 2° prior; 5° → 3.98° with a 5° prior), while its covariance stays consistent (NEES 0.49 against a χ²₀.₉₉₉(6) bound of 22.5), so recovery needs many windows of excitation. A calibration error is something the filter has to be told about, not something it will notice. The filter has no time-offset state, so the bench has no t_d sweep.

The S9 window policies (#458) leave the marginalization contract unchanged: removing the oldest or the second-newest clone keeps the principal submatrix exactly. What a policy changes is which information the update had a chance to use first, a pipeline question (#447).

Comparing alternative methods: S5 triangulation

Section titled “Comparing alternative methods: S5 triangulation”

A bench variant can run a different algorithm, not just different parameters. The S5 bench runs three alternatives to the shipped ray-perpendicular solve with Gauss-Newton refinement: the two-view midpoint, the DLT, and anchored inverse depth with Gauss-Newton. All four pass every fixture in every type, and the sweep shows where they differ (mean depth error in m, feature at 5 m):

parallaxpx noiseshippedmidpointDLTinverse depth
0.25°0.52.38fails1.602.41
5°10.1400.1680.1470.140
15°10.04550.05530.04650.0455
  • Inverse depth gives the shipped answer. Both minimize the reprojection error, so the parameterization changes the path, not the answer.
  • The DLT is biased toward the cameras. It wins where depth is barely observable and costs ≈5% where it is observable.
  • The midpoint is the worst. It is also the only one that loses precision with exact observations: its closed form cancels catastrophically at low parallax in float.
  • No method recovers a depth the geometry doesn’t hold. Below about 1° the parallax gate is the answer, not the solver.

The S4 inspector, which draws the front end’s tracks on EuRoC frames, now calls the extracted stage too. What it shows is what the estimator does.

A bench is a struct describing its stage: the I/O types, the codec, run (which calls stages::<sN_name>::apply, with research variants as extra branches), flatten (every output number, for the cross-type and replay comparisons), invariants (the stage’s checks), the built-in fixtures, and optionally a sweep. bench_main<B> turns it into the executable. The S9 bench (tools/include/branes/tools/bench/s9_marginalization_bench.hpp) is the template; the developer reference spells out the full contract.

Measured by touching the stage header, then rebuilding and running only the bench target (i7-12700K, gcc 13.3, -j4):

BuildRebuildRun: double, floatRun: + posit32
default (no -O)4.3 s0.08 s9.1 s
Release (-O3)8.4 s< 0.01 s1.1 s

Posits are software arithmetic, so posit work is best done in a Release build. Either way, a stage edit is re-checked in seconds.

PathWhat
tools/include/branes/tools/bench/bench.hppthe runner, the report, and the CLI (bench_main<B>)
tools/include/branes/tools/bench/fixture.hppthe fixture model, the exact JSON codec, and the capture writer
tools/include/branes/tools/bench/types.hppthe arithmetic types: double, float, posit32 (posit<32,2>), posit16 (posit<16,1>), long_double as the wider-than-double reference, and strict_double (strict_double.hpp, #449)
tools/include/branes/tools/bench/variant.hppthe variant slot: named alternative implementations of a stage
tools/include/branes/tools/bench/sweep.hppdeclarative characterization sweeps and CSV tables
tools/include/branes/tools/bench/s<N>_<transformation>_bench.hppone stage’s bench (the adapter)
tools/benches/s<N>_<transformation>_bench.cppthe bench executable, a one-line main
tools/benches/fixtures/s<N>_<transformation>/*.jsonthe stage’s committed fixtures
tests/tools/stage_bench.cppthe Catch2 integration: the framework’s guarantees, locked

The stage transformations the benches call are in sdk/include/branes/sdk/msckf/stages/ (#452), and the full pipeline is their call sequence, so what a bench establishes about a stage holds where the backend calls it.

BenchIssueVariantsSweep
s0_sensor_model_bench#454shipped (Newton undistortion), fixed_pointdistortion strength × radius → round trip, Jacobian, px per 1° extrinsic / per 1 ms offset
s1_initialization_bench#454, #472shipped (static), gravity_align, dynamic (VI alignment); SfM front end: sfm_two_view_pnp, sfm_adjacent_bootstrap, sfm_keyframes_4hzwindow × IMU noise × excitation → roll/pitch and scale error; observation noise × window length × camera speed for the SfM variants
s2_propagation_bench#455, #471shipped (first-order Φ, diagonal Q_d), canonical_qd, sqrt_covariance, higher_order_phi, invariant (R-IEKF)Δt × dynamics × Q scale → position error (body and invariant), yaw leak (first-order, second-order, exp(AΔt) and invariant Φ), diagonal-vs-canonical position-σ gap, λ_min/‖P‖
s3_augmentation_bench#455shipped, sqrt_covarianceclones × conditioning → block residual, λ_min/‖P‖
s4_frontend_bench#456shipped (FB gate off), fb_gate_1px, klt_window_7image noise × shift → survival, end-point RMS, FB median
s5_triangulation_bench#456shipped (linear + Gauss-Newton), linear_only, parallax_gate_2deg, midpoint_two_view, dlt_linear, inverse_depth_gnparallax × pixel noise → depth error per method, κ, gate rejection
s6a_jacobians_bench#457shippedobservations × depth → H_f, H_x finite-difference error, max |H_f|
s6b_nullspace_projection_bench#457shipped (Householder), givensobservations × depth → NᵀN − I, Nᵀ H_f, H₀·N per method
s6c_compression_bench#457shipped (identity), qrfeatures stacked → rows in/out, normal-equation error
s6d_gating_bench#457, #481shipped (χ²₀.₉₅ quantile), chi2_99, per_dof (5 per dof, the gate before #481), gate_offdof × outlier size → inlier NIS/dof, acceptance per gate
s6e_ekf_update_bench#457shipped (Joseph), sqrt_arraypixel noise × calibration σ → trace ratio, information along N, ‖δx‖, NIS/dof
s9_marginalization_bench#453 (worked example), #458shipped, direct_gather, policy_oldest, policy_second_newestclones × conditioning → residual, λ_min
s10_online_calibration_bench#458shipped (extrinsic estimated), fixed, folded_into_rextrinsic rotation error × prior → remaining error, NIS/dof per treatment
  • The S6 sub-steps share a scene (s6_scene.hpp): the synthetic world run through S2 and S3, so the covariance is the one those stages produce, with each sub-step’s input built by running the shipped upstream sub-steps in double. A sub-step’s bench (S6a_… to S6e_…) prints the S6 contract.
  • Slow stages in software arithmetic set static constexpr int kTimingReps (S2 and S3 use 1).
  • A stage that is not generic in T declares using SupportedTypes = TypeList<...>. S4’s KLT front end runs in double only; #449 decides whether it becomes generic. The S1 SfM variants run only in native types by default (runs_by_default): their RANSAC runs hundreds of eigensolves per window, and an emulated posit is ~10⁴× slower.
  • Image fixtures are stored losslessly as base64 bytes (pack(cv::Image), unpack_image).
  • Square-root variants start from a real filter state with psd_factor, a Cholesky that tolerates the exactly singular covariance a fresh clone produces.

The fixture kinds and the two runner checks are above. The known_answer.residual tolerance is safety · n · ε_T · scale (#445).

  • Variant binding. An expected output can belong to one implementation. The fixture’s variant field names it, and the runner applies the known-answer and replay checks only to that variant; it can also name a list ("shipped,direct_gather"). An empty variant means every variant must reproduce it. Captured fixtures record the variant that ran. S1 needs this: its static and dynamic variants legitimately estimate different things from the same input.
  • Comparisons against double references. Known answers, ground truth and stored rotations all pass through double, so a comparison is never held tighter than double’s ε, even when the stage ran in a wider type (tolerance_vs_double<T>).
  • Checks on small quantities. A posit is most precise near 1 and loses fraction bits as |x| moves away; at 1e-17 posit32 keeps about 14 of its 28. A check on a residual at the true state, or a covariance entry near σ², uses safety_at<T>(magnitude), which scales the safety factor by eps_at<T>(magnitude) / ε_T: T’s relative spacing at that size. It is 1 for IEEE types. Don’t apply it to a lower bound (an SPD margin): a larger safety factor makes those stricter.

A fixture file is one JSON document, schema branes.vio.fixture/1:

{"schema": "branes.vio.fixture/1", "stage": "S9_marginalization", "kind": "captured",
"arithmetic": "double", "source": "…", "seed": 0, "description": "…", "variant": "…",
"input": {…}, "expected": {…}, "truth": {…}}

The payloads use a small vocabulary: numbers, vectors, row-major matrices {"rows","cols","data"}, rotations as unit quaternions [w,x,y,z], and the full MSCKF state (navigation, calibration blocks, clone window, covariance P). The codec is bit-exact for every type whose values fit in a double, which is every type the benches store except long double where it is wider than double (x86 Linux): a fixture written from a long double run keeps only its double part:

  • every number goes through double, which holds every float, posit<32,2> and double value exactly;
  • nlohmann/json writes doubles in shortest round-trip form;
  • non-finite values are written as the strings "nan", "inf" and "-inf";
  • rotations are restored with SO3::from_unit_quaternion, without renormalizing, because renormalizing is not bit-idempotent.

The full contract is the header comment of bench.hpp:

struct S9MarginalizationBench {
static constexpr std::string_view kStage = "S9_marginalization";
static constexpr inv::Stage kInvStage = inv::Stage::S9_marginalization;
static std::vector<Variant> variants(); // "shipped" first
template <class T> struct Input; template <class T> struct Output;
template <class T> static Input<T> decode_input(const json&);
template <class T> static Output<T> decode_output(const json&);
template <class T> static json encode_output(const Output<T>&);
template <class T> static Output<T> run(const Input<T>&, std::string_view variant);
template <class T> static std::vector<double> flatten(const Output<T>&);
template <class T> static std::vector<inv::InvariantResult>
invariants(const Input<T>&, const Output<T>&, const Fixture&);
static std::vector<NamedFixture> builtin_fixtures();
// optional characterization sweep
static Sweep default_sweep();
static std::vector<std::string> sweep_columns();
template <class T> static std::vector<double> sweep_point(const Point&);
};
  • run calls the stage’s transformation, stages::<sN_name>::apply. A research variant is another branch of run, selected by name.
  • invariants returns the stage’s checks from branes/sdk/eval/invariants.hpp. The S9 bench checks the principal submatrix; symmetry and positive-semidefiniteness; the dimension and window bookkeeping; that the mean is untouched; and, for a ground-truth fixture, the kept clones against the truth.
  • flatten lists every output number, for the comparison against double and the replay check.

The report prints the stage contract from the registry (vio_stage_contracts.hpp), then, per fixture, variant and type: every invariant with its value, threshold, unit and verdict; the largest deviation from the double run; and the stage’s run time. --capture DIR writes the built-in fixtures as files.

--csv DIR writes:

  • <stage>_report.csv: one row per fixture × variant × type × invariant;
  • <stage>_sweep_<type>.csv (with --sweep): one per type, columns = the sweep axes followed by the bench’s metrics, ready for the docs-site figure scripts;
  • <stage>_precision.csv (with --precision).

A normal run exits non-zero if any invariant fails, or if nothing ran, so it can gate CI. Under --precision only a failure of the double reference, or an empty run, is fatal.

The benches build on what was already there rather than replacing it at once:

  • sdk/include/branes/sdk/eval/*_probe.hpp: the stage probes (sensor model, initialization, propagation, clone window, frontend, triangulation, update, calibration, observability). Their scenarios became the benches’ fixtures and sweeps in #454–#458.
  • tools/src/s<N>_*.cpp and s<N>_inspect.cpp: the contract printers, the CSV probes and the trace inspectors. A bench prints the same contract from the same registry, and its sweep CSVs feed the same figure scripts.
  • tests/sdk/ stage tests, e.g. msckf_stages.cpp for the stage transformations: these stay as the fast unit tests. The bench’s Catch2 integration covers its fixtures.