Task 1: Ground-Truth Data Schema and Storage Format

Complete · updated 2026-10-30 · graphs/docs/plan/calibration_workflow_refactoring.md

Context

We repeatedly re-run benchmarks on physical Jetson hardware to validate latency estimators. This is slow and wasteful because measurements don’t change between estimator iterations. We need a stable “ground truth” dataset that decouples hardware measurement (run once) from estimator validation (run repeatedly on dev machine).

We already have extensive measurement data: Jetson Orin AGX (4 power modes x 3 precisions x ~15 models), Jetson Orin NX variant, Intel i7-12700K, Ryzen 9 8945HS. The existing v1.0 JSON schema from measure_efficiency.py is 90% there but has critical gaps.

What Changes

1. Schema v2.0 (src/graphs/calibration/ground_truth.py)

New file with dataclasses. Extends v1.0 — all existing fields preserved, new fields added.

New top-level fields (added to measurement JSON):

@dataclass
class MeasurementRecord:
    # --- existing v1.0 fields (unchanged) ---
    schema_version: str           # "2.0"
    measurement_type: str         # "efficiency"
    model: str                    # "resnet18"
    hardware_id: str              # "jetson_orin_agx_maxn"
    device: str                   # "cuda"
    precision: str                # "FP32"
    thermal_profile: Optional[str]  # "MAXN"
    theoretical_peak_flops: float
    measurement_date: str         # ISO 8601
    tool_version: str
    subgraphs: List[SubgraphMeasurement]  # unchanged from v1.0

    # --- new v2.0 fields ---
    batch_size: int               # CRITICAL: was only in filename
    input_shape: List[int]        # e.g., [1, 3, 224, 224]

    model_summary: ModelSummary   # whole-model aggregates
    system_state: SystemState     # hardware conditions during measurement
    run_id: Optional[str]         # UUID for tracking repeat runs
    run_index: Optional[int]      # 1, 2, 3 for repeat sweeps

ModelSummary (derived, computed at save time):

@dataclass
class ModelSummary:
    total_flops: int
    total_bytes: int
    total_latency_ms: float       # sum of subgraph means
    total_latency_std_ms: float   # propagated from subgraphs
    num_subgraphs: int
    throughput_fps: float         # batch_size / (total_latency_ms / 1000)

SystemState (captured during measurement):

@dataclass
class SystemState:
    gpu_clock_mhz: Optional[int]
    mem_clock_mhz: Optional[int]
    power_mode: Optional[str]
    temperature_c: Optional[float]
    cpu_freq_mhz: Optional[float]
    cpu_governor: Optional[str]

Reuse existing SubgraphMeasurement, LatencyStats, EfficiencyStats from measure_efficiency.py, but move their definitions into ground_truth.py so the schema is self-contained in the calibration package.

Add to_dict() / from_dict() / save() / load() methods following the pattern in schema.py.

2. Storage Layout

Consolidate into single canonical location. Keep calibration_data/ as the home:

calibration_data/
  <hardware_id>/                      # e.g., jetson_orin_agx_maxn
    manifest.json                     # index of all available data
    measurements/
      <precision>/                    # fp32, fp16, bf16
        <model>_b<batch>.json         # one file per (model, batch_size)
    efficiency_curves/                # derived data (from aggregate_efficiency.py)
      <precision>/
        curves.json

Key decisions:

• Single directory: no more split between calibration_data/ and measurements/
• Batch in filename: resnet18_b1.json, resnet18_b16.json (AND in JSON)
• Measurements under hardware_id: not under precision first, to keep the tree shallow

3. Manifest Format (manifest.json)

Auto-generated by scanning measurement files. Serves as a fast index:

{
  "schema_version": "1.0",
  "hardware_id": "jetson_orin_agx_maxn",
  "device": "cuda",
  "generated_date": "2026-02-07T...",
  "summary": {
    "num_models": 15,
    "num_precisions": 3,
    "num_batch_sizes": 4,
    "total_measurements": 120
  },
  "measurements": [
    {
      "model": "resnet18",
      "precision": "fp32",
      "batch_size": 1,
      "file": "measurements/fp32/resnet18_b1.json",
      "total_latency_ms": 13.99,
      "total_flops": 3628899328,
      "num_subgraphs": 32,
      "measurement_date": "2026-02-05T..."
    }
  ],
  "coverage": {
    "models": ["resnet18", "resnet34", ...],
    "precisions": ["fp32", "fp16", "bf16"],
    "batch_sizes": [1, 4, 16, 64]
  }
}

4. GroundTruthLoader (src/graphs/calibration/ground_truth.py)

class GroundTruthLoader:
    """Load and query stored measurement data."""

    def __init__(self, data_dir: Path = None):
        """data_dir defaults to <repo>/calibration_data/"""

    def list_hardware(self) -> List[str]:
        """Available hardware IDs."""

    def list_models(self, hardware_id: str, precision: str = None) -> List[str]:
        """Models measured on this hardware."""

    def list_configurations(self, hardware_id: str) -> List[dict]:
        """All (model, precision, batch_size) combos available."""

    def load(self, hardware_id: str, model: str,
             precision: str, batch_size: int) -> MeasurementRecord:
        """Load one measurement. Raises FileNotFoundError if not found."""

    def load_model_summary(self, hardware_id: str, model: str,
                           precision: str, batch_size: int) -> ModelSummary:
        """Load just the summary (fast, from manifest if available)."""

    def load_all(self, hardware_id: str,
                 precision: str = None) -> List[MeasurementRecord]:
        """Load all measurements for a hardware target."""

    def rebuild_manifest(self, hardware_id: str) -> None:
        """Scan measurement files and regenerate manifest.json."""

The loader handles v1.0 backward compatibility internally:

• If schema_version is “1.0”, infer batch_size from filename
• Compute model_summary on the fly if missing
• Leave system_state as None for v1.0 files

5. Changes to cli/measure_efficiency.py

Minimal changes to emit v2.0:

• Accept --batch-size argument (already exists, just not saved to JSON)
• Add batch_size, input_shape to output JSON
• Add model_summary section (computed from subgraphs)
• Add system_state section (query GPU clocks at measurement time)
• Add run_id (UUID) and optional run_index
• Bump schema_version to “2.0”
• Import dataclasses from calibration.ground_truth instead of defining locally

6. Migration Script (cli/migrate_measurements.py)

One-time script to:

1. Scan both calibration_data/ and measurements/ directories
2. For each v1.0 file: read, infer batch_size from filename, compute model_summary, write v2.0
3. Consolidate into canonical layout under calibration_data/<hw_id>/measurements/
4. Generate manifest.json for each hardware_id
5. Report what was migrated

Files to Create/Modify

File	Action	Description
src/graphs/calibration/ground_truth.py	CREATE	Schema dataclasses + GroundTruthLoader
src/graphs/calibration/__init__.py	MODIFY	Export new types
cli/measure_efficiency.py	MODIFY	Emit v2.0 JSON, import from ground_truth
cli/migrate_measurements.py	CREATE	Migration script for existing data
cli/run_full_calibration.py	MODIFY	Use canonical storage path, rebuild manifest

Verification

1. Run migration: python cli/migrate_measurements.py --dry-run to preview
2. Run migration: python cli/migrate_measurements.py to execute
3. Verify: python -c "from graphs.calibration.ground_truth import GroundTruthLoader; loader = GroundTruthLoader(); print(loader.list_hardware()); print(len(loader.list_configurations('jetson_orin_agx_maxn')))"
4. Verify v1.0 compat: loader.load() on an unconverted v1.0 file still works
5. Verify new measurement: run measure_efficiency.py for one model, confirm v2.0 output
6. Run existing tests: python -m pytest tests/ to ensure no regressions