Develop, compare, and govern models through reproducible workflows, rigorous evaluation, and controlled lifecycle management.
Move systematically from problem framing and baselines to deployable candidates.
train --data sales_v21 --config configs/model.yaml --seed 7 --out runs/2026-11-16
Comparable experiments require captured context, not just final metrics.
| Artifact | Examples | Reason |
|---|---|---|
| Code & config | Commit hash, params, feature set | Recreate training logic |
| Data | Dataset/split versions, sampling rules | Explain result differences |
| Environment | Image/dependencies, hardware, seeds | Reduce nondeterminism |
| Outputs | Metrics, plots, checkpoints, logs | Compare and debug runs |
Reproducibility. The ability to rerun a process with the same inputs and configuration to obtain the same (or statistically consistent) results and trace differences when they occur.
Evaluate for real-world utility, not just leaderboard metrics.
Task metrics (AUC/F1/RMSE/etc.), calibration, confusion analysis, slice-based performance.
Behavior on edge cases, noisy inputs, out-of-distribution samples, and sensitive segments.
Latency, throughput, memory footprint, cost, and dependency requirements.
Online experiments, business KPIs, and decision thresholds aligned with objectives.
Control the lifecycle from candidate artifact to production deployment.
Model registry. A governed store for model artifacts and metadata that tracks versions, lineage, evaluation evidence, and lifecycle stages (e.g. staging, production, archived).
Two models are candidates: Model A has slightly higher offline AUC, Model B has lower latency and more stable performance across key segments. How do you decide?
Progress from baselines to candidates with structured iteration.
Experiments must capture code, data, config, and environment.
Optimize for utility under robustness and operational constraints.
Registries and controlled automation enable safe model evolution.
Next: Chapter 04 — Model Deployment and Serving