MLOps · Chapter 03

Model Development, Evaluation and Management

Develop, compare, and govern models through reproducible workflows, rigorous evaluation, and controlled lifecycle management.

6 lecture hoursSyllabus: 3.1–3.4Focus: experimentation to registry

Chapter outcomes

  1. Structure model development and training workflows from baseline to production candidate.
  2. Implement experiment tracking and reproducibility practices.
  3. Evaluate and optimize models using appropriate metrics and constraints.
  4. Manage model versions, registries, and automated training triggers.
3.1

Model Development and Training Workflows

Move systematically from problem framing and baselines to deployable candidates.

Development workflow

  1. Baseline: simple heuristic/rule-based or linear model; establish a meaningful reference.
  2. Data & feature iteration: improve inputs, labels, and representations.
  3. Model iteration: try architectures/algorithms suited to data size, latency, and interpretability needs.
  4. Error analysis: inspect slices, failure cases, calibration, and business-impact errors.
  5. Candidate selection: balance quality, robustness, cost, and operational feasibility.
Practical tip: Prefer improvements that are explainable and maintainable over marginal metric gains with high complexity.

Training workflows: notebook to pipeline

Exploration

  • Interactive notebooks for EDA and prototyping
  • Fast iteration, manual steps
  • Risk of hidden state and irreproducibility

Production training

  • Scripted, parameterized pipeline stages
  • Versioned inputs/outputs and automated runs
  • Testable, schedulable, and auditable
train --data sales_v21 --config configs/model.yaml --seed 7 --out runs/2026-11-16
3.2

Experiment Tracking and Reproducibility

Comparable experiments require captured context, not just final metrics.

What to track

ArtifactExamplesReason
Code & configCommit hash, params, feature setRecreate training logic
DataDataset/split versions, sampling rulesExplain result differences
EnvironmentImage/dependencies, hardware, seedsReduce nondeterminism
OutputsMetrics, plots, checkpoints, logsCompare and debug runs

Reproducibility. The ability to rerun a process with the same inputs and configuration to obtain the same (or statistically consistent) results and trace differences when they occur.

3.3

Model Evaluation and Optimization

Evaluate for real-world utility, not just leaderboard metrics.

Evaluation dimensions

Offline quality

Task metrics (AUC/F1/RMSE/etc.), calibration, confusion analysis, slice-based performance.

Robustness

Behavior on edge cases, noisy inputs, out-of-distribution samples, and sensitive segments.

Operational constraints

Latency, throughput, memory footprint, cost, and dependency requirements.

Product impact

Online experiments, business KPIs, and decision thresholds aligned with objectives.

\[ \text{Select model} = \arg\max_{m \in M} \; U(m) \;\; \text{s.t. constraints}(m) \]

Optimization techniques (overview)

  • Data-centric: better labels, feature quality, augmentation, resampling.
  • Model-centric: hyperparameter tuning, architecture/search, ensembling (with cost awareness).
  • Training efficiency: early stopping, distributed training, mixed precision where supported.
  • Inference-aware optimization: pruning, quantization, distillation, caching-friendly design.
Avoid overfitting the validation process: repeated tuning on the same test set leaks information; preserve a clean holdout/final evaluation protocol.
3.4

Model Versioning, Registry and Automated Training

Control the lifecycle from candidate artifact to production deployment.

Model management and registry

Model registry. A governed store for model artifacts and metadata that tracks versions, lineage, evaluation evidence, and lifecycle stages (e.g. staging, production, archived).

  • Link model to data, code, config, and evaluation results.
  • Support approvals, promotion, rollback, and audit trails.
  • Record signatures/schemas and compatibility constraints.

Automated training triggers

  • Schedule-based (periodic retraining)
  • Data/event-based (new data volume/quality thresholds)
  • Performance/drift-based (degradation signals)
Automated training should be paired with automated evaluation and promotion gates.

Checkpoint

Decision point

Two models are candidates: Model A has slightly higher offline AUC, Model B has lower latency and more stable performance across key segments. How do you decide?

Answer: Use system objectives and constraints (SLAs, cost, fairness/segment stability, business impact). If A violates latency/cost or regresses on critical slices, B may be the better production candidate. Document evidence and promotion rationale in the registry.

Summary

1

Workflow discipline

Progress from baselines to candidates with structured iteration.

2

Track to reproduce

Experiments must capture code, data, config, and environment.

3

Evaluate holistically

Optimize for utility under robustness and operational constraints.

4

Govern lifecycle

Registries and controlled automation enable safe model evolution.

References

  1. Chip Huyen. Designing Machine Learning Systems, model development and evaluation.
  2. MLflow documentation: tracking and model registry concepts.
  3. Google ML testing and production evaluation best practices.
End of chapter

Questions?

Next: Chapter 04 — Model Deployment and Serving