MLOps · Chapter 06

Monitoring and Maintenance of ML Systems

Sustain production ML performance by observing systems and data, detecting drift, and executing disciplined maintenance and retraining.

5 lecture hoursSyllabus: 6.1–6.4Focus: operations & lifecycle

Chapter outcomes

  1. Implement monitoring and observability for ML systems and services.
  2. Detect and interpret data distribution shift and model drift.
  3. Monitor model performance in production, including delayed feedback cases.
  4. Plan retraining, continual learning, and ongoing system maintenance.
6.1

ML System Monitoring and Observability

Visibility must cover infrastructure, application, data, and model behavior.

Observability signals

System metrics

Latency, throughput, CPU/GPU/memory, error rates, saturation, availability.

Application logs & traces

Request paths, dependency calls, model/version IDs, failures and timeouts.

Data signals

Input schema/quality, missing features, feature distributions, freshness.

Model signals

Prediction distributions, confidence scores, outcome proxies, delayed labels.

SLO-driven monitoring: define service objectives and model/quality guardrails with actionable alerts and runbooks.
6.2

Data Distribution Shift and Model Drift

Models degrade when production reality diverges from training assumptions.

Types of drift and shift

TypeWhat changesExampleDetection signals
Covariate/data driftInput distribution \(P(X)\)New user demographics, sensor calibration changeFeature stats, PSI/KS, schema/quality anomalies
Concept driftMapping \(P(Y|X)\)Fraud tactics evolvePerformance drop on labeled data, outcome shifts
Label/feedback issuesLabel availability/meaningDelayed or biased labelsLabel lag metrics, annotation changes
Upstream pipeline shiftFeature computation/data sourceETL bug, unit changeValidation failures, lineage/canary checks
Interpret carefully: Shift detection indicates investigation need, not always immediate retraining.
6.3

Model Performance and Production Monitoring

Track real-world effectiveness despite partial, delayed, or noisy feedback.

Production performance monitoring

  • Ground-truth metrics: compute task metrics when labels arrive (handle delay windows).
  • Proxy metrics: business KPIs or heuristics when labels are sparse/delayed.
  • Segment monitoring: evaluate by cohort/region/device to catch localized regressions.
  • Stability checks: prediction volume, class rates, confidence calibration trends.
Silent degradation risk: service may remain healthy (low errors, good latency) while predictions become wrong or biased. Monitor quality signals alongside system health.
6.4

Retraining, Continual Learning and System Maintenance

Close the loop with controlled updates and operational upkeep.

Maintenance and retraining strategies

Retraining approaches

  • Periodic: fixed schedule (simple, predictable)
  • Triggered: drift/performance/data thresholds
  • Incremental/continual: update with new data while managing forgetting/stability

System maintenance

  • Dependency/security updates, data pipeline fixes
  • Backfills, schema migrations, artifact cleanup
  • Runbook updates and incident postmortems
Safe update loop: detect → diagnose → retrain/fix → evaluate (offline + canary) → promote or rollback → document. Maintain human oversight for high-impact systems.

Checkpoint

Incident analysis

Prediction distribution shifts significantly, but labels are delayed by two weeks. What actions can you take now?

Answer: Validate upstream data/features and schema, compare with recent baselines by segment, check deployments and pipeline changes, inspect proxy/business metrics, add temporary guardrails/fallbacks if needed, and prepare labeled evaluation/retraining once ground truth arrives.

Summary

1

Observe broadly

Monitor system, data, and model signals together.

2

Understand drift

Different shifts require different diagnostics and responses.

3

Measure in production

Use delayed labels, proxies, and segment-level analysis.

4

Maintain continuously

Controlled retraining and upkeep keep systems reliable over time.

References

  1. Chip Huyen. Designing Machine Learning Systems, monitoring and continual learning.
  2. Evidently AI / monitoring tool guides on drift and production ML observability.
  3. Google SRE practices: monitoring, alerting, and incident response.
End of chapter

Questions?

Next: Chapter 07 — LLMOps and Agentic AI Systems