Sustain production ML performance by observing systems and data, detecting drift, and executing disciplined maintenance and retraining.
Visibility must cover infrastructure, application, data, and model behavior.
Latency, throughput, CPU/GPU/memory, error rates, saturation, availability.
Request paths, dependency calls, model/version IDs, failures and timeouts.
Input schema/quality, missing features, feature distributions, freshness.
Prediction distributions, confidence scores, outcome proxies, delayed labels.
Models degrade when production reality diverges from training assumptions.
| Type | What changes | Example | Detection signals |
|---|---|---|---|
| Covariate/data drift | Input distribution \(P(X)\) | New user demographics, sensor calibration change | Feature stats, PSI/KS, schema/quality anomalies |
| Concept drift | Mapping \(P(Y|X)\) | Fraud tactics evolve | Performance drop on labeled data, outcome shifts |
| Label/feedback issues | Label availability/meaning | Delayed or biased labels | Label lag metrics, annotation changes |
| Upstream pipeline shift | Feature computation/data source | ETL bug, unit change | Validation failures, lineage/canary checks |
Track real-world effectiveness despite partial, delayed, or noisy feedback.
Close the loop with controlled updates and operational upkeep.
Prediction distribution shifts significantly, but labels are delayed by two weeks. What actions can you take now?
Monitor system, data, and model signals together.
Different shifts require different diagnostics and responses.
Use delayed labels, proxies, and segment-level analysis.
Controlled retraining and upkeep keep systems reliable over time.
Next: Chapter 07 — LLMOps and Agentic AI Systems