MLOps · Chapter 01

AI System Architecture and ML Lifecycle

Design ML-powered products as production systems: frame the decision, specify the requirements, architect the flow, then study how real systems fail.

5 lecture hoursSyllabus: 1.1–1.4Focus: system thinking5 evidence-based cases

What you should be able to do

  1. Define an ML system, ML system design, a model, and MLOps, and explain why the distinctions matter.
  2. Translate a product objective into a decision, a prediction target, a label, and measurable requirements.
  3. Explain the ML lifecycle and the components and flows of a production reference architecture.
  4. Compare batch, online, and streaming systems on freshness, latency, cost, and failure risk.
  5. Diagnose failures across data, model, system, decision, and organizational layers — and propose controls.
Thread through the chapter: five documented systems — Zillow Offers, Epic Sepsis, a Thailand diabetic-retinopathy deployment, a commercial health-risk algorithm, and Uber ATG Tempe.

Roadmap

1.1

Frame

What decision changes? What does success mean? What constraints are non-negotiable?

1.2

Architect

Which data, model, serving, and control-plane components are required?

1.3

Operate

When must predictions run: on a schedule, on request, or on an event?

1.4

Break

How do systems fail, how do failures compound, and which controls actually help?

Recurring question: “Which part of the system produced this outcome?” rarely has the answer “the model alone.”
1.1

ML System Objectives, Requirements and Design

Start with the decision, not the model. Requirements are the bridge between product intent and architecture.

Definitions that must be precise

TermWorking definitionWhy it matters
AIUmbrella term for systems that perform tasks associated with human intelligence.Too broad to guide engineering decisions.
Machine learningA set of methods that learn behaviour from data rather than from explicitly programmed rules.Shifts the source of correctness from code to data and evaluation.
ModelA learned mapping from inputs to outputs, stored as an artifact with parameters and preprocessing.One component — not the whole product.
ML systemAn integrated software system that combines data, learned models, decision logic, serving infrastructure, feedback, and governance to produce predictions or actions under constraints. [1, 4]Explains why identical models behave differently in different systems.
MLOpsPractices, culture, and tooling that combine ML, DevOps, and data engineering to build, deploy, and maintain ML systems reliably and efficiently. [4, 15]Makes the lifecycle repeatable, observable, and auditable.
A model is not a system. Production behaviour is produced by data, code, configuration, infrastructure, interfaces, people, and incentives acting together.

What is ML system design?

ML system design — the engineering discipline of specifying, building, and operating systems that train, serve, and maintain learned models under production constraints. It joins two concerns: machine learning, which optimises data and model quality, and distributed systems, which constrain latency, throughput, reliability, and cost. [21]

AspectTraditional software systemML system
Core logicHand-written rules and business logicLearned parameters plus a decision policy
DataTransactional and schema-managedHigh-volume and variable; distributions and labels change
Failure signatureVisible: crashes, exceptions, failed testsOften silent: drift, skew, bias, miscalibration
TestingUnit, integration, and acceptance testsOffline evaluation, slice tests, shadow, online experiments
MaintenancePatch code; migrate schemaRetrain, revalidate, repair features, monitor decisions
Probabilistic output, deterministic contract. A prediction may be uncertain; the service level — latency, availability, traceability — is still a hard requirement. [21]

Anatomy of an ML system

  • Data sources and pipelines — events, transactions, sensors, documents, labels, feedback.
  • Features and metadata — transformations, definitions, lineage, point-in-time correctness.
  • Model and artifacts — training code, parameters, preprocessing, evaluation evidence.
  • Serving path — batch jobs, APIs, stream processors, caches, fallbacks.
  • Decision layer — thresholds, business rules, human review, automation level.
  • Operations and governance — monitoring, incident response, access control, audit.
Data Features Model Serving Decision User / product Feedback: outcomes, labels, overrides, complaints, incidents monitoring / retraining loop
Figure 1.0 — The model is one node in a feedback system.

From business objective to ML task

Business outcomee.g. reduce fraud loss Decisionblock / review / allow Prediction targetP(fraud | request) Label & metricchargeback + review cost Requirementslatency, cost, audit Each arrow is a place where the project can be mis-specified. A wrong proxy at this stage cannot be fixed by a better model later.
Design rule: write the decision, the error costs, and the feedback delay before choosing an algorithm. [1, 5, 7]

Is ML the right solution?

Signals that ML may fit

  • The decision repeats at scale.
  • The outcome is measurable and feedback is available.
  • The pattern is stable enough to learn but too complex to hand-code.
  • Inputs are available at prediction time.
  • Errors can be bounded, reviewed, or reversed.

Signals that it may not

  • A simple rule or lookup already meets the objective.
  • Labels are missing, delayed, or expensive.
  • The decision is rare or irreversible.
  • The environment changes faster than the system can learn.
  • Nobody owns the outcome or the failure.
Baseline first: start with the simplest defensible rule, then require ML to beat it on the real decision metric — not just on a model benchmark. [7]

Requirements are multi-dimensional

ClassQuestion to answerExample requirement
FunctionalWhat must the system do?Return a fraud score for every authorization request.
QualityHow good is “good enough”?Recall ≥ 0.80 at precision ≥ 0.90 on the frozen evaluation set.
Latency & throughputHow fast, at what load?p95 ≤ 120 ms; sustain 5,000 requests/s at peak.
ReliabilityWhat happens when it fails?99.95% monthly availability; fall back to rules in ≤ 50 ms.
CostWhat can we spend?≤ $0.002 per scored request; training ≤ $5k per run.
Security & privacyWhat must be protected?No raw card data in logs; PII encrypted at rest and in transit.
GovernanceWhat evidence is required?Decision log with model version; human review above $10k.
Non-functional requirements are architecture drivers. They decide whether you need caching, streaming, redundancy, a feature store, or a human review queue.

Make requirements measurable

Example: card-fraud scoring

  • SLI: measured p95 latency, precision at the operating threshold, alert rate.
  • SLO: p95 ≤ 120 ms; precision ≥ 0.90; false-positive rate ≤ 2%.
  • SLA: contractual availability and incident response, if applicable.
  • Latency budget: gateway 10 ms + features 25 ms + model 50 ms + post-process 10 ms + decision 5 ms.
\[ \text{end-to-end } p95 \le \text{gateway} + \text{features} + \text{model} + \text{post-process} \]

SLI — the measured signal. SLO — the internal target for that signal. SLA — the external commitment, usually with consequences.

A requirement that cannot be measured cannot be tested, monitored, or rolled back safely. [20]

Beware averages: a good mean latency can hide a terrible tail. Specify percentiles and the measurement window.

Five objectives shape every architecture

ObjectiveQuestion the design must answerArchitectural leverExample measure
ScalabilityCan volume grow without a redesign?Stateless inference, sharding, queues, autoscalingPeak requests/s at target latency
Low latencyWhat is the end-to-end budget, including dependencies?Caches, precomputed features, smaller or compiled modelsp95 / p99 response time
ReliabilityWhat happens when a node or dependency fails?Replicas, timeouts, retries with backoff, fallbacksAvailability; fallback rate
AdaptabilityHow fast can the system follow changed data?Versioned data, continuous training, canary, rollbackRetrain-to-production lead time
Explainability & fairnessCan we explain a decision and prove subgroup performance?Lineage, slice metrics, model cards, audit logsSubgroup gap; explanation coverage
These objectives compete. The architecture is the record of which one was traded away, for which workload, and with whose sign-off. [21]

Design trade-offs are unavoidable

Accuracy vs latency / cost

Larger models and richer features can improve quality, but increase compute, memory, and response time.

Freshness vs stability

Frequent updates improve relevance but raise regression risk and operational load.

Automation vs control

Automated decisions scale, but high-impact errors need review, override, and appeal paths.

Personalization vs privacy

User-level features improve fit but increase data sensitivity and regulatory exposure.

Complexity vs maintainability

Bespoke pipelines can fit today's data but become brittle and expensive to change.

Global vs local models

One model is simpler; per-market models fit local behaviour but multiply validation and operations.

Record the decision: options considered, criteria, chosen trade-off, owner, and review date. Undocumented trade-offs become invisible technical debt. [2]

Design review checklist

  1. What decision changes because of this system, and who owns it?
  2. What is the cost of a false positive versus a false negative?
  3. Which inputs are genuinely available at prediction time?
  4. What is the fallback when the model, features, or a dependency is unavailable?
  5. How will we detect that the system is wrong — not just slow or down?
  6. Who can override a prediction, and how is that recorded?
  7. What is the rollback plan, and how quickly can it execute?
  8. What privacy, security, fairness, and audit constraints apply?
  9. What are the unit economics at expected and peak volume?
  10. What evidence is required before launch and before each model change?
Stop rule: if questions 1, 2, and 5 have no clear answer, do not start modelling yet.
1.2

ML Lifecycle and Production System Architecture

Production ML is a loop, not a pipeline that ends at deployment.

The ML lifecycle is iterative

Framerequirements Datadataset Developmodel Evaluatemetrics Deployrelease Monitorsignals feedback, drift, incidents, new data, changed objectives
Each stage has an exit gate. Skipping evaluation or monitoring does not remove the step; it only moves the failure into production. Lifecycle variants exist (CRISP-DM, TDSP, MLOps), but the loop is universal. [1, 4]

Stages, artifacts, and exit gates

StageKey activitiesArtifactsExit gate
FrameDecision analysis, baseline, error costs, constraintsProblem brief, success metrics, risk registerObjective and decision are owned and measurable
DataCollection, labelling, validation, point-in-time featuresDataset version, schema, validation report, lineageData meets quality and availability contracts
DevelopTraining, experiments, error analysis, calibrationModel artifact, config, experiment recordReproducible and beats the baseline
EvaluateHoldout, slices, robustness, latency, costEvaluation report, model card, approval recordMeets quality and operational gates
DeployPackage, integrate, shadow, canary, rollback drillRelease manifest, runbook, dashboardsSafe release and tested rollback
Monitor / improveDetect drift, incidents, feedback, retrainMonitoring records, postmortems, new candidatesClosed loop with owners and thresholds
Gate design: gates should be evidence-based and automated where possible — not meetings that can be skipped under deadline pressure. [3, 4]

Production reference architecture

APPLICATION decisions & UX Product UX Business rules Human review SERVING how predictions run Batch scorer Online API Stream processor MODEL how models are built Training pipeline Evaluation & gates Model registry DATA what the system knows Sources & ingestion Validation & lineage Features & store CONTROL PLANE · orchestration · CI/CD/CT · monitoring · metadata · security · cost training data versioned model predictions requests
Figure 1.1 — A reference architecture, not a vendor stack. Solid arrows: data and model flow. Dashed arrow: requests and feedback. Online features are fetched from the feature store at request time. [4, 15]
Read the diagram by boundaries: each arrow is a contract — schema, version, latency, ownership, and failure behaviour must be defined.

Data plane and control plane

CONTROL PLANE configures, orchestrates, observes, governs Orchestration CI / CD / CT Monitoring Metadata / lineage Security / cost applies policy and observes every layer DATA PLANE moves data, features, predictions, and requests Ingestion Features Training Serving Logs / events

Data plane. The components that actually move and transform data and serve requests. It produces the prediction.

Control plane. The components that configure, schedule, observe, secure, and govern the data plane. It decides how the prediction is produced and whether it is allowed.

Failure pattern: the data plane keeps serving while the control plane has deployed a stale configuration. The system looks healthy and returns wrong answers. [6]

Offline path: training and evaluation

OFFLINE / TRAINING PATH Snapshot Validate Features Split Train Evaluate Register dataset vreportfeature setsplitsmodelmetricsmodel v Reproducible run = versioned data + code + config + environment + randomness
  • Snapshot the data at a consistent point in time; record the version.
  • Validate schema, ranges, freshness, and label quality before training.
  • Build features with the same definitions that will be used in serving.
  • Evaluate on a clean holdout and on critical slices, not just an aggregate metric.
  • Register the artifact with metrics, lineage, and approval evidence.
  • Gate promotion on quality, latency, cost, and safety thresholds.
Reproducibility is a system property: if you cannot rebuild the run from recorded inputs, you cannot debug a regression. [3, 4]

Training at scale: parallelism and checkpoints

StrategyHow work is splitBuys youCosts you
Data parallelismReplicate the model; split each batch; exchange gradients.Throughput beyond one device.Communication grows; the slowest worker sets step time.
Model parallelismSplit layers (pipeline) or operators (tensor) across devices.Models larger than one device.Idle bubbles; cross-device traffic.
Sharded data parallelismPartition parameters, gradients, and optimiser state.Large models, still data-parallel.All-gather traffic; tuning complexity.

Parameter server vs all-reduce

  • Parameter server: workers push gradients to servers that own the weights; asynchronous updates suffer stale gradients and stragglers.
  • All-reduce: workers exchange gradients collectively; synchronous and bandwidth-bound, so all wait for the slowest.

Checkpoints are a requirement, not a nicety

  • Save weights, optimiser state, schedule, RNG, and data position together.
  • Validate a checkpoint by restoring it and reproducing the next step.
  • On preemption, resume instead of restarting days of work.
Scale changes the failure mode: a distributed job often fails slowly, as lost throughput or quiet divergence. [21]

Online path: request-time serving

ONLINE / REQUEST PATH Client Gateway Features Model Post-process Decision —auth · 10 ms25 ms50 ms10 ms5 ms p95 budget ≤ 100 ms
  • Gateway: authentication, rate limiting, request validation, routing.
  • Features: fetch precomputed online features and compute request-time features.
  • Model: run the versioned artifact with its packaged preprocessing.
  • Post-process: calibrate, threshold, apply business rules, format the response.
  • Telemetry: log inputs, model version, prediction, latency, and outcome link.
  • Fallback: if a dependency fails, return a safe default or rule-based decision.
The serving path defines the model's real inputs. If preprocessing differs from training, the artifact is no longer the model you evaluated. [1, 2]

Serving fast, failing soft

Caches

  • Feature cache: entity features keyed by id with a TTL, avoiding a database round trip.
  • Prediction cache: memoised scores for repeated requests; key on model and feature version.
  • Artifact cache: weights and preprocessing held warm in memory; load once, never per request.

Indexes

  • Vector index (HNSW, IVF-PQ): approximate nearest neighbours over embeddings.
  • Inverted index: lexical retrieval over tokens; exact, interpretable, good for rare terms.
  • Hash index: exact key lookup, dedupe, and shard routing in constant time.

Retrieval then ranking

  • Generate cheap candidates from an index.
  • Score only the shortlist with the expensive model.
  • Keeps latency bounded while the model stays large.
Fallback ladder: cached score → smaller model → heuristic rule → safe default or abstain. Each rung has an explicit latency budget and logs why it was taken. [21]
Stale is a correctness bug: a cache that outlives the model or data version serves scores from a model that is gone. [21]

Training–serving skew

Training–serving skew. A difference between the data, features, or preprocessing used in training and those used in serving, causing production performance to diverge from offline estimates.

TRAINING Raw logs Batch featurecode Training set Model SERVING Request Online featurecode Feature vector Model server SKEW CONTROL shared definitions · point-in-time joins contract tests · parity checks

Common causes

  • Separate batch and online feature code.
  • Point-in-time errors: using future information in training.
  • Different defaults for missing values.
  • Library or model-version mismatch.

Controls

  • One feature definition used by both paths.
  • Log training-time feature values and compare with serving.
  • Contract tests and shadow scoring.
  • Package preprocessing with the model.

Feature store, registry, and metadata

Feature store

A system for defining, computing, storing, and serving features consistently to offline training and online inference.

Value: reuse, point-in-time correctness, online/offline parity.

Model registry

A governed catalog of model versions, artifacts, lineage, evaluation evidence, and lifecycle stage.

Value: promotion, rollback, audit, reproducibility.

Metadata & lineage

A record of datasets, jobs, parameters, artifacts, and their relationships across the lifecycle.

Value: impact analysis, debugging, compliance.

These are control-plane services. They reduce skew and make rollback and audit possible; they do not improve the model directly. Document intended use and limitations with model cards and datasheets. [4, 18, 19]

Automation: CI, CD, and CT

PracticeTriggerAutomatesEvidence / gate
CI — continuous integrationCode, config, or schema changeBuild, lint, unit/integration tests, data validationChange is safe to run through the pipeline
CD — continuous delivery / deploymentApproved model or service artifactPackaging, staging deploy, release, rollback setupService passes integration, load, and canary checks
CT — continuous trainingSchedule, new data, drift or performance signalRetraining, evaluation, candidate registrationCandidate beats baseline and passes quality gates
A passing build is not a passing model. CI must validate data and code; CT must validate model behaviour and decision impact; CD must validate the running system. [3, 15, 16]

Promotion path

Dev → staging → shadow → canary → production. Each step has a defined owner, evidence, and rollback trigger.

Rollback path

Versioned model + versioned config + feature compatibility + a tested switch. Rollback is a first-class feature, not an afterthought.

Architecture quality attributes and anti-patterns

Quality attributes to design for

  • Reproducibility: rebuild any run and artifact.
  • Observability: know what happened and why.
  • Modifiability: change one component without breaking others.
  • Scalability: meet peak load without redesign.
  • Security: least privilege, isolation, audit.
  • Cost efficiency: unit economics at expected volume.

Recurring anti-patterns [2]

  • Glue code: bespoke scripts holding the system together.
  • Pipeline jungles: unmanaged, overlapping data paths.
  • Dead experimental code: abandoned branches still running.
  • Configuration debt: hidden settings that change behaviour.
  • Undeclared consumers: downstream users of outputs nobody tracks.
  • CACE: changing anything changes everything.
Technical debt in ML is systemic. It accumulates at data and configuration boundaries, where correctness is hard to see and ownership is often unclear.
1.3

Batch, Online and Real-Time ML Systems

The operating mode is a product and architecture decision, not a default.

Three operating modes

ModeTriggerLatency / freshnessTypical useMain challenge
BatchSchedule or explicit trigger over a datasetMinutes to hours; periodic freshnessNightly scores, reports, recommendations, backfillsStale results; large blast radius
OnlineA request from a client or serviceMilliseconds to seconds; fresh at request timeFraud checks, ranking, personalization, pricingTail latency; dependency availability
Streaming / real-timeAn event or continuous streamMilliseconds to seconds after the eventAnomaly detection, live features, dynamic pricingState, ordering, duplicates, backpressure
Online is not real-time. Online means “on request.” Real-time means “within a bounded freshness relative to events.” A system can be online but use stale features, or streaming but not user-facing. [1, 6]

Batch architecture

Scheduler Snapshot Batch job Predictions App / report model v + config runs on schedule; writes results for later use

Properties

  • Throughput-oriented: process large partitions efficiently.
  • Simple operations: retries, backfills, and replays are straightforward.
  • Staleness: predictions reflect the last run, not the current moment.
  • Blast radius: a bad run can write wrong scores for every entity at once.
  • Controls: freshness SLIs, row-count checks, idempotent writes, output validation.
Use when: decisions tolerate minutes-to-hours of staleness and the cost of per-request inference is unjustified. [6]

Online architecture

Client Gateway Features Model Response online store fallback synchronous request/response; latency budget dominates

Properties

  • Latency-sensitive: p95/p99 matter more than averages.
  • Dependency-heavy: feature stores, caches, databases, and model servers.
  • Elastic: autoscaling, load balancing, rate limits, and queueing.
  • Failure-aware: timeouts, circuit breakers, fallbacks, and graceful degradation.
  • Controls: end-to-end tracing, per-dependency latency, saturation alerts.
Tail latency is a system property: one slow dependency can dominate the user experience even when the model itself is fast.

Streaming architecture

Producer Broker Processor Outputs state store event time, ordering, duplicates, and state are first-class

Properties

  • Event-driven: reacts to a stream rather than a request or schedule.
  • Stateful: windows, joins, and aggregations require managed state.
  • Correctness challenges: late events, out-of-order delivery, duplicates, exactly-once semantics.
  • Operational complexity: consumer lag, backpressure, replay, schema evolution.
  • Controls: watermarks, idempotent sinks, dead-letter queues, lag alerts.
Use when: the value depends on reacting within seconds and the event stream is the natural source of truth. [6]

Hybrid architectures are the norm

PatternTrainingInferenceTypical use
Offline scoringBatchBatchChurn, risk tiers, reporting
Classic online servingBatchOnlineFraud, ranking, recommendations
Fresh-feature servingBatch + streaming featuresOnlinePersonalization, velocity checks, pricing
Streaming decisionsBatch or onlineStreamingAnomaly detection, live monitoring, triggers
Continual learningIncremental / onlineOnline or streamingRare, high-risk; needs strong guardrails
Choose per component, not per company. A batch-trained model can serve online; a streaming feature pipeline can feed an online model; a batch scorer can feed a dashboard.
Do not conflate training freshness with serving freshness. Retraining cadence and feature freshness are independent design choices.

Choosing a mode: a worked example

Questions to settle

  1. How fresh must the decision be: seconds, minutes, hours, or days?
  2. What is the end-to-end latency budget?
  3. What is the peak volume and cost per decision?
  4. Can the product tolerate stale or missing predictions?
  5. What is the operational cost of stateful streaming?
  6. What is the cost of a wrong decision versus a late one?

Card fraud

  • Batch only: nightly analysis; too late to block a transaction.
  • Online: score at authorization; <100 ms; fetch recent velocity features.
  • Streaming: update velocity and merchant-risk features as events arrive; alert on bursts.
  • Decision: hybrid online inference + streaming features + batch retraining.
Rule of thumb: start with the simplest mode that meets the freshness requirement; add streaming only where the value of event-time reaction is measurable. [1, 5]

Failure modes by operating mode

ModeCharacteristic failureEarly signalControl
BatchJob silently fails or writes partial/stale resultsFreshness SLI, row counts, partition completenessData contracts, idempotent writes, output validation, alerting
OnlineTail latency, dependency outage, missing featurep99 latency, error rate, saturation, fallback rateTimeouts, circuit breakers, autoscaling, safe defaults
StreamingOut-of-order or duplicate events corrupt stateConsumer lag, watermark delay, duplicate rateIdempotent sinks, dead-letter queues, replay, schema checks
Observability must match the mode. A batch job needs freshness and completeness checks; an online service needs latency and dependency health; a stream needs lag, ordering, and state integrity. [6, 20]

Batch question

“Is the output complete, current, and valid?”

Online question

“Is the response fast, available, and correct for this request?”

Stream question

“Is the state consistent with the event history?”

1.4

ML System Challenges, Trade-offs and Failure Modes

Failures are usually system failures wearing a model costume.

A failure taxonomy

LayerTypical failuresRoot causeControl
DataMissing/stale data, schema change, label leakage, sampling bias, distribution shiftUpstream change, no contracts, unrepresentative collectionValidation, lineage, point-in-time joins, drift monitoring
ModelOverfitting, miscalibration, shortcut learning, hidden subgroup regressionEvaluation mismatch, spurious correlation, aggregate-only metricsExternal validation, calibration, slice metrics, behavioural tests
SystemTraining–serving skew, latency spikes, dependency outage, version mismatchSeparate code paths, config drift, capacity limitsShared pipelines, contracts, canary, autoscaling, rollback
Decision / productWrong objective, bad threshold, automation complacency, harmful feedback loopProxy metrics, incentives, unclear error costsDecision analysis, human override, A/B tests, thresholds from costs
OrganizationNo owner, no audit trail, no incident process, no external reviewUnclear accountability, weak governanceOwnership, model cards, postmortems, audit, risk review
Use the taxonomy as a checklist: for any incident, name the layer, the evidence, and the control that should have caught it. [2, 5, 17]

Drift, skew, degradation: name the failure

TermWhat changedExampleDetect with
Data drift (covariate shift)The input distribution P(x) moves; the input→label relationship is still valid.Traffic shifts from desktop to mobile; a new customer segment appears.Feature-distribution tests against a training baseline.
Concept driftThe relationship P(y | x) moves, so the same input now implies a different label.Fraud patterns adapt to the current detector; demand shifts after a shock.Delayed-label performance; drift detectors on error residuals.
Label driftHow labels are produced, delayed, or defined changes.Annotation guidelines revised; one label source stops reporting.Label latency, label-distribution checks, audit samples.
Training–serving skewFeatures or preprocessing differ between the two paths.Batch imputes zero, online imputes the mean; library versions diverge.Training-serving parity checks on logged requests.
Degradation is the observable, drift is one explanation. A metric drop can also come from a broken dependency, a stale feature, or a decision-policy change. Diagnose before retraining. [21]
Retraining is a treatment, not a diagnosis. If the cause is skew or a pipeline bug, retraining hides the symptom and ships the bug again. [15, 21]

How failures compound

Small datadefect Feature orlabel shift Modeldegrades Wrongdecision Businessharm delayed feedback no monitoring high automation no fallback Amplifiers turn a small defect into a large, visible failure. The earlier the control, the cheaper the intervention.
Silent failures are the dangerous ones. A wrong prediction delivered on time looks healthy on every system dashboard; only decision- and outcome-level signals reveal it. The Google Flu parable is the canonical case: strong correlations, weak causal mechanism, and no correction loop. [5, 8, 14]

Case 1 · Zillow Offers (2021)

What happened

  • Zillow Offers made automated cash offers for homes, relying on forecasts of home prices 3–6 months ahead. [9]
  • In Q3 2021 it purchased 9,680 homes and ended the quarter with 9,790 in inventory.
  • It recorded a $304 million inventory write-down after buying above its own estimates of future selling prices.
  • The Homes segment lost $422 million before taxes; average gross loss was about $80,771 per home sold.
  • On 2 Nov 2021 it announced a wind-down, a ~25% workforce reduction, and expected a further $240–265 million in Q4 losses.

System view

The system converted a price forecast into an irreversible capital commitment. Forecast error, inventory exposure, and slow feedback compounded: the model was a point estimate, but the business needed a risk-aware decision policy.

Missing controls

Uncertainty-aware offer limits; exposure caps by market; faster mark-to-market feedback; circuit breaker; independent risk review of the decision rule.

Lesson: a point prediction is not a decision policy. When actions are expensive or irreversible, model risk becomes financial risk.

Case 2 · Epic Sepsis Model (2021)

What happened

  • A proprietary sepsis prediction model implemented at hundreds of US hospitals. [10]
  • External validation at Michigan Medicine: 27,697 patients, 38,455 hospitalizations.
  • Hospitalization-level AUC was 0.63 (95% CI 0.62–0.64) — poor discrimination.
  • Sepsis occurred in 2,552 (7%); the model missed 1,709 (67%) of them.
  • It alerted for 6,971 of 38,455 hospitalizations (18%), creating a large alert burden.

System view

Vendor-reported performance did not hold in a new population. A low-yield alert at a high threshold trained clinicians to ignore the system. The failure sits at the model–workflow boundary, not only in the model.

Missing controls

External validation; calibration checks; alert-burden and positive-predictive-value analysis; local threshold governance; a plan for clinician trust and escalation.

Lesson: validation is local. Alert fatigue is a system failure, not a user failure.

Case 3 · Thailand diabetic-retinopathy screening (2020)

What happened

  • A deep-learning system for diabetic eye disease was studied across 11 clinics in Pathum Thani and Chiang Mai over 8 months. [11]
  • Lighting, room layout, camera handling, and patient mobility produced variable image quality.
  • “Ungradable” images created tension between the model's quality thresholds and the nurses' workflow.
  • Nurses had to manage retakes, referrals, and patient expectations; the model's output did not fit the existing process.
  • The study protocol was changed so specialists reviewed ungradable images with records instead of automatic referral.

System view

The input distribution includes the clinic environment and the people operating the camera. Lab accuracy did not imply clinical utility. Data quality is a socio-technical property, not only an image property.

Missing controls

Human-centred evaluation before scale; image-quality feedback; flexible thresholds; workflow co-design; a clear pathway for uncertain results.

Lesson: deployment context is part of the input distribution. Design the workflow, not just the model.

Case 4 · Commercial health-risk algorithm (2019)

What happened

  • A widely used commercial algorithm affected millions of patients by predicting health-care costs to identify those needing extra care. [12]
  • At the same risk score, Black patients were considerably sicker than White patients.
  • The bias arose because the algorithm predicted cost, not illness; unequal access meant less was spent on Black patients.
  • Remedying the disparity would increase the share of Black patients receiving additional help from 17.7% to 46.5%.

System view

The label was a convenient proxy for the true target. Standard accuracy metrics can look good while the objective is wrong. The failure is in problem framing and measurement, not in the optimizer.

Missing controls

Outcome-oriented labels; subgroup impact analysis; fairness review; validation of the proxy against the intended construct; ongoing monitoring of allocation outcomes.

Lesson: predicting a convenient proxy can optimize the wrong thing. Bias can be invisible to aggregate accuracy.

Case 5 · Uber ATG, Tempe (2018)

What happened

  • An automated test vehicle moving at 45 mph struck and killed a pedestrian crossing outside a crosswalk at night. [13]
  • The ADS detected the pedestrian 5.6 seconds before impact but never accurately classified her as a pedestrian or predicted her path.
  • By the time a collision was judged imminent, the situation exceeded the braking response specification.
  • System design precluded emergency braking for collision mitigation; it relied on the operator.
  • The operator was visually distracted by a phone and looked up about 1 second before impact.

System view

Perception, classification, prediction, planning, and actuation form one chain. Detection alone was not enough. The human “backup” was not engaged, and the machine had no independent braking authority.

Missing controls

Safety envelope and fallback manoeuvre; independent emergency braking; driver monitoring; operator oversight; a safety culture that treats automation complacency as a design risk.

Lesson: a system is only as safe as its weakest handoff. Human backup is not a control if the human is not engaged.

Cross-case synthesis

CaseFailed layerWhat looked acceptableMissing control
Zillow OffersDecision / risk + distribution shiftPrice forecast and offer volumeUncertainty-aware policy, exposure limits, kill switch
Epic SepsisModel validation + workflowVendor-reported performanceExternal validation, calibration, alert-burden analysis
Thailand DRData quality + human workflowLab accuracyHuman-centred evaluation, image QC, workflow co-design
Health-risk algorithmLabel / proxy + fairnessCost prediction accuracyOutcome labels, subgroup analysis, fairness review
Uber ATGPerception + safety + human oversightPedestrian detectionClassification/prediction, braking authority, driver monitoring
Pattern: in every case, the model was not the only thing that failed. The missing control was usually at a boundary — data, workflow, decision, or human handoff.

Trade-offs and the cost of errors

\[ \text{Expected cost} = P(\text{error}) \times \text{cost}(\text{error}) \]
DomainFalse positiveFalse negativeDesign implication
Card fraudBlocked legitimate purchase; customer frictionLoss from missed fraudThreshold by transaction amount and review capacity
Sepsis alertAlert fatigue; unnecessary testsMissed deteriorationHigh sensitivity may be justified, but burden must be measured
DR screeningUnnecessary referral, travel, anxietyMissed sight-threatening diseaseUncertain cases need a review pathway, not automatic referral
Credit / hiringWrongful denial or rejectionRisky approval or missed talentLegal and fairness constraints dominate; explainability required
Do not use 0.5 by default. The operating threshold should come from error costs, capacity, and policy — and it should be revisited when the environment changes. [1, 5]

Controls that actually prevent failures

Problem & design

Decision analysis; baseline; pre-mortem; risk register; explicit error costs; owner.

Data

Contracts; validation; lineage; point-in-time correctness; freshness and drift monitors.

Model

External validation; calibration; slice metrics; behavioural tests; model card.

Release

CI/CD/CT; shadow; canary; feature flags; tested rollback; change approval.

Runtime

SLOs; fallbacks; circuit breakers; rate limits; human override; incident runbooks.

Organization

Ownership; on-call; postmortems; audit; independent review; governance.

A dashboard is not a control. A control changes behaviour when a threshold is crossed: it blocks a release, fails a pipeline, routes to a fallback, or stops the action. [3, 17]

Incident response and learning

  1. Detect: an SLI, quality signal, or user report crosses a threshold.
  2. Triage: scope, severity, affected users, and immediate risk.
  3. Mitigate: rollback, fallback, disable the feature, or reduce automation.
  4. Diagnose: data, model, system, decision, or organizational layer?
  5. Learn: blame-free postmortem with a timeline and contributing factors.
  6. Change: assign owners and verify that the control actually works.

Artifacts to keep

  • Decision logs with model and feature versions.
  • Dashboards for system, data, model, and outcome signals.
  • Runbooks with rollback and escalation steps.
  • Incident timeline and postmortem.
  • Action items with owners and due dates.
  • Updated risk register and test cases.
Measure what matters: time to detect, time to mitigate, percentage of incidents caught by monitoring rather than users, and repeat-incident rate. [20]

Self-check

Work through these before moving on
  1. Define an ML system and explain why a model alone is not a system.
  2. Turn “reduce customer churn” into a decision, a prediction target, a label, and two non-functional requirements.
  3. Compare batch, online, and streaming for a card-fraud system: freshness, latency, and failure risk.
  4. Name three causes of training–serving skew and one control for each.
  5. Choose one case study. Which layer failed, what looked healthy, and which control was missing?
  6. Why can a model with high AUC still fail in production? Give two distinct reasons.
  7. Pick three of the five objectives. For a fraud scorer, name the trade-off between two of them and the measure you would report.
  8. Classify each as data drift, concept drift, label drift, or skew: (a) mobile traffic doubles; (b) fraudsters copy the detector; (c) online imputes the mean but batch imputes zero; (d) annotations were redefined.
Hint: answer in terms of decisions, data, architecture, and controls — not only model metrics.

Summary

1

Frame the decision

Objectives, error costs, and feedback delay determine whether ML is appropriate and what “good” means.

2

Requirements become architecture

Latency, reliability, cost, privacy, and governance drive components, contracts, and controls.

3

The lifecycle is a loop

Data, model, deployment, monitoring, and retraining are connected by evidence gates.

4

Mode follows the decision

Batch, online, and streaming trade freshness against latency, complexity, and cost.

5

Failures are systemic

Data, model, system, decision, and organizational layers fail together; controls must be layered.

6

Evidence over intuition

External validation, real-world evaluation, and postmortems are how the system learns.

If you remember one thing: production ML is a system, not a model. Design the system around the decision it changes.

References

  1. Huyen, C. Designing Machine Learning Systems. O’Reilly Media, 2022.
  2. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., Dennison, D. “Hidden Technical Debt in Machine Learning Systems.” Advances in Neural Information Processing Systems (NeurIPS), 2015, pp. 2503–2511.
  3. Breck, E., Cai, S., Nielsen, E., Salib, M., Sculley, D. “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction.” IEEE International Conference on Big Data, 2017, pp. 1123–1132. DOI: 10.1109/BigData.2017.8258038.
  4. Kreuzberger, D., Kühl, N., Hirschl, S. “Machine Learning Operations (MLOps): Overview, Definition, and Architecture.” IEEE Access, vol. 11, 2023, pp. 31866–31879. DOI: 10.1109/ACCESS.2023.3262138.
  5. Paleyes, A., Urma, R.-G., Lawrence, N. D. “Challenges in Deploying Machine Learning: A Survey of Case Studies.” ACM Computing Surveys, vol. 55, no. 6, 2023, pp. 1–29. DOI: 10.1145/3533378.
  6. Kleppmann, M. Designing Data-Intensive Applications. O’Reilly Media, 2017.
  7. Zinkevich, M. “Rules of Machine Learning: Best Practices for ML Engineering.” Google, 2017. https://developers.google.com/machine-learning/guides/rules-of-ml
  8. Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., Bouchachia, A. “A Survey on Concept Drift Adaptation.” ACM Computing Surveys, vol. 46, no. 4, 2014, Article 44. DOI: 10.1145/2523813.
  9. Zillow Group, Inc. “Q3 2021 Shareholder Letter.” 2 November 2021. SEC Exhibit 99.3. https://www.sec.gov/Archives/edgar/data/1617640/000161764021000085/exhibit993.htm
  10. Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M., Singh, K. “External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients.” JAMA Internal Medicine, vol. 181, no. 8, 2021, pp. 1065–1070. DOI: 10.1001/jamainternmed.2021.2626.
  11. Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., Vardoulakis, L. M. “A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy.” Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, pp. 1–12. DOI: 10.1145/3313831.3376718.
  12. Obermeyer, Z., Powers, B., Vogeli, C., Mullainathan, S. “Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations.” Science, vol. 366, no. 6464, 2019, pp. 447–453. DOI: 10.1126/science.aax2342.
  13. National Transportation Safety Board. Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian, Tempe, Arizona, March 18, 2018. Highway Accident Report NTSB/HAR-19/03, 2019. https://www.ntsb.gov/investigations/Pages/HWY18MH010.aspx
  14. Lazer, D., Kennedy, R., King, G., Vespignani, A. “The Parable of Google Flu: Traps in Big Data Analysis.” Science, vol. 343, no. 6176, 2014, pp. 1203–1205. DOI: 10.1126/science.1248506.
  15. Google Cloud. “MLOps: Continuous Delivery and Automation Pipelines in Machine Learning.” Architecture Center. https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
  16. Sato, D., Wider, A., Windheuser, C. “Continuous Delivery for Machine Learning (CD4ML).” Martin Fowler, 2019. https://martinfowler.com/articles/cd4ml.html
  17. Tabassi, E. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, National Institute of Standards and Technology, 2023. DOI: 10.6028/NIST.AI.100-1.
  18. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., Gebru, T. “Model Cards for Model Reporting.” Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*), 2019, pp. 220–229. DOI: 10.1145/3287560.3287596.
  19. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., Crawford, K. “Datasheets for Datasets.” Communications of the ACM, vol. 64, no. 12, 2021, pp. 86–92. DOI: 10.1145/3458723.
  20. Beyer, B., Jones, C., Petoff, J., Murphy, N. R. Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media, 2016.
  21. Ul Haq, F. “ML System Design: A Complete Guide (2026).” System Design Handbook. https://www.systemdesignhandbook.com/guides/ml-system-design/
DOI links resolve at https://doi.org/<DOI>. Web sources were accessed in November 2026.
End of chapter

Questions?

Next: Chapter 02 — Data Engineering for ML Systems