Deliver models as dependable production services through the right serving architecture, packaging, and performance-reliability engineering.
Turn a trained artifact into a stable, integrable prediction capability.
Mirror traffic to new model without user impact; compare outputs and latency.
Gradually shift traffic with guardrails and fast rollback.
Switch between environments for rapid cutover and rollback.
Measure user/business impact; requires experiment design and safeguards.
Inference mode should match product latency, freshness, and throughput needs.
| Pattern | Execution | Latency/freshness | Typical use | Considerations |
|---|---|---|---|---|
| Batch | Scheduled jobs over datasets | High latency tolerance, periodic freshness | Scoring, recommendations, reports | Efficient throughput, storage of outputs, staleness |
| Online (sync) | Per-request API calls | Low latency, on-demand freshness | Fraud checks, ranking, personalization | SLA, autoscaling, dependency latency |
| Real-time/streaming | Event-driven continuous processing | Near real-time reaction | Anomaly detection, monitoring, triggers | State, ordering, retries, exactly-once semantics trade-offs |
Package runtime environments for portability and consistent deployments.
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY model/ inference.py config.yaml ./
CMD ["python", "inference.py"]
Production images should be versioned, scanned, minimal, and configured for secure runtime operation.
Meet SLOs under real traffic while controlling cost and failure impact.
Batching, caching, efficient serialization, quantization/pruning/distillation, hardware acceleration.
Horizontal replicas, autoscaling on load/custom metrics, queue-based async processing, load balancing.
Timeouts, retries with backoff, circuit breakers, fallbacks, health checks, graceful degradation.
Canaries, rollback plans, versioned configs/models, traffic shadowing and monitoring.
An online recommendation API has unpredictable traffic spikes and strict p95 latency targets. What deployment and serving measures would you apply?
Serving includes interfaces, dependencies, and full inference logic.
Batch, online, and real-time fit different product constraints.
Containers and infrastructure enable portable, repeatable deployments.
Optimize, scale, and design for reliability and safe releases.
Next: Chapter 05 — MLOps and Automation