MLOps · Chapter 04

Model Deployment and Serving

Deliver models as dependable production services through the right serving architecture, packaging, and performance-reliability engineering.

6 lecture hoursSyllabus: 4.1–4.4Focus: serving systems

Chapter outcomes

  1. Explain model deployment and serving architectures and their components.
  2. Compare batch, online, and real-time inference patterns.
  3. Describe containerized ML systems and infrastructure considerations.
  4. Apply inference optimization, scaling, and reliability strategies.
4.1

Model Deployment and Serving Architectures

Turn a trained artifact into a stable, integrable prediction capability.

Deployment architecture components

  • Model artifact & runtime: serialized model, inference code, dependencies, config.
  • Interface: REST/gRPC/batch job contract, input/output schema, versioning.
  • Pre/post-processing: feature transformation, validation, decoding, thresholding.
  • Dependencies: feature store, databases, external services, caches.
  • Operations: logging, metrics, tracing, auth, rate limiting.
Common failure: training-serving skew when preprocessing or feature logic differs between environments. Package and test the full inference path, not just the model file.

Release strategies

Shadow deployment

Mirror traffic to new model without user impact; compare outputs and latency.

Canary release

Gradually shift traffic with guardrails and fast rollback.

Blue/green

Switch between environments for rapid cutover and rollback.

A/B testing

Measure user/business impact; requires experiment design and safeguards.

4.2

Batch, Online and Real-Time Inference

Inference mode should match product latency, freshness, and throughput needs.

Inference patterns

PatternExecutionLatency/freshnessTypical useConsiderations
BatchScheduled jobs over datasetsHigh latency tolerance, periodic freshnessScoring, recommendations, reportsEfficient throughput, storage of outputs, staleness
Online (sync)Per-request API callsLow latency, on-demand freshnessFraud checks, ranking, personalizationSLA, autoscaling, dependency latency
Real-time/streamingEvent-driven continuous processingNear real-time reactionAnomaly detection, monitoring, triggersState, ordering, retries, exactly-once semantics trade-offs
4.3

Containerized ML Systems and Infrastructure

Package runtime environments for portability and consistent deployments.

Containers in ML deployment

  • Benefits: reproducible runtime, dependency isolation, portable deployment units, integration with CI/CD.
  • Serving stack: container image, model server/runtime, orchestration platform, networking/ingress.
  • Infrastructure needs: CPU/GPU resources, storage access, secrets/config, observability agents.
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY model/ inference.py config.yaml ./
CMD ["python", "inference.py"]

Production images should be versioned, scanned, minimal, and configured for secure runtime operation.

4.4

Inference Optimization, Scaling and Reliability

Meet SLOs under real traffic while controlling cost and failure impact.

Optimization and scaling techniques

Inference optimization

Batching, caching, efficient serialization, quantization/pruning/distillation, hardware acceleration.

Scaling

Horizontal replicas, autoscaling on load/custom metrics, queue-based async processing, load balancing.

Reliability

Timeouts, retries with backoff, circuit breakers, fallbacks, health checks, graceful degradation.

Release safety

Canaries, rollback plans, versioned configs/models, traffic shadowing and monitoring.

Define SLOs: e.g. p95 latency, availability, error rate, and model-quality guardrails where observable.

Checkpoint

Scenario

An online recommendation API has unpredictable traffic spikes and strict p95 latency targets. What deployment and serving measures would you apply?

Answer: Containerized service with autoscaling, load balancing, efficient model/runtime and caching, dependency timeouts/fallbacks, request batching where appropriate, canary releases, and SLO-based monitoring/alerts with tested rollback.

Summary

1

Architecture matters

Serving includes interfaces, dependencies, and full inference logic.

2

Choose inference mode

Batch, online, and real-time fit different product constraints.

3

Package consistently

Containers and infrastructure enable portable, repeatable deployments.

4

Engineer for SLOs

Optimize, scale, and design for reliability and safe releases.

References

  1. Chip Huyen. Designing Machine Learning Systems, deployment and inference optimization.
  2. KServe/Seldon/BentoML and Kubernetes documentation (serving concepts).
  3. Google SRE principles: SLOs, release engineering, and reliability patterns.
End of chapter

Questions?

Next: Chapter 05 — MLOps and Automation