MLOps · Chapter 02

Data Engineering for ML Systems

Build reliable training and inference foundations through disciplined data collection, validation, feature management, and pipeline orchestration.

6 lecture hoursSyllabus: 2.1–2.4Focus: data lifecycle

Chapter outcomes

  1. Design processes for training data collection, preparation, and sampling.
  2. Assess data quality and distribution and implement validation strategies.
  3. Apply feature engineering and feature management practices.
  4. Implement data versioning, pipelines, and workflow orchestration concepts.
2.1

Training Data: Collection, Preparation and Sampling

The right data, prepared correctly, determines what a model can learn.

Data collection and labeling

  • Sources: application logs, databases, sensors, third-party APIs, human labels, synthetic data.
  • Collection strategy: define population, time window, instrumentation, consent/policy constraints.
  • Labeling: guidelines, annotator quality control, inter-annotator agreement, weak supervision where appropriate.
  • Documentation: provenance, known biases, collection conditions.
Common issues: selection bias, non-representative samples, delayed/noisy labels, and feedback loops from prior model decisions. Capture metadata to detect and reason about these issues later.

Preparation and sampling

  • Preparation: cleaning, deduplication, normalization, parsing, handling missing values/outliers.
  • Splitting: train/validation/test with leakage prevention (time-based, group-based splits when needed).
  • Sampling: random, stratified, importance/active sampling, class-imbalance handling.
  • Imbalance strategies: resampling, class weights, threshold tuning, appropriate metrics.

Data leakage. Information unavailable at prediction time influencing training/evaluation, causing overly optimistic results and poor production performance.

Rule: Construct datasets to mirror the real prediction context as closely as possible.
2.2

Data Quality, Distribution and Validation

Detect bad or changed data before it corrupts models and decisions.

Data quality and distribution checks

Check typeExamplesPurpose
Schema/validityTypes, ranges, enums, regex, constraintsCatch format and contract violations
Completeness & freshnessNull rates, missing partitions, ingestion lagEnsure data availability and timeliness
ConsistencyCross-field rules, referential integrityDetect logical contradictions
Distribution/statisticsMean/variance, cardinality, histograms, PSI/KSDetect anomalies and shifts
Duplicate/leakage checksEntity overlap across splits, future fieldsProtect evaluation integrity
Validation strategy: Automate checks in pipelines with severity levels (warn, quarantine, fail) and alerting.
2.3

Feature Engineering and Feature Management

Transform raw data into reusable, consistent signals for training and serving.

Feature engineering practices

  • Transformations: encoding, scaling, binning, aggregations, text/image embeddings, time-window features.
  • Point-in-time correctness: use only information available at prediction time.
  • Feature selection: relevance, redundancy, leakage risk, cost/latency of computation.
  • Shared logic: reuse the same definitions across batch training and online inference to avoid skew.

Feature management. The governance of feature definitions, ownership, computation, storage, discovery, and serving across models and teams.

Feature stores (covered in later systems contexts) help with reuse and online/offline consistency when complexity grows.
2.4

Data Versioning, Pipelines and Workflow Orchestration

Make data transformations reproducible, traceable, and automatable.

Versioning and lineage

  • Version artifacts: raw snapshots, processed datasets, schemas, labels, split definitions.
  • Metadata: source, time window, code commit, parameters, quality results, owner.
  • Lineage: track upstream inputs and downstream consumers for impact analysis and debugging.
dataset: features_v18
source: events_2026_q1
transform: build_features.py@9ac3d1
schema: v5
checks: [schema_ok, freshness_ok, psi_ok]
Ingest Validate Transform Train set Serving feats
Figure 2.1 — Orchestrated pipelines produce traceable datasets and features.

Workflow orchestration concepts

  • DAG-based workflows: explicit task dependencies and execution order.
  • Scheduling & triggers: time-based, event-based, or manual runs.
  • Reliability: retries, backoff, idempotent tasks, checkpointing, alerts.
  • Backfills & reprocessing: safely recompute historical data when logic changes.
  • Observability: task logs, metrics, data quality reports, lineage records.
Pitfall: Non-idempotent pipeline steps make retries and backfills dangerous and can duplicate or corrupt data.

Checkpoint

Scenario

A training job’s performance drops after a pipeline change. Which data engineering artifacts help you diagnose the issue quickly?

Answer: Dataset/snapshot versions, schema and transformation code versions, data validation reports, distribution comparisons, lineage to upstream sources/jobs, and feature definition/serving parity checks.

Summary

1

Data foundation

Collection, preparation, and sampling shape model behavior.

2

Validate continuously

Quality and distribution checks prevent silent failures.

3

Manage features

Consistent definitions reduce leakage and skew.

4

Orchestrate reliably

Versioned, observable pipelines enable reproducibility and recovery.

References

  1. Chip Huyen. Designing Machine Learning Systems, data engineering and feature chapters.
  2. Data validation and quality frameworks (e.g. Great Expectations, TFDV concepts).
  3. Workflow orchestration docs (Airflow, Dagster, Prefect) — DAGs, retries, backfills.
End of chapter

Questions?

Next: Chapter 03 — Model Development, Evaluation and Management