MLOps · Project labs

WatchTower: One System, Eight Labs

A CCTV surveillance platform built incrementally. Every lab ships a working increment and the evidence that it works — in the same repository.

8 labs Chapters 1–8 Local AWS via Floci GPU from Lab 3

Why one project instead of eight exercises

01

A spine

Same repository, same domain, same constraints from Lab 1 to Lab 8. Requirements written in Lab 1 are still load-bearing in Lab 8.

02

Compounding evidence

Each artifact is the next lab's input: dataset → model → service → pipeline → monitors → agent.

03

One demo

Final review is a walkthrough of a running system, not a folder of disconnected notebooks.

Rule: if a lab does not change the running system, it is not a lab — it is homework.

The eight labs

LabChapterYou buildProof you submit
01ArchitectureLocal cloud, repo skeleton, requirements and ADRmake verify green from a clean clone
02Data engineeringFrame ingestion, sampling, labelling, dataset versioningVersioned dataset + quality report
03Model developmentDetector training, tracking, day/night evaluationModel + experiment record + slice metrics
04DeploymentBatch scorer and real-time inference serviceLatency/throughput benchmark under budget
05MLOps automationCI/CD/CT: test, build, retrain, promoteGreen pipeline + auto-registered candidate
06MonitoringDrift, alert quality, dashboards, runbooksDashboard + one real incident postmortem
07LLMOpsIncident summariser and natural-language event queryEval set + cost and latency guardrails
08Scale & responsibilityMulti-camera scale-out, privacy and audit controlsLoad test + privacy impact assessment
Each lab has the same contract: goal → build → run → prove → reflect (a short decision record).

Where each lab plugs in

ONLINE PATH Cameras40 feeds · RTSP IngestL02 · sampling Raw storeL01 · S3 DetectorL03 · L04 Event queueL01 · SQS AlertsL06 · noise OFFLINE PATH FramesL02 · labels DatasetL02 · versioned TrainingL03 · GPU RegistryL05 · MLflow CI / CD / CTL05 LLM analystL07 reuse footage deploy model CONTROL PLANE · orchestration · CI/CD/CT · monitoring · privacy & audit · cost · L05 · L06 · L08 Every box already exists after Lab 01; later labs add behaviour, not new folders. Fatigue, drift and privacy are designed in — not discovered in the demo.
Figure L.1 — Labs 01–08 touch this map in order. Small grey captions name the lab that introduces each component.

The loop every lab uses

1

Up

make up — start the local cloud and services.

2

Bootstrap

make bootstrap — idempotent resource creation.

3

Seed

make seed — realistic cameras and clips, never hand-typed.

4

Verify

make verify — machine-checkable acceptance.

Then teardown: make down. If the lab only works on your laptop because of leftover state, it does not work.
Definition of done: a classmate clones your repo, runs make up && make verify, and gets a green result without asking you anything.

How the labs are graded

WeightCriterionWhat earns full marks
40%ReproducibilityClean clone to green make verify; no hidden state, no manual console clicks.
20%Design decisionsADR states the options, the chosen trade-off, and the consequence the team accepts.
20%Code and testsAdapters are small and tested; failure paths return actionable errors.
20%ReflectionEvidence of what broke, how it was detected, and which control now prevents it.

The system dossier

One docs/ folder accumulates requirements, ADRs, evaluation reports, runbooks and the privacy assessment. It is the revision material for the final walkthrough.

Demo day (Lab 8)

Ten minutes: show an alert, explain the model behind it, show the dashboard, then show what happens when a camera goes dark.

LAB 01

Platform bring-up and system framing

Before any model: a reproducible local cloud, a repository every later lab extends, and a written decision about what “working” means.

Lab 1 brief

Deliverables

  1. Requirements — decision, users, SLI/SLO, error costs, retention and privacy constraints.
  2. ADR 0001 — architecture: local AWS via Floci, S3 as source of truth, DynamoDB catalogue, SQS for frame jobs.
  3. Working platform — make up / bootstrap / seed / verify / down.
  4. Seed data — 3 cameras and 5 clip records with preserved timestamps.

Scenario

A facilities company runs 40 cameras across three sites. They want person and vehicle events with a 60-second alerting budget — and a defensible privacy position.

Not yet

No model training in this lab. You are building the ground the model will stand on.

Budget: 90 minutes · setup 30 · requirements 25 · ADR 20 · verification 15.

Lab 1 setup

ToolWhy
Docker + Compose v2Runs the emulator and, later, your services.
FlociLocal AWS on :4566 — no account, no token.
AWS CLI v2Same commands you would run against real AWS.
Python 3.11 + venvAdapters, seed and verification scripts.
makeOne interface for the whole lab series.
# Option A — Floci CLI
floci start
eval $(floci env)

# Option B — Docker Compose
docker compose up -d
export AWS_ENDPOINT_URL=http://localhost:4566
export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test

aws s3 ls          # must return, not hang
Check before you build: if aws s3 ls reaches real AWS, AWS_ENDPOINT_URL is not exported — stop and fix it. The console is at localhost:4566/_floci/ui (needs the Docker socket).

Lab 1 — files to create

watchtower/
├── compose.yaml          floci :4566
├── Makefile              up·bootstrap·seed·verify
├── .env.example          endpoint, names
├── pyproject.toml        boto3, pytest, ruff
├── infra/
│   ├── bootstrap.sh      idempotent resources
│   └── teardown.sh
├── src/watchtower/
│   ├── config.py         env-driven settings
│   ├── storage.py        S3 put/get/list
│   ├── events.py         SQS send/receive
│   └── catalog.py        DynamoDB cameras/events
├── scripts/
│   ├── seed_cameras.py   3 cameras · 5 clips
│   └── verify.py         acceptance, non-zero on fail
├── tests/
└── docs/
    ├── requirements.md
    └── adr/0001-system-architecture.md

What each layer proves

  • infra/ — resources can be created twice without failing; that is idempotency, not luck.
  • src/ — thin adapters. One place knows the endpoint, so Labs 4–8 inherit it.
  • scripts/ — seed and verify are code, so they run in CI unchanged.
  • docs/ — the decision record the demo will be judged against.
Resource model: S3 wt-raw, wt-curated, wt-models; DynamoDB wt-cameras, wt-events; SQS wt-frame-jobs + DLQ; log group /watchtower/lab1.

Lab 1 — verify and submit

Acceptance checks

  • make bootstrap twice: second run is a no-op, not an error.
  • 3 buckets, 2 tables, 1 queue + DLQ, 1 log group exist.
  • S3 round-trip is byte-exact; the test object is deleted.
  • SQS send → receive → delete leaves depth at 0.
  • Seed gives exactly 3 cameras and 5 clips with timestamps intact.
  • make down removes everything; make test is green.

Full brief: labs/lab-01.md

$ make verify
[ok] buckets  wt-raw wt-curated wt-models
[ok] tables   wt-cameras wt-events (+ camera-time gsi)
[ok] queue    wt-frame-jobs (+ dlq, maxReceiveCount 5)
[ok] logs     /watchtower/lab1
[ok] s3       round-trip 22 bytes
[ok] sqs      round-trip 1 message
[ok] seed     3 cameras, 5 clips
PASS lab-01

Troubleshooting

Port 4566 busy → stop the other emulator. CLI hits real AWS → export AWS_ENDPOINT_URL. Console dead → Docker socket not mounted (optional).

Submit: repo link + the output of make verify + your ADR. Screenshots are not evidence; exit codes are.
End of Lab 01

Platform ready

Next: Lab 02 — Video ingestion and the dataset factory