# Lab 01 — Platform bring-up and system framing

**WatchTower · Chapter 1 · 90 minutes · individual or pairs**

You are the platform team for a facilities company that operates 40 cameras across three sites.
Before anyone trains a model, the company needs a place where footage can land, a catalogue of
cameras, a queue for work, and a written definition of what "working" means.

This lab builds that ground. It is deliberately unglamorous: no model, no detector, no dashboard.
Everything you build here is load-bearing for Labs 02–08.

---

## Learning outcomes

By the end of this lab you can:

1. Stand up a reproducible local AWS environment and prove it is isolated from real AWS.
2. Encode infrastructure as idempotent scripts rather than console clicks.
3. Write an SLI/SLO and an error-cost argument that a non-ML stakeholder can act on.
4. Record an architecture decision with options, trade-offs and accepted consequences.
5. Produce machine-checkable acceptance evidence (`make verify`) instead of screenshots.

---

## Scenario

| Fact | Value |
| --- | --- |
| Sites | 3 (loading bay, main entrance, car park) |
| Cameras | 40 RTSP feeds, 1080p, 15 fps |
| Alerts wanted | "person" and "vehicle" events |
| Alerting budget | p95 from frame captured to operator alert ≤ 60 s |
| Retention | raw frames 30 days, curated clips 12 months, events 24 months |
| Constraint | no audio capture; neighbouring property masked; every alert auditable |

The company has no cloud account for this project yet. You build against a local emulator with the
same API surface as AWS, so the move to a real account later is a configuration change, not a
rewrite.

---

## Before you start

| Tool | Version | Check |
| --- | --- | --- |
| Docker Engine + Compose v2 | 24+ | `docker compose version` |
| AWS CLI v2 | 2.15+ | `aws --version` |
| Python | 3.11+ | `python3 --version` |
| make | any GNU make | `make --version` |
| Git | any | `git --version` |

You need roughly 8 GB of free disk for the emulator image and Python environment; Lab 02 adds
about 5 GB for clips and datasets.

> **No Docker?** You cannot complete the lab. Emulator-based labs require a container runtime;
> talk to the instructor before the session rather than working around it.

---

## 90-minute plan

| Minutes | Activity |
| ---: | --- |
| 0–10 | Read the brief, create the repo, copy `.env.example` |
| 10–30 | Bring up Floci, create buckets/tables/queue, prove isolation |
| 30–50 | Write `docs/requirements.md` (decision, users, SLIs, error costs) |
| 50–70 | Write `docs/adr/0001-system-architecture.md` |
| 70–85 | Seed data, run `make verify`, fix what fails |
| 85–90 | Submit and reflect |

If you finish early, do not add features. Harden the failure paths: what does `make bootstrap` do
when run twice? What does `make verify` say when a bucket is missing?

---

## Step 1 — Create the repository

```bash
mkdir watchtower && cd watchtower
git init
mkdir -p infra src/watchtower scripts tests docs/adr

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
```

Create `pyproject.toml` with `boto3` as a runtime dependency and `pytest`, `moto[s3,sqs,dynamodb]`
and `ruff` as development dependencies, with the package under `src/watchtower`. Install it in
editable mode:

```bash
python -m pip install -e ".[dev]"
```

Write `.gitignore` **before** your first commit. At minimum: `.env`, `.venv/`, `__pycache__/`,
`.pytest_cache/`, `.ruff_cache/`, `*.egg-info/`, `data/`.

---

## Step 2 — Bring up the local cloud

Use Compose (recommended) or the Floci CLI if you already have it.

**Option A — Compose.** Create `compose.yaml` with the `floci/floci:latest` image, port
`4566:4566`, and the Docker socket mounted if you want the web console:

```yaml
services:
  floci:
    image: floci/floci:latest
    ports:
      - "4566:4566"
    environment:
      - DEBUG=0
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
```

Then create `.env` from `.env.example` and start it:

```bash
docker compose up -d
```

**Option B — Floci CLI.**

```bash
floci start
eval $(floci env)
```

Both paths must leave these four variables set in your shell (or in `.env`):

```bash
export AWS_ENDPOINT_URL=http://localhost:4566
export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
```

**Prove isolation before you create anything:**

```bash
aws s3 ls
aws sts get-caller-identity
```

`aws s3 ls` must return quickly with an empty list. If it hangs, or `get-caller-identity` returns
a real account number, `AWS_ENDPOINT_URL` is not set. Stop and fix that first: creating resources
in the wrong account is the classic way to fail this lab.

> Console (optional): <http://localhost:4566/_floci/ui>. It needs the Docker socket mounted.
> If the console is blank, that is a console problem, not a platform problem.

---

## Step 3 — Infrastructure as code

Write `infra/bootstrap.sh` and `infra/teardown.sh`. `make bootstrap` calls the first; `make down`
calls the second before stopping the container.

`bootstrap.sh` must create, if and only if absent:

| Resource | Name | Notes |
| --- | --- | --- |
| S3 bucket | `wt-raw` | source of truth for ingested footage |
| S3 bucket | `wt-curated` | sampled frames, labels, dataset versions |
| S3 bucket | `wt-models` | exported model artefacts |
| DynamoDB table | `wt-cameras` | PK `camera_id` (string) |
| DynamoDB table | `wt-events` | PK `event_id` (string), GSI `camera_id-captured_at-index` |
| SQS queue | `wt-frame-jobs` | main work queue |
| SQS queue | `wt-frame-dlq` | redrive target, `maxReceiveCount` = 5 |
| CloudWatch log group | `/watchtower/lab1` | 7-day retention |

Rules that are graded:

- **Idempotent.** Running it twice prints `=` for existing resources, `+` for created ones, and
  exits 0 both times.
- **Non-interactive.** No prompts, no browser login, no console clicks.
- **Fail loudly.** A missing `aws` binary or unreachable endpoint must exit non-zero with a
  message that says what to check.
- **Reads configuration from `.env`**, not from hard-coded names scattered through the script.

Write `infra/teardown.sh` to reverse it exactly: purge and delete queues, delete tables, empty and
delete buckets, delete the log group. Teardown must also be safe to run when nothing exists.

> **Emulator caveat.** Some Floci builds do not expose CloudWatch Logs. If
> `aws logs create-log-group` fails, set `WT_SKIP_LOGS=1` in `.env`, record the substitution in
> your reflection, and note that production would use a log group per service. Everything else in
> this lab is unaffected.

---

## Step 4 — Adapters, not scripts

Put the AWS calls in `src/watchtower/`, not in shell one-liners:

| Module | Responsibility |
| --- | --- |
| `config.py` | Read `.env`, expose one immutable `Settings` object |
| `storage.py` | S3: ensure buckets, put/get bytes, list keys |
| `events.py` | SQS: ensure queue + DLQ, send, receive, delete, depth |
| `catalog.py` | DynamoDB: ensure tables, cameras and events CRUD, counts |

Design constraints:

- Adapters take an optional client so tests can inject a mocked one.
- No module except `config.py` reads `os.environ`.
- Errors carry the resource name, the operation and a suggested next step.
- Bytes in, bytes out. The storage layer never parses frames or JSON.

Then `scripts/seed_cameras.py` creates **3 cameras** in `wt-cameras` and **5 clip records** under
`s3://wt-raw/clips/<camera_id>/<clip_id>.json`. Clip records carry `captured_at` as ISO-8601 UTC
with an explicit offset — timestamps are data, not decoration. The seed is deterministic: fixed
IDs, fixed timestamps, safe to re-run.

`scripts/verify.py` implements the acceptance checks below and exits `0` on success, `1` on
failure. It is the only thing you submit as proof.

---

## Step 5 — Make it one command

Your `Makefile` must expose exactly this interface, because Labs 02–08 and CI depend on it:

```bash
make up          # start the emulator and wait until it answers
make bootstrap   # create resources (idempotent)
make seed        # deterministic camera and clip data
make verify      # acceptance checks, non-zero exit on failure
make down        # remove resources, stop the emulator
```

Plus `make test` for unit tests and `make lint` for static checks. Every target must work from a
clean clone with no arguments.

---

## Acceptance checks

`make verify` passes only when all of these hold:

1. `make bootstrap` run twice: second run creates nothing and exits 0.
2. Buckets `wt-raw`, `wt-curated`, `wt-models` exist.
3. Tables `wt-cameras`, `wt-events` exist, and `wt-events` has the camera/time GSI.
4. Queue `wt-frame-jobs` exists with a redrive policy pointing at `wt-frame-dlq`.
5. Log group `/watchtower/lab1` exists (or the documented `WT_SKIP_LOGS` substitution).
6. S3 round-trip is byte-exact for a 22-byte object, and the probe object is deleted afterwards.
7. SQS round-trip: one message sent, received, deleted; queue depth returns to 0.
8. Seed produced exactly 3 camera items and 5 clip objects, with timestamps unchanged.
9. `make down` followed by `make up && make bootstrap && make seed && make verify` still passes.
10. `make test` is green and covers the three adapters.

Expected output:

```text
$ make verify
[ok] buckets  wt-raw wt-curated wt-models
[ok] tables   wt-cameras wt-events (+ camera-time gsi)
[ok] queue    wt-frame-jobs (+ dlq, maxReceiveCount 5)
[ok] logs     /watchtower/lab1
[ok] s3       round-trip 22 bytes
[ok] sqs      round-trip 1 message
[ok] seed     3 cameras, 5 clips
PASS lab-01
```

Exit codes are the evidence. A screenshot of the console is not.

---

## Deliverables

| # | Artefact | Where |
| --- | --- | --- |
| 1 | Working repository | Git remote or submitted archive |
| 2 | `make verify` output | `evidence/verify.txt` (captured with `tee`) |
| 3 | Requirements document | `docs/requirements.md` |
| 4 | Architecture decision record | `docs/adr/0001-system-architecture.md` |
| 5 | Reflection, max 300 words | `docs/reflection.md` |

### `docs/requirements.md` must contain

- **The decision being supported.** One paragraph: what the system does, for whom, and what
  happens if it is wrong.
- **Users and jobs.** Operator, site manager, auditor, platform engineer. What each needs.
- **SLIs and SLOs.** At least: ingest freshness, alert latency, event precision/recall targets,
  availability, and cost per camera-month. Each with a measurement point.
- **Error costs.** Quantify a missed event versus a nuisance alert, and show how that changes the
  operating threshold. "False positives are annoying" is not a cost.
- **Data and privacy constraints.** Retention per data class, lawful basis, masking, audit trail,
  deletion path.
- **Out of scope for Lab 01**, explicitly.

### `docs/adr/0001-system-architecture.md` must contain

- Status, date, deciders.
- Context and forces.
- The decision, in one sentence.
- At least two considered options, with the trade-off you rejected and why.
- Consequences: what becomes easy, what becomes hard, what you accept.
- Follow-ups that later labs will resolve.

Minimum content for the decision: local AWS via Floci, S3 as source of truth, DynamoDB for the
camera and event catalogue, SQS for frame jobs, and one configuration object for the endpoint.

---

## Rubric

| Weight | Criterion | Full marks |
| ---: | --- | --- |
| 40% | Reproducibility | A classmate clones, runs `make up && make verify`, gets green with no questions and no leftover state |
| 20% | Design decisions | Requirements and ADR name options, trade-offs and accepted consequences |
| 20% | Code and tests | Small adapters, injected clients, tested failure paths, actionable errors |
| 20% | Reflection | Names what broke, how it was detected, and which control prevents recurrence |

Automatic zeroes: resources created outside `infra/`, credentials committed, `make verify`
hard-coded to print `PASS`, or a clean clone that needs manual steps.

---

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `aws s3 ls` hangs | endpoint not set | `export AWS_ENDPOINT_URL=http://localhost:4566` |
| `Connection refused` on 4566 | emulator not running | `docker compose up -d`, then `make wait` |
| Port 4566 already allocated | another emulator or old container | `docker ps`, stop the other container |
| `BucketAlreadyOwnedByYou` | bootstrap not idempotent | check existence with `head-bucket` before creating |
| `InvalidClientTokenId` | CLI is talking to real AWS | check `AWS_ENDPOINT_URL` and `AWS_ACCESS_KEY_ID=test` |
| Console is blank | Docker socket not mounted | optional; remount `/var/run/docker.sock` |
| `logs` calls fail | emulator build lacks CloudWatch Logs | `WT_SKIP_LOGS=1`, document it |
| Verify passes on your machine only | leftover state | `make down`, then re-run the full loop |

---

## Submission checklist

- [ ] `.env` is git-ignored; `.env.example` is committed.
- [ ] `make up && make bootstrap && make seed && make verify` passes from a fresh clone.
- [ ] `make bootstrap` twice is a no-op the second time.
- [ ] `make down` removes buckets, tables and queues; `make up` recreates them.
- [ ] `make test` green; `make lint` clean.
- [ ] `docs/requirements.md` has SLIs, SLOs, error costs and retention.
- [ ] `docs/adr/0001-system-architecture.md` has options, trade-off, consequences.
- [ ] `docs/reflection.md` is under 300 words and names one thing that broke.
- [ ] `evidence/verify.txt` contains the real output with exit code.

---

## Going further (optional, not graded)

- Add a `Camera` dataclass with validation instead of passing dictionaries around.
- Add a `make ci` target that runs lint, test and verify in that order.
- Add a second `.env.prod.example` that points at real AWS with no endpoint override, and show
  that only configuration changes.
- Write one test that proves `verify.py` returns 1 when a bucket is missing.
