Skip to content

feat: add make e2e and the first end-to-end API tests - #72

Open
Acruve15 wants to merge 3 commits into
pyronear:feat/fake-reolink-camerafrom
Acruve15:alexis/e2e-baseline
Open

Acruve15 wants to merge 3 commits into
pyronear:feat/fake-reolink-camerafrom
Acruve15:alexis/e2e-baseline

Conversation

@Acruve15

@Acruve15 Acruve15 commented Nov 12, 2026 •

Copy link
Copy Markdown

Problem

#65 gives us the fake camera and three scenarios, but nothing waits for the engine to produce an alert and checks what reached the API. No single command runs the chain either, and the expected result of a scenario is only written in prose (#51).

What this adds

  • make e2e SCENARIO=<name>: fresh stack, up -d --wait, pytest on tests/e2e, logs in e2e-artifacts/.
  • An expected block in scenario.json (alerts, sequences, location, timeout). tests/test_scenarios.py validates it offline for every scenario.
  • tests/e2e/: 5 tests reading that block (first alert, alert count, sequences per camera and preset, detection images, location).
  • GET /scenario returns the scenario name -> the tests stop if the cameras play something else than SCENARIO.
  • make test skips tests/e2e and the tests that need a live camera (new integration marker). The CI job skips tests/e2e.

Flow

Example

$ make e2e SCENARIO=smoke-then-clear
...
SKIPPED [1] tests/e2e/test_baseline_api.py:102: scenario does not check the location
============ 4 passed, 1 skipped, 2 warnings in 1479.60s (0:24:39) =============

data/scenarios/smoke-then-clear/scenario.json (7 sequences, one per pose):

"expected": {
  "alerts": 5,
  "sequences": [{"camera": "croix-augas-01", "preset": "1", "min_detections": 15}, ...],
  "timeout_s": 2400
}

Checks

Check Result
Offline suite: pytest tests --ignore=tests/e2e -m "not integration" 29 passed, 6 deselected
make e2e SCENARIO=smoke-then-clear 4 passed, 1 skipped, 1534 s wall
make e2e SCENARIO=never-smoke 1 passed, 4 skipped, 129 s wall
Cameras on one scenario, tests on another fails in 2 s, names both scenarios
black 23.3.0 and ruff 0.0.290, run by hand clean, E501 silenced per file in the 5 new test files
pre-commit hooks not run, the env build failed locally: hadolint, safety and the check-* hooks are missing
Python for the run 3.12 through uv run: tests/requirements.txt installs neither on 3.14 (psycopg2-binary 2.9.9) nor on 3.9 (pytest 9.0.3)

Timings are single runs on an arm64 Mac, app images under amd64 emulation.

What it does not prove yet

  • never-smoke passes but cannot fail: 1 frame per camera, and the engine skips a frame identical to the previous one. Asked on feat: fake Reolink camera driven by scenarios #65.
  • smoke-then-clear expects 5 alerts and 7 sequences because the 7 poses play the same frames (same counts on 3 runs). The "one alert" baseline of Baseline end-to-end test #52 needs one frame folder per preset -> next PR, stacked on this one.
  • false-positive has no expected block yet.
  • The first run failed on a dropped connection while polling the API. With GET retries on the test session the next run was green. Probable cause: uvicorn closes idle connections after 5 s, same as the poll interval. Recurrent: 11 retries in one 600 s wait.

Part of #52, stacked on #65.

A test can now prove which scenario a fake camera is playing before it asserts on it. The four tests that need a live camera on 127.0.0.1:8443 get the integration marker.
…o.json

scenario.json gains an expected block (alerts, sequences, location, timeout_s), validated offline by tests/test_scenarios.py. tests/e2e waits for the first alert, then for every fake camera to finish, and compares what the API returns with that block. Tests that need Postgres or S3 get the integration marker so the offline suite passes with no stack.
Fresh stack, in-repo images built, engine and front profiles up, then the e2e tests. The exit code is kept and the compose logs, ps and junit go to e2e-artifacts/. The three images with no arm64 manifest are pinned to linux/amd64 so the stack starts on Apple Silicon.
@Acruve15

Copy link
Copy Markdown
Author

@MateoLostanlen @fe51 can you review these? I can't request reviews on this repo (read access only). Each one stacks on the previous, so in this order:

  1. feat: add make e2e and the first end-to-end API tests #72 -> make e2e and the API tests
  2. feat: one frame folder per preset and the smoke-one-camera baseline #73 -> one frame folder per preset, the smoke-one-camera scenario
  3. test: first browser tests with Playwright #74 -> first browser tests (Playwright)
  4. ci: reusable e2e workflow and image variables in compose #75 -> draft, the reusable workflow

Before #73 merges, smoke-one-camera.zip has to be copied from the scenarios release of my fork to the one here. #75 stays red until then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant