[autodev] Update configuration and documentation for cycle-1
ci/woodpecker/push/02-build-push Pipeline failed

- Enhanced `.env.example` with detailed CMake build flags and replay-mode strategy flags for development and CI environments.
- Updated `.gitignore` to include a new deploy rollback bookmark.
- Revised `_docs/_autodev_state.md` to reflect the current task status and steps.
- Added new lessons to `_docs/LESSONS.md` regarding testing and architectural improvements.
- Documented changes in `_docs/02_document/deployment/ci_cd_pipeline.md` to reflect the relaxed OpenCV version pin.
- Updated test data documentation in `_docs/02_document/tests/test-data.md` to clarify fixture usage and paths.

This commit continues the cycle-1 documentation sync and addresses various configuration updates for improved clarity and functionality.
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-20 08:05:35 +03:00
parent ab92946833
commit bf13549b32
34 changed files with 3689 additions and 42 deletions
+11 -6
View File
@@ -7,14 +7,17 @@
| `still-image-set-60` | 60 nadir aerial images `AD000001-60.jpg` from `_docs/00_problem/input_data/` with WGS84 frame-center GT in `coordinates.csv` and per-image accuracy table in `expected_results/position_accuracy.csv`. Captured at 400 m AGL with ADTi 20MP 20L V1 (per `data_parameters.md`). Slow cadence (~1 per 2-3 s), so suitable for satellite-anchor frame-center tests, NOT frame-to-frame VIO. | FT-P-01, FT-P-03, FT-P-05, FT-P-06, FT-P-15, FT-P-19, NFT-RES-03 (Monte Carlo), NFT-PERF-04 | Bind-mounted from `_docs/00_problem/input_data/` to `/test-data` in `e2e-runner` (read-only) | None — read-only fixture |
| `still-image-sat-refs-2` | Two paired Google Maps reference images `AD000001_gmaps.png`, `AD000002_gmaps.png`. Insufficient for full satellite-anchor coverage of the 60-image set; supplements the tile-cache fixture for AC-2.1b cross-validation only. | FT-P-05 (subset), FT-P-19 | Same as above | Same |
| `derkachi-fixture` | Cropped nadir flight footage `flight_derkachi/flight_derkachi.mp4` (H.264, 880×720, 30 fps, ~490.07 s = 14,700 frames) plus synchronized FC telemetry `flight_derkachi/data_imu.csv` (4,900 rows @ 10 Hz, columns `timestamp(ms)`, `Time`, `SCALED_IMU2.*`, `GLOBAL_POSITION_INT.*`). Three video frames per telemetry row. The `GLOBAL_POSITION_INT` columns are the trajectory ground truth. | FT-P-02, FT-P-04, FT-P-07, FT-P-10, FT-N-01 (synth on top), FT-N-02, FT-N-03 (synth), FT-N-04 (synth), NFT-PERF-01, NFT-PERF-02, NFT-RES-01, NFT-RES-02, NFT-RES-03 (Monte Carlo), NFT-RES-04, NFT-LIM-02 (8 h synth load loop) | Same bind mount as above | Same |
| `tile-cache-fixture` | Pre-built FAISS HNSW index + tile filesystem covering: (a) the 60 still-image footprints at 0.3-0.5 m/px, (b) the Derkachi route bbox at the same resolution. Built once per CI run by `tests/fixtures/tile-cache-builder/` from the `_gmaps.png` references and from a curated public-data subset (when D-PROJ-3 is resolved — until then, stub-tile content for footprints not paired with `_gmaps.png`). Tile manifest schema per `restrictions.md` § Satellite Imagery. | FT-P-01, FT-P-05, FT-P-15, FT-P-16, FT-P-17, FT-P-19, FT-N-05, FT-N-06, NFT-LIM-03, NFT-PERF-01, NFT-PERF-04, NFT-SEC-01 (poisoning test), NFT-SEC-02 (egress) | Built into named Docker volume `tile-cache-fixture`; mounted read-only into SUT at `/var/azaion/tile-cache` | Volume removed at teardown |
| `synth-age-tile-set` | Two clones of the tile-cache-fixture with manifest `capture_date` field synthetically aged: `synth-age-7mo` (>6 mo, exceeds AC-8.2 active-conflict threshold) and `synth-age-13mo` (>12 mo, exceeds rear threshold). Tile pixels unchanged; only manifest dates differ. | FT-N-05, FT-N-06 | Built from `tile-cache-fixture` by date-mutating script in `tests/fixtures/age-injector/` | Volume removed at teardown |
| `outlier-injection-derkachi` | Synthetic adversarial overlay on `derkachi-fixture`: every Nth frame replaced by a random crop from a far-away tile (>350 m offset, per AC-3.1) to inject a visual outlier. Three injection densities: `light` (1 in 100), `medium` (1 in 10), `heavy` (1 in 3). Generated at runtime by `tests/fixtures/injectors/outlier.py`. | FT-N-01 | Generated at scenario start, written to `tmpfs` in `e2e-runner`, mounted into SUT as a derived frame source | Auto-cleared at teardown (tmpfs) |
| `tile-cache-fixture` | Pre-built FAISS HNSW index + tile filesystem covering: (a) the 60 still-image footprints at 0.3-0.5 m/px, (b) the Derkachi route bbox at the same resolution. Built once per CI run by `e2e/fixtures/tile-cache-builder/` from the `_gmaps.png` references and from a curated public-data subset (when D-PROJ-3 is resolved — until then, stub-tile content for footprints not paired with `_gmaps.png`). Tile manifest schema per `restrictions.md` § Satellite Imagery. | FT-P-01, FT-P-05, FT-P-15, FT-P-16, FT-P-17, FT-P-19, FT-N-05, FT-N-06, NFT-LIM-03, NFT-PERF-01, NFT-PERF-04, NFT-SEC-01 (poisoning test), NFT-SEC-02 (egress) | Built into named Docker volume `tile-cache-fixture`; mounted read-only into SUT at `/var/azaion/tile-cache` | Volume removed at teardown |
| `synth-age-tile-set` | Two clones of the tile-cache-fixture with manifest `capture_date` field synthetically aged: `synth-age-7mo` (>6 mo, exceeds AC-8.2 active-conflict threshold) and `synth-age-13mo` (>12 mo, exceeds rear threshold). Tile pixels unchanged; only manifest dates differ. | FT-N-05, FT-N-06 | Built from `tile-cache-fixture` by date-mutating script in `e2e/fixtures/age-injector/` | Volume removed at teardown |
| `outlier-injection-derkachi` | Synthetic adversarial overlay on `derkachi-fixture`: every Nth frame replaced by a random crop from a far-away tile (>350 m offset, per AC-3.1) to inject a visual outlier. Three injection densities: `light` (1 in 100), `medium` (1 in 10), `heavy` (1 in 3). Generated at runtime by `e2e/fixtures/injectors/outlier.py`. | FT-N-01 | Generated at scenario start, written to `tmpfs` in `e2e-runner`, mounted into SUT as a derived frame source | Auto-cleared at teardown (tmpfs) |
| `blackout-spoof-derkachi` | Synthetic overlay on `derkachi-fixture`: pure-black frames inserted in 5 s / 15 s / 35 s windows AND simultaneous spoofed-GPS injection on the FC inbound stream. Spoof pattern: realistic-looking GPS jumps the trajectory 200-500 m in `north_east_random_direction`. Three windows produce three sub-scenarios per AC-NEW-8. Generated at runtime. | FT-N-04, NFT-RES-04 | Same | Same |
| `multi-segment-derkachi` | Synthetic overlay: 3+ blackout segments distributed across the Derkachi flight to exercise satellite-reference re-localization (AC-3.3) without spoofing. Generated at runtime. | FT-P-08 | Same | Same |
| `cold-boot-fixture` | The state needed to validate AC-NEW-1: a frozen FC pose (`GLOBAL_POSITION_INT` snapshot at flight-resume time) + the tile-cache-fixture + a blank FDR. Test cold-boots the SUT and measures TTFF. | NFT-PERF-03 (AC-NEW-1) | The frozen FC pose is a JSON fixture in `tests/fixtures/cold-boot/`; SUT is restarted (`docker compose restart gps-denied-onboard`) and TTFF is measured from container-ready event to first valid `GPS_INPUT` / `MSP2_SENSOR_GPS` arrival at SITL | Container restart only |
| `mavlink-passkey` | A test-only MAVLink 2.0 signing passkey (32-byte hex). Used for D-C8-9 ArduPilot-track signing channel. NEVER reused outside test environment; checked-in as `tests/fixtures/secrets/mavlink-test-passkey.txt` with explicit comment "TEST ONLY". | FT-P-09 (AP track), NFT-SEC-03 | Loaded via Docker secret into SUT environment | None — fixture file |
| `cve-jpeg-fixture` | Crafted JPEG that triggers CVE-2025-53644 (uninitialized stack pointer → heap buffer write) in OpenCV 4.10/4.11. The pinned ≥4.12.0 must process it without crash and either decode safely or reject. | NFT-SEC-04 | Local-data-only fixture file at `tests/fixtures/security/cve-2025-53644.jpg` (sourced from public PoC, license-checked) | None — fixture file |
| `cold-boot-fixture` | The state needed to validate AC-NEW-1: a frozen FC pose (`GLOBAL_POSITION_INT` snapshot at flight-resume time) + the tile-cache-fixture + a blank FDR. Test cold-boots the SUT and measures TTFF. | NFT-PERF-03 (AC-NEW-1) | The frozen FC pose is a JSON fixture in `e2e/fixtures/cold-boot/`; SUT is restarted (`docker compose restart gps-denied-onboard`) and TTFF is measured from container-ready event to first valid `GPS_INPUT` / `MSP2_SENSOR_GPS` arrival at SITL | Container restart only |
| `mavlink-passkey` | A test-only MAVLink 2.0 signing passkey (32-byte hex). Used for D-C8-9 ArduPilot-track signing channel. NEVER reused outside test environment; checked-in as `e2e/fixtures/secrets/mavlink-test-passkey.txt` with explicit comment "TEST ONLY". | FT-P-09 (AP track), NFT-SEC-03 | Loaded via Docker secret into SUT environment | None — fixture file |
| `cve-jpeg-fixture` | Crafted JPEG that triggers CVE-2025-53644 (uninitialized stack pointer → heap buffer write) in OpenCV 4.10/4.11. The currently-pinned `opencv-python>=4.11.0.86,<4.12` must process it without crash and either decode safely or reject. NFT-SEC-04 also exercises ASan to confirm no buffer overflow. The original D-CROSS-CVE-1 spec required `>=4.12.0`; the pin is held below 4.12 because gtsam==4.2 ships only numpy-1 wheels (see `_docs/_process_leftovers/2026-05-11_d_cross_cve_1_opencv_pin_deferred.md` — D-CROSS-CVE-1 leftover OPEN until upstream gtsam targets numpy>=2). | NFT-SEC-04 | Local-data-only fixture file at `e2e/fixtures/security/cve-2025-53644.jpg` (sourced from public PoC, license-checked) | None — fixture file |
| `sitl-replay-fixture-p01` | Pre-captured FT-P-01 FDR-replay set built by `e2e/fixtures/sitl_replay_builder/build_p01_fixtures.py` from the 60 still images. Contains `outbound_messages_<fc_kind>_<host>.json` (per-image lat/lon emitted by SUT; `null` entries encode timeouts), `observer_<fc_kind>_<host>.json` (sitl_observer config), `stills.mp4` (60-image stitched video), `stationary.tlog` (synthetic stationary IMU/ATTITUDE), `fdr.jsonl` (FDR archive). Activated by `E2E_SITL_REPLAY_DIR=e2e/fixtures/sitl_replay/p01` (see environment.md § Replay-Mode Skip Gating). | FT-P-01 | Pre-committed at `e2e/fixtures/sitl_replay/p01/`; rebuild via `python -m e2e.fixtures.sitl_replay_builder.build_p01_fixtures --input-dir _docs/00_problem/input_data --output-dir e2e/fixtures/sitl_replay/p01 --fc-kind ardupilot --host sitl-host` | None — committed fixture |
| `sitl-replay-fixture-p02` | Pre-captured FT-P-02 Derkachi drift FDR-replay set built by `e2e/fixtures/sitl_replay_builder/build_p02_fixtures.py` from `flight_derkachi.mp4` + `data_imu.csv`. Contains `derkachi.tlog`, `fdr/fdr.jsonl`, `observer_<fc_kind>_<host>.json`. iNav not supported by current builder — ArduPilot only. | FT-P-02 | Pre-committed at `e2e/fixtures/sitl_replay/p02/`; rebuild via `python -m e2e.fixtures.sitl_replay_builder.build_p02_fixtures --derkachi-dir _docs/00_problem/input_data/flight_derkachi --output-dir e2e/fixtures/sitl_replay/p02 --fc-kind ardupilot --host sitl-host` | None — committed fixture |
| `fc-proxy-schedule` | JSON schedule loaded by `e2e/fixtures/injectors/fc_proxy.BlackoutSpoofProxy` to drive FT-N-04 blackout + spoofed-GPS windows on the FC inbound stream. Schedule format: `window_start_ms`, `window_end_ms`, `spoof_pattern` per window. Loaded via `BlackoutSpoofProxy.from_schedule_file(schedule_path)` and replayed by `runner/helpers/fc_proxy_runtime.drive_fc_proxy(...)` (AZ-596). | FT-N-04, NFT-RES-04 | Generated alongside the scenario's `blackout-spoof-derkachi` overlay; written to per-test tmpfs OR pre-captured under `e2e/fixtures/sitl_replay/<scenario>/proxy_schedule.json` when in FDR-replay mode | Auto-cleared at teardown (tmpfs) or committed (FDR-replay) |
## Data Isolation Strategy
@@ -24,6 +27,8 @@ For Tier-2 (Jetson hardware), the same isolation discipline applies but at the s
Synthetic-injection fixtures (`outlier-injection-derkachi`, `blackout-spoof-derkachi`, `multi-segment-derkachi`, `synth-age-tile-set`) are generated into per-test tmpfs and never written back to a persistent volume.
`sitl-replay-fixture-*` and `fc-proxy-schedule` (when in FDR-replay mode) are committed under `e2e/fixtures/sitl_replay/<scenario>/` and read read-only by the replay-mode scenarios. They are not regenerated per test — the builders under `e2e/fixtures/sitl_replay_builder/` are invoked manually (or by a fixture-refresh CI job) when the SUT replay contract changes. When `E2E_SITL_REPLAY_DIR` is unset, the gated scenarios skip cleanly via the `sitl_replay_ready` pytest marker (per AZ-594/595/598/599) and the harness falls back to live-mode (which requires the full Docker compose stack).
## Input Data Mapping
| Input Data File | Source Location | Description | Covers Scenarios |