gps-denied-onboard/_docs/02_document/tests/test-data.md

Test Data Management

Seed Data Sets

Data Set	Description	Used by Tests	How Loaded	Cleanup
project_60_still_images	60 nadir images with WGS84 frame-center coordinates from coordinates.csv; height 400 m	FT-P-01, FT-P-02, FT-N-01, NFT-PERF-01	Mounted from _docs/00_problem/input_data/	Read-only
project_gmaps_reference_subset	Google Maps reference images available for the first sample frames	FT-P-02, FT-N-01	Mounted from _docs/00_problem/input_data/	Read-only
expected_frame_centers	Expected lat/lon and thresholds derived from coordinates.csv	FT-P-01, FT-P-02	_docs/00_problem/input_data/expected_results/results_report.md	Read-only
derkachi_video_telemetry	Cropped nadir MP4 synchronized with IMU and GLOBAL_POSITION_INT trajectory: 880 x 720, 30 fps, ~490.07 s; telemetry 10 Hz, 4,900 rows	FT-P-03, NFT-PERF-02, NFT-RES-02	Mounted from _docs/00_problem/input_data/flight_derkachi/	Read-only
public_nadir_vio_candidates	MUN-FRL, ALTO, EPFL fixed-wing, Kagaru, AerialVL/VPAir slices, EuRoC/UZH FPV proxy slices	FT-P-03, FT-P-04, NFT-PERF-02, NFT-RES-02	Downloaded or mounted by replay harness; exact files pinned during implementation	Reset fixture volume
sitl_spoofing_scenarios	Generated ArduPilot Plane SITL GPS loss/spoofing traces	FT-N-02, NFT-RES-01, NFT-SEC-03	Generated by test harness	Discard generated logs after report
cache_integrity_fixtures	Fresh, stale, unsigned, hash-mismatched, and low-resolution cache manifests	FT-N-03, NFT-SEC-01, NFT-SEC-02	Mounted fixture volume	Read-only

Public Dataset Coverage Plan

Public Data Source	Fit For This Project	Limitations	Planned Use
MUN-FRL	Strong nadir camera + IMU + GNSS/ground truth candidate	Helicopter/hexacopter, not fixed-wing	BASALT/OpenVINS/Kimera replay and covariance calibration
ALTO	Strong nadir aerial imagery with GPS/INS, altimeter, orthophotos	Helicopter/airborne collection, access/details must be pinned	VPR, satellite alignment, VO/geolocalization replay
EPFL fixed-wing micro UAV	Strong fixed-wing relevance with camera/navigation sensors	Availability and exact raw IMU packaging must be verified	Fixed-wing path realism and photogrammetry-style validation
Kagaru airborne vision	Fixed-wing/farmland relevance, downward stereo, INS/GPS	Older dataset; exact sensor compatibility must be verified	Agricultural terrain and fixed-wing motion checks
AerialVL	Strong UAV-to-satellite localization and VPR benchmark	IMU availability is less clear than image/GNSS/reference-map data	Satellite retrieval, anchor verification, visual localization
VPAir	Strong aircraft nadir VPR/localization with GPS-derived poses	Academic-use restriction; raw IMU not confirmed	VPR and cross-view localization only if license allows
EuRoC MAV	Excellent synchronized camera/IMU/ground-truth VIO benchmark	Not fixed-wing nadir, indoor MAV	BASALT/OpenVINS/Kimera baseline sanity tests
UZH FPV	Synchronized camera/IMU/ground-truth high-dynamics benchmark	Not nadir fixed-wing; non-commercial license	Stress VIO robustness only if license allows

Data Isolation Strategy

Every replay test uses read-only fixture mounts and writes results to a fresh test-results/<run-id>/ directory. The system under test may write FDR and generated COG tiles only to run-scoped temporary volumes.

Input Data Mapping

Input Data File	Source Location	Description	Covers Scenarios
AD000001.jpg ... AD000060.jpg	_docs/00_problem/input_data/	Project still-image set with expected WGS84 centers	FT-P-01, FT-P-02, NFT-PERF-01
coordinates.csv	_docs/00_problem/input_data/coordinates.csv	Machine-readable expected frame centers	FT-P-01, FT-P-02
data_parameters.md	_docs/00_problem/input_data/data_parameters.md	Height 400 m and camera model	FT-P-01, NFT-PERF-01
AD000001_gmaps.png, AD000002_gmaps.png	_docs/00_problem/input_data/	Reference map screenshots for sample sanity checks	FT-P-02
flight_derkachi/flight_derkachi.mp4 + flight_derkachi/data_imu.csv	_docs/00_problem/input_data/flight_derkachi/	Cropped nadir video synchronized with IMU and GLOBAL_POSITION_INT GPS trajectory	FT-P-03, NFT-PERF-02, NFT-RES-02
Public dataset slices	External fixture paths pinned during implementation	Synchronized camera/IMU/GNSS/ground truth where available	FT-P-03, FT-P-04, NFT-PERF-02, NFT-RES-02

Expected Results Mapping

Test Scenario ID	Input Data	Expected Result	Comparison Method	Tolerance	Expected Result Source
FT-P-01	AD000001.jpg ... AD000060.jpg	Output WGS84 frame center per mapped row; >=80% within 50 m, >=50% within 20 m	Haversine distance threshold + aggregate pass rate	50 m primary, 20 m stretch	input_data/expected_results/results_report.md
FT-P-02	Same 60 images + map references where present	Output includes source label, covariance semi-major axis, and anchor age for every emitted estimate	Required-field validation + geolocation threshold	Required fields present; geolocation thresholds as above	input_data/expected_results/results_report.md
FT-P-03	derkachi_video_telemetry plus public synchronized VIO dataset slice when available	BASALT + wrapper emits trajectory with calibrated covariance and no optimistic under-reporting	Compare Derkachi output to GLOBAL_POSITION_INT trajectory for smoke/relative validation; compare public/representative calibrated runs to ground truth for final accuracy	Derkachi threshold is calibration-gated; final threshold is dataset-specific and pinned after camera calibration	data_imu.csv trajectory plus public dataset ground truth
FT-P-04	AerialVL/ALTO/VPAir-style aerial localization slice	Satellite retrieval returns candidate chunks and local verification produces accepted/rejected anchors	Georeference error + MRE + source-label checks	AC-1.1/1.2 and AC-2.2 thresholds where dataset supports them	Public dataset ground truth/reference map
FT-N-01	Low-texture/repetitive frames from sample or public data	System emits degraded confidence or rejects anchor rather than confident false fix	Source label and covariance threshold	No satellite_anchored label unless gates pass	Fixture-specific
FT-N-02	Plane SITL GPS spoof/loss trace	Spoofed GPS rejected; system promotes own estimate within <3 s when trigger conditions are met	Event timing and MAVLink field checks	<3 s promotion; blackout thresholds from AC-NEW-8	Generated SITL trace
FT-N-03	Stale/unsigned/hash-mismatched cache fixtures	Anchors rejected or downgraded; stale tile never emits satellite_anchored	Manifest validation + emitted label check	0 accepted stale/invalid anchors	Cache fixture manifest

External Dependency Mocks

External Service	Mock/Stub	How Provided	Behavior
Azaion Suite Satellite Service	Offline cache stub	Local COG/manifest/descriptor fixture	Provides only preloaded tiles; no in-flight network fetch
Flight controller	ArduPilot Plane SITL and MAVLink replay	SITL container/process and recorded/generated tlogs	Emits IMU, attitude, altitude, airspeed, GPS health/spoofing events
QGroundControl	MAVLink observer/log parser	Test-side parser	Verifies downsampled status and STATUSTEXT events

Data Validation Rules

Data Type	Validation	Invalid Examples	Expected System Behavior
Image frame	Existing file, readable image, expected timestamp/order metadata if sequence replay	Missing image, corrupt image, unsupported resolution	Mark estimate unavailable/degraded, log error, continue if possible
Expected coordinate	Valid WGS84 latitude/longitude	Out-of-range lat/lon, missing row	Reject test fixture before replay
Video/telemetry pair	MP4 duration matches telemetry duration, frame-to-telemetry ratio is stable, timestamps are monotonic	Duration drift >250 ms, missing trajectory columns, non-monotonic timestamps	Reject fixture before replay
IMU trace	Monotonic timestamps, angular rate/accel fields, calibrated units	Non-monotonic timestamps, missing samples	Reject fixture or enter degraded mode depending scenario
GPS trajectory trace	Valid WGS84 lat/lon, altitude, velocity, and heading fields	Out-of-range lat/lon, impossible altitude, missing GLOBAL_POSITION_INT columns	Reject trajectory comparison while allowing pure video replay if applicable
Cache tile manifest	CRS, m/px, capture date, source, hashes, signature/provenance	Stale, unsigned, hash mismatch, low resolution	Reject or down-confidence per AC-8.2 and AC-NEW-6
MAVLink output	Valid GPS_INPUT fields and fix type/accuracy semantics	Missing horiz_accuracy, impossible fix type	Fail test; output contract violated

Phase 3 Validation Gate Result

Test Scenario ID	Shape	Required Input Data	Required Expected Result	Input Provided?	Expected Result Provided?	Validation Decision
FT-P-01	Input/output	60 project images + coordinates.csv	WGS84 center per image with 50 m / 20 m thresholds	Yes	Yes	Keep
FT-P-02	Input/output	60 project images + output schema expectations	Required confidence/source-label fields and thresholds	Yes	Yes	Keep
FT-P-03	Input/output	Derkachi synchronized video/IMU/GPS fixture; public or calibrated representative dataset for final accuracy	Derkachi GLOBAL_POSITION_INT trajectory for smoke/relative validation; calibrated ground truth for final covariance checks	Yes for Derkachi; public/calibrated dataset still useful for final signoff	Yes for Derkachi GPS trajectory; calibrated camera thresholds pending	Keep with calibration gate
FT-P-04	Input/output	Public aerial localization or project cache fixture	Georeference, MRE, and source-label checks	Accepted as required external fixture	Accepted as dataset/reference-map ground truth	Keep with acquisition task
FT-N-01	Behavioral/input-output	Ambiguous low-texture/repetitive frames	0 confident false anchors	Accepted as project/public fixture	Yes	Keep
FT-N-02	Behavioral	Generated Plane SITL spoof/blackout trace	Timing and MAVLink field thresholds from AC-NEW-8	Generated by test harness	Yes	Keep
FT-N-03	Behavioral/input-output	Cache integrity fixtures	0 trusted anchors from stale/invalid tiles	Generated fixture	Yes	Keep
NFT-PERF-01	Input/output	60 project images	p95 latency and drop-rate thresholds	Yes	Yes	Keep
NFT-PERF-02	Input/output	Derkachi synchronized video/IMU/GPS fixture; public/representative synchronized camera/IMU dataset	VO registration, latency, memory thresholds	Yes for Derkachi	Yes	Keep with calibration gate
NFT-PERF-03	Behavioral/input-output	Precomputed descriptor/cache fixture	Trigger-path latency and MRE thresholds	Generated fixture	Yes	Keep
NFT-PERF-04	Behavioral	Cold-start harness and cache fixture	<30 s p95 over 50 runs	Generated by test harness	Yes	Keep
NFT-RES-*	Behavioral	Fault triggers and generated traces	AC-defined timing/status thresholds	Generated by test harness	Yes	Keep
NFT-SEC-*	Behavioral/input-output	Cache/MAVLink/network fixtures	Rejection/no-fetch/no-promote thresholds	Generated fixture	Yes	Keep
NFT-RES-LIM-*	Behavioral	Jetson/cache/FDR monitoring environment	Numeric resource thresholds	Environment-dependent	Yes	Keep

Coverage after validation: 49/49 AC and restriction groups remain covered. No tests were removed.

Acquisition tasks required downstream:

• Pin camera intrinsics, lens distortion, raw camera feed parameters, and camera-to-body mounting transform for the Derkachi fixture or future representative recordings.
• Pin and download at least one strong synchronized nadir camera + IMU + ground-truth dataset, preferably MUN-FRL or ALTO, with EPFL fixed-wing and Kagaru as fixed-wing/farmland candidates.
• Pin license-compatible VPR/localization datasets for satellite anchor tests; VPAir and UZH FPV have non-commercial restrictions and must not be used for commercial acceptance unless license terms allow it.
• Create generated fixtures for Plane SITL spoofing, stale cache manifests, signed/unsigned manifests, FDR load, and thermal/resource monitoring during implementation.