gps-denied-desktop/_docs/01_solution/solution_draft04.md

# Solution Draft

## Assessment Findings


| Old Component Solution | Weak Point | New Solution |
| --- | --- | --- |
| Stage 2: SuperPoint+LightGlue ONNX FP16 | **Performance (Moderate)**: SP+LG achieves only 54-58% hit rate in Hard mode on satellite-aerial benchmarks. LiteSAM achieves 62-77% — up to +19pp improvement that directly impacts AC compliance. | Replace with LiteSAM end-to-end semi-dense matcher. 6.31M params, best hit rate among semi-dense methods on UAV-VisLoc and self-made datasets. |
| SuperPoint + LightGlue as two separate models | **Performance (Low)**: Two models loaded (SuperPoint ~400MB + LightGlue ~500MB = ~900MB VRAM). Two separate feature caches. | LiteSAM is a single end-to-end model (~400MB VRAM). Simpler pipeline, lower VRAM. |
| SuperPoint features cached per satellite tile | **Functional (Low)**: SuperPoint features must be pre-computed and cached separately from DINOv2 embeddings. | LiteSAM does not require per-tile feature caching — features are computed jointly during matching. Only DINOv2 embeddings cached for coarse retrieval. |


## Product Solution Description

A Python-based GPS-denied visual navigation service that determines GPS coordinates of consecutive UAV photo centers using a hierarchical localization approach: fast visual odometry for frame-to-frame motion, two-stage satellite geo-referencing (coarse retrieval + LiteSAM fine matching) for absolute positioning, and factor graph optimization for trajectory refinement. The system operates as a background REST API service with real-time SSE streaming.

**Core approach**: Consecutive images are matched using SuperPoint+LightGlue (learned features with contextual matching) to estimate relative motion (visual odometry). Each image is geo-referenced against satellite imagery through a two-stage process: DINOv2 ViT-S/14 coarse retrieval selects the best-matching satellite tile using patch-level features, then LiteSAM (lightweight semi-dense matcher, 6.31M params) refines the alignment to subpixel precision. LiteSAM achieves 77.3% hit rate in Hard conditions on satellite-aerial benchmarks — significantly better than SuperPoint+LightGlue (58.3%). A GTSAM iSAM2 factor graph fuses VO constraints (BetweenFactorPose2) and satellite anchors (PriorFactorPose2) in local ENU coordinates to produce an optimized trajectory. The system handles route disconnections by treating each continuous VO chain as an independent segment, geo-referenced through satellite matching and connected via the shared WGS84 coordinate frame.

```
┌─────────────────────────────────────────────────────────────────────┐
│                        Client (Desktop App)                         │
│   POST /jobs (start GPS, camera params, image folder)               │
│   GET  /jobs/{id}/stream (SSE)                                      │
│   POST /jobs/{id}/anchor (user manual GPS input)                    │
│   GET  /jobs/{id}/point-to-gps (image_id, pixel_x, pixel_y)        │
└──────────────────────┬──────────────────────────────────────────────┘
                       │ HTTP/SSE (JWT auth)
┌──────────────────────▼──────────────────────────────────────────────┐
│                     FastAPI Service Layer                            │
│   Job Manager → Pipeline Orchestrator → SSE Event Publisher         │
│   (asyncio.Queue-based publisher, heartbeat, Last-Event-ID)         │
└──────────────────────┬──────────────────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────────────────┐
│                     Processing Pipeline                              │
│                                                                      │
│  ┌──────────────┐  ┌──────────────┐  ┌─────────────────────────┐   │
│  │ Image        │  │  Visual      │  │ Satellite Geo-Ref       │   │
│  │ Preprocessor │→│  Odometry    │→│ Satellite Geo-Ref       │   │
│  │ (downscale,  │  │ (SuperPoint │  │ Stage 1: DINOv2-S patch  │   │
│  │  rectify)    │  │ + LightGlue)│  │   retrieval (CPU faiss)  │   │
│  │              │  │  matcher)   │  │ Stage 2: LiteSAM fine    │   │
│  │              │  │             │  │   matching (subpixel)     │   │
│  └──────────────┘  └──────────────┘  └─────────────────────────┘   │
│         │                │                       │                   │
│         ▼                ▼                       ▼                   │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │       GTSAM iSAM2 Factor Graph Optimizer                    │   │
│  │  Pose2 + BetweenFactorPose2 (VO) + PriorFactorPose2 (sat)  │   │
│  │  Local ENU coordinates → WGS84 output                       │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                              │                                       │
│  ┌───────────────────────────▼──────────────────────────────────┐   │
│  │                  Segment Manager                              │   │
│  │   (drift thresholds, confidence decay, user input triggers)   │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │          Multi-Provider Satellite Tile Cache                  │   │
│  │  (Google Maps + Mapbox + user tiles, session tokens,          │   │
│  │   DEM cache, request budgeting)                               │   │
│  └──────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────┘
```

## Architecture

### Component: Image Preprocessor


| Solution | Tools | Advantages | Limitations | Requirements | Security | Performance | Fit |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Downscale + rectify + validate | OpenCV resize, NumPy | Normalizes input. Consistent memory. Validates before loading. | Loses fine detail in downscaled images. | OpenCV, NumPy | Magic byte validation, dimension check before load | <10ms per image | **Best** |


**Selected**: Downscale + rectify + validate pipeline.

**Preprocessing per image**:

1. Validate file: check magic bytes (JPEG/PNG/TIFF), reject unknown formats
2. Read image header only: check dimensions, reject if either > 10,000px
3. Load image via OpenCV (cv2.imread)
4. Downscale to max 1600 pixels on longest edge (preserving aspect ratio)
5. Store original resolution for GSD: `GSD = (effective_altitude × sensor_width) / (focal_length × original_width)` where `effective_altitude = flight_altitude - terrain_elevation` (terrain from Copernicus DEM)
6. If estimated heading is available: rotate to approximate north-up for satellite matching
7. If no heading (segment start): pass unrotated
8. Convert to grayscale for feature extraction
9. Output: downscaled grayscale image + metadata (original dims, GSD, heading if known)

### Component: Feature Extraction


| Solution | Tools | Advantages | Limitations | Requirements | Security | Performance | Fit |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SuperPoint (for VO) | superpoint (PyTorch) | Learned features, robust to viewpoint/illumination. 256-dim descriptors. Proven in visual odometry pipelines. | Not rotation-invariant. | NVIDIA GPU, PyTorch, CUDA | Model weights from official source | ~80ms GPU | **Best for VO** |
| LiteSAM (for satellite matching) | LiteSAM (PyTorch) | Best hit rate on satellite-aerial benchmarks (77.3% Hard). 6.31M params. Subpixel refinement via MinGRU. End-to-end semi-dense matcher. | Not rotation-invariant. | PyTorch, NVIDIA GPU | Model weights from Google Drive | ~140-210ms on RTX 2060 (est.) | **Best for satellite** |
| SIFT (rotation fallback) | OpenCV cv2.SIFT | Rotation-invariant. Scale-invariant. Proven SIFT+LightGlue hybrid for UAV mosaicking (ISPRS 2025). | Slower. Less discriminative in low-texture. | OpenCV | N/A | ~200ms CPU | **Rotation fallback** |


**Selected**: SuperPoint+LightGlue for VO, LiteSAM for satellite fine matching, SIFT+LightGlue as rotation-heavy fallback.

**VRAM budget**:


| Model | VRAM | Loaded When |
| --- | --- | --- |
| SuperPoint | ~400MB | Always (VO every frame) |
| LightGlue ONNX FP16 | ~500MB | Always (VO every frame) |
| DINOv2 ViT-S/14 | ~300MB | Satellite coarse retrieval |
| LiteSAM (6.31M params) | ~400MB | Satellite fine matching |
| **Peak total** | **~1.6GB** | Satellite matching phase |


### Component: Feature Matching


| Solution | Tools | Advantages | Limitations | Requirements | Security | Performance | Fit |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SuperPoint+LightGlue (VO) | SuperPoint + LightGlue-ONNX FP16 | High-quality learned features + contextual matching. LightGlue ONNX FP16 on Turing. Well-proven pipeline. | Not rotation-invariant. | PyTorch, ONNX Runtime, NVIDIA GPU | Model weights from official source | ~130-180ms on RTX 2060 | **Best for VO** |
| LiteSAM (satellite fine matching) | LiteSAM (PyTorch) | Best hit rate on satellite-aerial benchmarks (77.3% Hard, 92.1% Easy). 6.31M params (2.4x fewer than EfficientLoFTR). Subpixel refinement via MinGRU. Single end-to-end model. | Not rotation-invariant. | PyTorch, NVIDIA GPU | Model weights from Google Drive | ~140-210ms on RTX 2060 (est.) | **Best for satellite** |
| SIFT+LightGlue (rotation fallback) | OpenCV SIFT + LightGlue | SIFT rotation invariance + LightGlue contextual matching. Proven superior for high-rotation UAV (ISPRS 2025). | Slower. | OpenCV + ONNX Runtime | N/A | ~250ms total | **Rotation fallback** |


**Selected**: SuperPoint+LightGlue for VO, LiteSAM for satellite fine matching, SIFT+LightGlue as rotation fallback.

### Component: Visual Odometry (Consecutive Frame Matching)


| Solution | Tools | Advantages | Limitations | Requirements | Security | Performance | Fit |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Homography VO with essential matrix fallback | OpenCV findHomography (USAC_MAGSAC), findEssentialMat, decomposeHomographyMat | Homography: optimal for flat terrain. Essential matrix: non-planar fallback. Known altitude resolves scale. | Homography assumes planar. 4-way decomposition ambiguity. | OpenCV, NumPy | N/A | ~5ms for estimation | **Best** |


**Selected**: Homography VO with essential matrix fallback and DEM terrain-corrected GSD.

**VO Pipeline per frame**:

1. Extract SuperPoint features from current image (~80ms)
2. Match with previous image using LightGlue ONNX FP16 (~50-100ms)
3. **Triple failure check**: match count ≥ 30 AND RANSAC inlier ratio ≥ 0.4 AND motion magnitude consistent with expected inter-frame distance (100m ± 250m)
4. If checks pass → estimate homography (cv2.findHomography with USAC_MAGSAC, confidence 0.999, max iterations 2000)
5. If RANSAC inlier ratio < 0.6 → additionally estimate essential matrix as quality check
6. **Decomposition disambiguation** (4 solutions from decomposeHomographyMat):
   a. Filter by positive depth: triangulate 5 matched points, reject if behind camera
   b. Filter by plane normal: normal z-component > 0.5 (downward camera → ground plane normal points up)
   c. If previous direction available: prefer solution consistent with expected motion
   d. Orthogonality check: verify R^T R ≈ I (Frobenius norm < 0.01). If failed, re-orthogonalize via SVD: U,S,V = svd(R), R_clean = U @ V^T
   e. First frame pair in segment: use filters a+b only
7. **Terrain-corrected GSD**: query Copernicus DEM at estimated position → `effective_altitude = flight_altitude - terrain_elevation` → `GSD = (effective_altitude × sensor_width) / (focal_length × original_image_width)`
8. Convert pixel displacement to meters: `displacement_m = displacement_px × GSD`
9. Update position: `new_pos = prev_pos + rotation @ displacement_m`
10. Track cumulative heading for image rectification
11. If triple failure check fails → trigger segment break

### Component: Satellite Image Geo-Referencing (Two-Stage)


| Solution | Tools | Advantages | Limitations | Requirements | Security | Performance | Fit |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Stage 1: DINOv2 ViT-S/14 patch retrieval | dinov2 ViT-S/14 (PyTorch), faiss (CPU) | Fast (50ms). 300MB VRAM. Patch tokens capture spatial layout better than CLS alone. Semantic matching robust to seasonal change. | Coarse only (~tile-level). Lower precision than ViT-B/ViT-L. | PyTorch, faiss-cpu | Model weights from official source | ~50ms extract + <1ms search | **Best coarse** |
| Stage 2: LiteSAM fine matching | LiteSAM (PyTorch) | Best satellite-aerial hit rate (77.3% Hard). Subpixel accuracy via MinGRU. 6.31M params, ~400MB VRAM. End-to-end semi-dense matching. | Not rotation-invariant. No ONNX yet. | PyTorch, NVIDIA GPU | Model weights from Google Drive | ~140-210ms on RTX 2060 (est.) | **Best fine** |


**Selected**: Two-stage hierarchical matching — DINOv2 coarse retrieval + LiteSAM fine matching.

**Satellite Matching Pipeline**:

1. Estimate approximate position from VO
2. **Stage 1 — Coarse retrieval**:
   a. Define search area: 500m radius around VO estimate (expand to 1km if segment just started or drift > 100m)
   b. Pre-compute DINOv2 ViT-S/14 patch embeddings for all satellite tiles in search area. Method: extract patch tokens (not CLS), apply spatial average pooling to get a single descriptor per tile. Cache embeddings.
   c. Extract DINOv2 ViT-S/14 patch embedding from UAV image (same pooling)
   d. Find top-5 most similar satellite tiles using faiss (CPU) cosine similarity
3. **Stage 2 — Fine matching** (on top-5 tiles, stop on first good match):
   a. Warp UAV image to approximate nadir view using estimated camera pose
   b. **Rotation handling**:
      - If heading known: single attempt with rectified image
      - If no heading (segment start): try 4 rotations {0°, 90°, 180°, 270°}
   c. Run LiteSAM on (uav_warped, sat_tile) → semi-dense correspondences with subpixel accuracy
   d. **Geometric validation**: require ≥15 inliers, inlier ratio ≥ 0.3, reprojection error < 3px
   e. If valid: estimate homography → transform image center → satellite pixel → WGS84
   f. Report: absolute position anchor with confidence based on match quality
4. If all 5 tiles fail Stage 2 with LiteSAM:
   a. Try SIFT+LightGlue on top-3 tiles (rotation-invariant). Trigger: best LiteSAM inlier ratio was < 0.15.
   b. Try zoom level 17 (wider view)
5. If still fails: mark frame as VO-only, reduce confidence, continue

**Satellite matching frequency**: Every frame when available, but async — satellite matching for frame N overlaps with VO processing for frame N+1. Satellite result arrives and gets added to factor graph retroactively via iSAM2 update.

### Component: GTSAM Factor Graph Optimizer


| Solution | Tools | Advantages | Limitations | Requirements | Security | Performance | Fit |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GTSAM iSAM2 factor graph (Pose2) | gtsam==4.2 (pip) | Incremental smoothing. Proper uncertainty propagation. Native BetweenFactorPose2 and PriorFactorPose2. Backward smoothing on new evidence. Python bindings. | C++ backend (pip binary). Learning curve. | gtsam==4.2, NumPy | N/A | ~5-10ms incremental update | **Best** |


**Selected**: GTSAM iSAM2 with Pose2 variables.

**Coordinate system**: Local East-North-Up (ENU) centered on starting GPS. All positions computed in ENU meters, converted to WGS84 for output. Conversion: pyproj or manual geodetic math (WGS84 ellipsoid).

**Factor graph structure**:

- **Variables**: Pose2 (x_enu, y_enu, heading) per image
- **Prior Factor** (PriorFactorPose2): first frame anchored at ENU origin (0, 0, initial_heading) with tight noise (sigma_xy = 5m if GPS accurate, sigma_theta = 0.1 rad)
- **VO Factor** (BetweenFactorPose2): relative motion between consecutive frames. Noise model: `Diagonal.Sigmas([sigma_x, sigma_y, sigma_theta])` where sigma scales inversely with RANSAC inlier ratio. High inlier ratio (0.8) → sigma 2m. Low inlier ratio (0.4) → sigma 10m. Sigma_theta proportional to displacement magnitude.
- **Satellite Anchor Factor** (PriorFactorPose2): absolute position from satellite matching. Position noise: `sigma = reprojection_error × GSD × scale_factor`. Good match (0.5px × 0.4m/px × 3) = 0.6m. Poor match = 5-10m. Heading component: loose (sigma = 1.0 rad) unless estimated from satellite alignment.

**Optimizer behavior**:

- On each new frame: add VO factor, run iSAM2.update() → ~5ms
- On satellite match arrival: add PriorFactorPose2, run iSAM2.update() → backward correction
- Emit updated positions via SSE after each update
- Refinement events: when backward correction moves positions by >1m, emit "refined" SSE event
- No custom Python factors — all factors use native GTSAM C++ implementations for speed

### Component: Segment Manager

The segment manager tracks independent VO chains, manages drift thresholds, and handles reconnection.

**Segment lifecycle**:

1. **Start condition**: First image, OR VO triple failure check fails
2. **Active tracking**: VO provides frame-to-frame motion within segment
3. **Anchoring**: Satellite two-stage matching provides absolute position
4. **End condition**: VO failure (sharp turn, outlier >350m, occlusion)
5. **New segment**: Starts, attempts satellite anchor immediately

**Segment states**:

- `ANCHORED`: At least one satellite match → HIGH confidence
- `FLOATING`: No satellite match yet → positioned relative to segment start → LOW confidence
- `USER_ANCHORED`: User provided manual GPS → MEDIUM confidence

**Drift monitoring (replaces GTSAM custom drift factor)**:

- Track cumulative VO displacement since last satellite anchor per segment
- **100m threshold**: emit warning SSE event, expand satellite search radius to 1km, increase matching attempts per frame
- **200m threshold**: emit `user_input_needed` SSE event with configurable timeout (default: 30s)
- **500m threshold**: mark all subsequent positions as VERY LOW confidence, continue processing
- **Confidence formula**: `confidence = base_confidence × exp(-drift / decay_constant)` where base_confidence is from satellite match quality, drift is distance from nearest anchor, decay_constant = 100m

**Segment reconnection**:

- When a segment becomes ANCHORED, check for nearby FLOATING segments (within 500m of any anchored position)
- Attempt satellite-based position matching between FLOATING segment images and tiles near the ANCHORED segment
- DEM consistency: verify segment elevation profile is consistent with terrain
- If no match after all frames tried: request user input, auto-continue after timeout

### Component: Multi-Provider Satellite Tile Cache


| Solution | Tools | Advantages | Limitations | Requirements | Security | Performance | Fit |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Multi-provider progressive cache with DEM | aiohttp, aiofiles, sqlite3, faiss-cpu | Multiple providers. Async download. DINOv2/feature pre-computation. DEM cached. Session token management. | Needs internet. Provider API differences. | Google Maps Tiles API + Mapbox API keys | API keys in env vars only. Session tokens managed internally. | Async, non-blocking | **Best** |


**Selected**: Multi-provider progressive cache.

**Provider priority**:

1. User-provided tiles (highest priority — custom/recent imagery)
2. Google Maps (zoom 18, ~0.4m/px) — 100K free requests/month, 15K/day
3. Mapbox Satellite (zoom 16-18, ~0.6-0.3m/px) — 200K free requests/month

**Google Maps session management**:

1. On job start: POST to `/v1/createSession` with API key → receive session token
2. Use session token in all subsequent tile requests for this job
3. Token has finite lifetime — handle expiry by creating new session
4. Track request count per day per provider

**Cache strategy**:

1. On job start: download tiles in 1km radius around starting GPS from primary provider
2. Pre-compute DINOv2 ViT-S/14 patch embeddings for all cached tiles
3. As route extends: download tiles 500m ahead of estimated position
4. **Request budgeting**: track daily API requests per provider. At 80% daily limit (12,000 for Google): switch to Mapbox. Log budget status.
5. Cache structure on disk:
   ```
   cache/
   ├── tiles/{provider}/{zoom}/{x}/{y}.jpg
   ├── embeddings/{provider}/{zoom}/{x}/{y}_dino.npy  (DINOv2 patch embedding)
   └── dem/{lat}_{lon}.tif                            (Copernicus DEM tiles)
   ```
6. Cache persistent across jobs — tiles and features reused for overlapping areas
7. **DEM cache**: Copernicus DEM GLO-30 tiles from AWS S3 (free, no auth). `s3://copernicus-dem-30m/`. Cloud Optimized GeoTIFFs, 30m resolution. Downloaded via HTTPS (no AWS SDK needed): `https://copernicus-dem-30m.s3.amazonaws.com/Copernicus_DSM_COG_10_{N|S}{lat}_00_{E|W}{lon}_DEM/...`

**Tile download budget**:

- Google Maps: 100,000/month, 15,000/day → ~7 flights/day from cache misses, ~50 flights/month
- Mapbox: 200,000/month → additional ~100 flights/month
- Per flight: ~2000 satellite tiles (~80MB) + ~200 DEM tiles (~10MB)

### Component: API & Real-Time Streaming


| Solution | Tools | Advantages | Limitations | Requirements | Security | Performance | Fit |
| --- | --- | --- | --- | --- | --- | --- | --- |
| FastAPI + SSE (Queue-based) + JWT | FastAPI ≥0.135.0, asyncio.Queue, uvicorn, python-jose | Native SSE. Queue-based publisher avoids generator cleanup issues. JWT auth. OpenAPI auto-generated. | Python GIL (mitigated with asyncio). | Python 3.11+, uvicorn | JWT, CORS, rate limiting, CSP headers | Async, non-blocking | **Best** |


**Selected**: FastAPI + Queue-based SSE + JWT authentication.

**SSE implementation**:

- Use `asyncio.Queue` per client connection (not bare async generators)
- Server pushes events to queue; client reads from queue
- On disconnect: queue is garbage collected, no lingering generators
- SSE heartbeat: send `event: heartbeat` every 15 seconds to detect stale connections
- Support `Last-Event-ID` header for reconnection: include monotonic event ID in each SSE message. On reconnect, replay missed events from in-memory ring buffer (last 1000 events per job).

**API Endpoints**:

```
POST /auth/token
  Body: { api_key }
  Returns: { access_token, token_type, expires_in }

POST /jobs
  Headers: Authorization: Bearer <token>
  Body: { start_lat, start_lon, altitude, camera_params, image_folder }
  Returns: { job_id }

GET /jobs/{job_id}/stream
  Headers: Authorization: Bearer <token>
  SSE stream of:
    - { event: "position", id: "42", data: { image_id, lat, lon, confidence, segment_id } }
    - { event: "refined", id: "43", data: { image_id, lat, lon, confidence, delta_m } }
    - { event: "segment_start", id: "44", data: { segment_id, reason } }
    - { event: "drift_warning", id: "45", data: { segment_id, cumulative_drift_m } }
    - { event: "user_input_needed", id: "46", data: { image_id, reason, timeout_s } }
    - { event: "heartbeat", id: "47", data: { timestamp } }
    - { event: "complete", id: "48", data: { summary } }

POST /jobs/{job_id}/anchor
  Headers: Authorization: Bearer <token>
  Body: { image_id, lat, lon }

GET /jobs/{job_id}/point-to-gps?image_id=X&px=100&py=200
  Headers: Authorization: Bearer <token>
  Returns: { lat, lon, confidence }

GET /jobs/{job_id}/results?format=geojson
  Headers: Authorization: Bearer <token>
  Returns: full results as GeoJSON or CSV (WGS84)
```

**Security measures**:

- JWT authentication on all endpoints (short-lived tokens, 1h expiry)
- Image folder whitelist: resolve to canonical path (os.path.realpath), verify under configured base directories
- Image validation: magic byte check (JPEG FFD8, PNG 89504E47, TIFF 4949/4D4D), dimension check (<10,000px per side), reject others
- Pin Pillow ≥11.3.0 (CVE-2025-48379 mitigation)
- Max concurrent SSE connections per client: 5
- Rate limiting: 100 requests/minute per client
- All provider API keys in environment variables, never logged or returned
- CORS configured for known client origins only
- Content-Security-Policy headers
- SSE heartbeat prevents stale connections accumulating

### Component: Interactive Point-to-GPS Lookup

For each processed image, the system stores the estimated camera-to-ground transformation. Given pixel coordinates (px, py):

1. If image has satellite match: use computed homography to project (px, py) → satellite tile coordinates → WGS84. HIGH confidence.
2. If image has only VO pose: use camera intrinsics + DEM-corrected altitude + estimated heading to ray-cast (px, py) to ground plane → WGS84. MEDIUM confidence.
3. Confidence score derived from underlying position estimate quality.

## Processing Time Budget


| Step | Component | Time | GPU/CPU | Notes |
| --- | --- | --- | --- | --- |
| 1 | Image load + validate + downscale | <10ms | CPU | OpenCV |
| 2 | SuperPoint feature extraction | ~80ms | GPU | 256-dim descriptors |
| 3 | LightGlue ONNX FP16 matching | ~50-100ms | GPU | Contextual matcher |
| 4 | Homography estimation + decomposition | ~5ms | CPU | USAC_MAGSAC |
| 5 | GTSAM iSAM2 update (VO factor) | ~5ms | CPU | Incremental |
| 6 | SSE position emit | <1ms | CPU | Queue push |
| **VO subtotal** | | **~150-200ms** | | **Per-frame critical path** |
| 7 | DINOv2 ViT-S/14 extract (UAV image) | ~50ms | GPU | Patch tokens |
| 8 | faiss cosine search (top-5 tiles) | <1ms | CPU | ~2000 vectors |
| 9 | LiteSAM fine matching (per tile, up to 5) | ~140-210ms | GPU | End-to-end semi-dense, est. RTX 2060 |
| 10 | Geometric validation + homography | ~5ms | CPU | |
| 11 | GTSAM iSAM2 update (satellite factor) | ~5ms | CPU | Backward correction |
| **Satellite subtotal** | | **~201-271ms** | | **Overlapped with next frame's VO** |
| **Total per frame** | | **~350-470ms** | | **Well under 5s budget** |


## Testing Strategy

### Integration / Functional Tests

- End-to-end pipeline test using provided 60-image sample dataset with ground truth GPS
- Verify 80% of positions within 50m of ground truth
- Verify 60% of positions within 20m of ground truth
- Test sharp turn handling: simulate 90° turn with non-overlapping images
- Test segment creation, satellite anchoring, and cross-segment reconnection
- Test user manual anchor injection via POST endpoint
- Test point-to-GPS lookup accuracy against known ground coordinates
- Test SSE streaming delivers results within 1s of processing completion
- Test with FullHD resolution images (pipeline must not fail)
- Test with 6252×4168 images (verify downscaling and memory usage)
- Test DINOv2 ViT-S/14 coarse retrieval finds correct satellite tile with 100m VO drift
- Test multi-provider fallback: block Google Maps, verify Mapbox takes over
- Test with outdated satellite imagery: verify confidence scores reflect match quality
- Test outlier handling: 350m gap between consecutive photos
- Test image rotation handling: apply 45° and 90° rotation, verify 4-rotation retry works
- Test SIFT+LightGlue fallback triggers when LiteSAM inlier ratio < 0.15
- Test GTSAM PriorFactorPose2 satellite anchoring produces backward correction
- Test drift warning at 100m cumulative displacement without satellite anchor
- Test user_input_needed event at 200m cumulative displacement
- Test SSE heartbeat arrives every 15s during long processing
- Test SSE reconnection with Last-Event-ID replays missed events
- Test homography decomposition disambiguation for first frame pair (no previous direction)
- Test LiteSAM fine matching produces valid correspondences on satellite-aerial pair
- Test LiteSAM subpixel accuracy improves homography estimation vs pixel-level only

### Non-Functional Tests

- Processing speed: <5s per image on RTX 2060 (target <300ms with ONNX optimization)
- Memory: peak RAM <16GB, VRAM <6GB during 3000-image flight at max resolution
- VRAM: verify peak stays under 1GB during satellite matching phase (LiteSAM)
- Memory stability: process 3000 images, verify no memory leak (stable RSS over time)
- Concurrent jobs: 2 simultaneous flights, verify isolation and resource sharing
- Tile cache: verify tiles and DINOv2 embeddings cached and reused
- API: load test SSE connections (10 simultaneous clients)
- Recovery: kill and restart service mid-job, verify job can resume from last processed image
- DEM download: verify Copernicus DEM tiles fetched from AWS S3 and cached correctly
- GTSAM optimizer: verify backward correction produces "refined" events
- Session token lifecycle: verify Google Maps session creation, usage, and expiry handling

### Security Tests

- JWT authentication enforcement on all endpoints
- Expired/invalid token rejection
- Provider API keys not exposed in responses, logs, or error messages
- Image folder path traversal prevention (attempt to access /etc/passwd via image_folder)
- Image folder whitelist enforcement (canonical path resolution)
- Image magic byte validation: reject non-image files renamed to .jpg
- Image dimension validation: reject >10,000px images
- Input validation: invalid GPS coordinates, negative altitude, malformed camera params
- Rate limiting: verify 429 response after exceeding limit
- Max SSE connection enforcement
- CORS enforcement: reject requests from unknown origins
- Content-Security-Policy header presence
- Pillow version ≥11.3.0 verified in requirements

## References

- [LiteSAM (Remote Sensing, Oct 2025)](https://www.mdpi.com/2072-4292/17/19/3349) — Lightweight satellite-aerial feature matching, 6.31M params, RMSE@30=17.86m on UAV-VisLoc, 77.3% Hard HR on self-made dataset
- [LiteSAM GitHub](https://github.com/boyagesmile/LiteSAM) — Official code, pretrained weights available, built upon EfficientLoFTR
- [EfficientLoFTR (CVPR 2024)](https://github.com/zju3dv/EfficientLoFTR) — LiteSAM's base architecture, 15.05M params
- [YFS90/GNSS-Denied-UAV-Geolocalization](https://github.com/YFS90/GNSS-Denied-UAV-Geolocalization) — <7m MAE with terrain-weighted constraint optimization
- [SatLoc-Fusion (2025)](https://www.mdpi.com/2072-4292/17/17/3048) — hierarchical DINOv2+XFeat+optical flow, <15m on edge hardware
- [CEUSP (2025)](https://arxiv.org/abs/2502.11408) — DINOv2-based cross-view UAV self-positioning
- [DINOv2 UAV Self-Localization (2025)](https://ui.adsabs.harvard.edu/abs/2025IRAL...10.2080Y/) — 86.27 R@1 on DenseUAV
- [LightGlue-ONNX](https://github.com/fabio-sim/LightGlue-ONNX) — 2-4x speedup via ONNX/TensorRT, FP16 on Turing
- [SIFT+LightGlue UAV Mosaicking (ISPRS 2025)](https://isprs-archives.copernicus.org/articles/XLVIII-2-W11-2025/169/2025/) — SIFT superior for high-rotation conditions
- [LightGlue rotation issue #64](https://github.com/cvg/LightGlue/issues/64) — confirmed not rotation-invariant
- [DALGlue (2025)](https://www.nature.com/articles/s41598-025-21602-5) — 11.8% MMA improvement over LightGlue for UAV
- [SALAD: DINOv2 Optimal Transport Aggregation (2024)](https://arxiv.org/abs/2311.15937) — improved visual place recognition
- [NaviLoc (2025)](https://www.mdpi.com/2504-446X/10/2/97) — trajectory-level optimization, 19.5m MLE, 16x improvement
- [GTSAM v4.2](https://github.com/borglab/gtsam) — factor graph optimization with Python bindings
- [GTSAM GPSFactor docs](https://gtsam.org/doxygen/a04084.html) — GPSFactor works with Pose3 only
- [GTSAM Pose2 SLAM Example](https://gtbook.github.io/gtsam-examples/Pose2SLAMExample.html) — BetweenFactorPose2 + PriorFactorPose2
- [OpenCV decomposeHomographyMat issue #23282](https://github.com/opencv/opencv/issues/23282) — non-orthogonal matrices, 4-solution ambiguity
- [Copernicus DEM GLO-30 on AWS](https://registry.opendata.aws/copernicus-dem/) — free 30m global DEM, no auth via S3
- [Google Maps Tiles API](https://developers.google.com/maps/documentation/tile/satellite) — satellite tiles, 100K free/month, session tokens required
- [Google Maps Tiles API billing](https://developers.google.com/maps/documentation/tile/usage-and-billing) — 15K/day, 6K/min rate limits
- [Mapbox Satellite](https://docs.mapbox.com/data/tilesets/reference/mapbox-satellite/) — alternative tile provider, up to 0.3m/px regional
- [FastAPI SSE](https://fastapi.tiangolo.com/tutorial/server-sent-events/) — EventSourceResponse
- [SSE-Starlette cleanup issue #99](https://github.com/sysid/sse-starlette/issues/99) — async generator cleanup, Queue pattern recommended
- [CVE-2025-48379 Pillow](https://nvd.nist.gov/vuln/detail/CVE-2025-48379) — heap buffer overflow, fixed in 11.3.0
- [FAISS GPU wiki](https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU) — ~2GB scratch space default, CPU recommended for small datasets
- [Oblique-Robust AVL (IEEE TGRS 2024)](https://ieeexplore.ieee.org/iel7/36/10354519/10356107.pdf) — rotation-equivariant features for UAV-satellite matching

## Related Artifacts

- Previous assessment research: `_docs/00_research/gps_denied_nav_assessment/`
- Draft02 assessment research: `_docs/00_research/gps_denied_draft02_assessment/`
- This assessment research: `_docs/00_research/litesam_satellite_assessment/`
- Previous AC assessment: `_docs/00_research/gps_denied_visual_nav/00_ac_assessment.md`