gps-denied-onboard/_docs/02_tasks/done/AZ-320_c11_idempotent_retry.md

C11 Idempotent Retry — In-Call Retry Loop on Partial-Success Batches

Task: AZ-320_c11_idempotent_retry Name: C11 Idempotent Retry Decorator Description: Implement IdempotentRetryTileUploader, a decorator that wraps the AZ-319 TileUploader Protocol impl and adds bounded in-call retry on partial-success batches. After the underlying uploader returns outcome=partial, the decorator re-queries C6's pending_uploads (already-acked tiles were mark_uploaded'd, so the second pass naturally targets only the unacked subset), waits an exponential-backoff delay, and re-invokes the underlying upload. Caps at config.c11.max_in_call_retries (default 3); on budget exhaustion, the final report's outcome stays partial and next_retry_at_s carries an operator hint for when to retry later. A per-tile rejection counter in C6 metadata (upload_attempts) bounds the per-tile retry budget — after config.c11.max_per_tile_attempts (default 5), the tile is moved to voting_status = upload_giveup (a new enum value added by this task) and surfaced via FDR for human review. Complexity: 3 points Dependencies: AZ-263_initial_structure, AZ-269_config_loader, AZ-266_log_module, AZ-273_fdr_client_ringbuf, AZ-303_c6_storage_interfaces, AZ-319_c11_tile_uploader Component: c11_tilemanager (epic AZ-251 / E-C11) Tracker: AZ-320 Epic: AZ-251 (E-C11)

Document Dependencies

• _docs/02_document/contracts/c11_tilemanager/tile_uploader.md — the underlying Protocol this decorator wraps; the decorator itself implements the same Protocol (drop-in replacement).
• _docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md — consumed: pending_uploads, mark_uploaded, update_voting_status. This task adds an upload_attempts integer field and an upload_giveup value to VotingStatus — a contract change that bumps tile_metadata_store.md to v1.1.0 (non-breaking minor).
• _docs/02_document/contracts/shared_fdr_client/fdr_record_schema.md — kind="c11.upload.giveup" envelope.
• _docs/02_document/components/12_c11_tilemanager/tests.md — C11-IT-05 test scenario.

Problem

Without bounded in-call retry:

• C11-IT-05 ("idempotent uploads on retry — re-running upload_pending after a partial-success batch only POSTs the tiles that weren't acknowledged before") relies on the operator manually re-invoking upload_pending. Operators tolerate one re-invocation but resent doing it 3-4 times after transient satellite-provider flakiness.
• A single tile that ALWAYS fails (e.g., truncated tile_blob in C6 fails ingest validation forever) becomes a poison pill — every retry attempt re-uploads it AND every other unacked tile, wasting bandwidth and signing cycles. Without a per-tile budget, the operator cannot distinguish transient failures from terminal ones.
• The next_retry_at_s field of UploadBatchReport (per AZ-319 contract) has no producer — without backoff calculation, the field is always None and the operator gets no hint on retry timing.
• The parent suite's voting layer assumes uploaded tiles are eventually-consistent; an unbounded retry loop with no per-tile state would create lockstep retry storms.

This task delivers the retry decorator. It changes NO underlying logic in AZ-319; it composes.

Outcome

• An IdempotentRetryTileUploader class at src/gps_denied_onboard/components/c11_tilemanager/idempotent_retry.py:
  • Implements the TileUploader Protocol (drop-in for HttpTileUploader).
  • Constructor: __init__(self, *, inner: TileUploader, tile_metadata_store: TileMetadataStore, fdr_client: FdrClient, logger: Logger, clock: Clock, config: C11RetryConfig).
  • C11RetryConfig is a frozen dataclass with max_in_call_retries: int = 3, max_per_tile_attempts: int = 5, backoff_base_s: float = 2.0, backoff_cap_s: float = 60.0.
• upload_pending_tiles(request) flow:
  1. Calls inner.upload_pending_tiles(request) once.
  2. If the inner returns outcome in (success, failure) → return as-is.
  3. If outcome == partial:
     • For each PerTileStatus.status == rejected, increments the tile's upload_attempts in C6 via a new tile_metadata_store.increment_upload_attempts(tile_id) method.
     • For each tile whose upload_attempts >= config.max_per_tile_attempts, calls tile_metadata_store.update_voting_status(tile_id, VotingStatus.UPLOAD_GIVEUP); emits FDR kind="c11.upload.giveup" with {tile_id, attempts, last_rejection_reason}; emits ERROR log.
     • If retries_used < config.max_in_call_retries AND there are still tiles with voting_status == pending:
       • Sleeps min(config.backoff_base_s ** retries_used, config.backoff_cap_s) seconds via injected Clock.sleep.
       • Recurses with retries_used += 1 (via internal helper, NOT actual recursion — bounded loop).
     • Else (budget exhausted):
       • Aggregates the final UploadBatchReport: outcome = partial; retry_count = retries_used; next_retry_at_s = clock.now() + config.backoff_cap_s (operator hint).
       • Returns the aggregated report.
• enumerate_pending_tiles(flight_id) and confirm_flight_state() pass through to the inner unchanged.
• A new VotingStatus.UPLOAD_GIVEUP enum value is added to AZ-303's VotingStatus (in C6's _types.py); this is a non-breaking minor bump of tile_metadata_store.md to v1.1.0 — the producer (AZ-303) stays in v1, but C6's contract file's Change Log is appended by this task with a note pointing to the bump.
• A new tile_metadata_store.increment_upload_attempts(tile_id) -> int method is added to AZ-303's TileMetadataStore Protocol; returns the new attempt count post-increment. This is a Protocol surface addition (minor bump). The implementation lives in AZ-305's PostgresFilesystemStore. This task adds:
  • The Protocol method declaration in c6_tile_cache/interface.py.
  • The impl in c6_tile_cache/postgres_filesystem_store.py (a single SQL UPDATE ... SET upload_attempts = upload_attempts + 1 WHERE tile_id = $1 RETURNING upload_attempts).
  • The Postgres column upload_attempts INTEGER NOT NULL DEFAULT 0 via a NEW alembic migration _alembic/0002_upload_attempts.sql (NOT modifying AZ-304's 0001 migration; per coderule.mdc migrations are append-only).
• The composition root wraps HttpTileUploader with IdempotentRetryTileUploader by default. A config.c11.disable_retry_decorator: bool = false lets operators bypass the decorator for debugging.
• INFO log on session start with retry config; INFO log per retry attempt with attempt_number, sleep_s, remaining_pending_count; ERROR log on per-tile giveup; FDR kind="c11.upload.giveup" per tile.

Scope

Included

• IdempotentRetryTileUploader decorator class.
• C11RetryConfig frozen dataclass.
• VotingStatus.UPLOAD_GIVEUP enum value addition (in C6's _types.py).
• tile_metadata_store.increment_upload_attempts(tile_id) -> int Protocol method addition + AZ-305 SQL impl.
• _alembic/0002_upload_attempts.sql migration — adds upload_attempts INTEGER NOT NULL DEFAULT 0 column to the tiles table.
• Composition-root wiring (decorate HttpTileUploader by default; config.c11.disable_retry_decorator lets operators opt out).
• Bumping tile_metadata_store.md to v1.1.0 with a Change Log entry.
• INFO/ERROR logs and FDR c11.upload.giveup emission.
• Conformance test: isinstance(IdempotentRetryTileUploader(...), TileUploader).

Excluded

• The underlying HttpTileUploader impl — owned by AZ-319.
• The decision rule for what counts as a transient vs. terminal rejection — the decorator treats EVERY rejection as transient until the per-tile attempt budget is hit; the operator may manually move UPLOAD_GIVEUP tiles back to pending after investigation (out-of-band SQL UPDATE; no API surface).
• A separate background-retry daemon — the retry happens within upload_pending_tiles; the operator decides when to invoke it.
• Cross-process retry coordination — the C12 lockfile already prevents concurrent C11 invocations.
• Surfacing UPLOAD_GIVEUP in the operator-tooling CLI — owned by E-C12.
• Auto-promotion of UPLOAD_GIVEUP back to pending after manual fixes — operator concern; out of scope.

Acceptance Criteria

AC-1: Success on first attempt — no retry Given the inner uploader returns outcome = success on the first call When upload_pending_tiles(request) is called Then the decorator returns immediately; ZERO calls to Clock.sleep; ZERO calls to increment_upload_attempts; report passes through unchanged

AC-2: Partial-success with retry budget available Given inner returns outcome=partial with 3 of 10 tiles rejected (per_tile_status), and max_in_call_retries=3 When the decorator processes the partial Then increment_upload_attempts is called 3 times (one per rejected tile); Clock.sleep(2.0) is called once; inner is re-invoked; if the second attempt is success, the final aggregated report shows outcome = success and retry_count = 1

AC-3: Per-tile budget exhausted moves tile to UPLOAD_GIVEUP Given a tile whose upload_attempts reaches max_per_tile_attempts=5 When the decorator increments the counter Then update_voting_status(tile_id, UPLOAD_GIVEUP) is called; ONE FDR kind="c11.upload.giveup" is emitted with {tile_id, attempts=5, last_rejection_reason}; ONE ERROR log; the tile is NOT re-uploaded in subsequent retries (since pending_uploads excludes UPLOAD_GIVEUP)

AC-4: In-call retry budget exhausted Given inner consistently returns outcome=partial with the same rejected tile, and max_in_call_retries=3 When the decorator runs out of in-call retries Then 3 retries are attempted (4 total inner calls including the first); Clock.sleep is called 3 times with backoffs 2.0, 4.0, 8.0; the final report has outcome=partial, retry_count=3, next_retry_at_s = clock.now() + backoff_cap_s

AC-5: Backoff cap honoured Given max_in_call_retries=10 and backoff_cap_s=10 When the decorator computes the 6th retry delay Then Clock.sleep(10.0) is called (capped at 10s, not 2^6 = 64s)

AC-6: VotingStatus.UPLOAD_GIVEUP enum exposed Given the AZ-303 VotingStatus enum (post-this-task) When a consumer imports it Then VotingStatus.UPLOAD_GIVEUP is present alongside PENDING, TRUSTED, REJECTED; the contract file's Change Log shows v1.1.0

AC-7: increment_upload_attempts returns new count Given a tile with upload_attempts = 2 When increment_upload_attempts(tile_id) is called Then the SQL row's upload_attempts is now 3; the method returns 3; concurrent invocations on different tiles produce no contention (per-row lock)

AC-8: Migration 0002 adds the column Given a fresh DB at AZ-304's 0001 migration When 0002 is applied Then the tiles table has an upload_attempts INTEGER NOT NULL DEFAULT 0 column; existing rows have upload_attempts = 0; the migration is reversible (drops the column on rollback)

AC-9: Decorator is a drop-in for the Protocol Given an IdempotentRetryTileUploader instance When isinstance(impl, TileUploader) is checked under runtime_checkable Then the result is True; consumers that depend on the Protocol see no shape difference

AC-10: disable_retry_decorator config bypass Given config.c11.disable_retry_decorator = true When the composition root constructs the uploader Then build_tile_uploader(config) returns the bare HttpTileUploader (no decorator); a debug INFO log records the bypass

AC-11: Pass-through methods Given the decorator When enumerate_pending_tiles(flight_id) and confirm_flight_state() are called Then both delegate to inner directly with no added logic

AC-12: Inner exception propagates without retry Given inner raises FlightStateNotOnGroundError or SatelliteProviderError When the decorator catches the exception Then it re-raises immediately; no retry is attempted (these are not partial-success cases); ZERO Clock.sleep calls

AC-13: Idempotent across re-invocations (the C11-IT-05 scenario) Given a 50-tile batch where 30 succeed and 20 are rejected on first call (no in-call retry to avoid mixing); operator re-invokes after 5 minutes When the second call runs Then pending_uploads returns only the 20 tiles (the 30 are already voting_status = uploaded); inner.upload_pending_tiles is called with the request; only those 20 are POSTed; the 30 are NOT re-sent

Non-Functional Requirements

Performance

• Decorator overhead per upload_pending_tiles call ≤ 5 ms (plus Clock.sleep time, which is intentional).
• increment_upload_attempts SQL call ≤ 5 ms p99 against the local Postgres.

Compatibility

• The new migration is append-only (NOT a modification of AZ-304's 0001 migration).
• The new VotingStatus.UPLOAD_GIVEUP value is additive (non-breaking).
• increment_upload_attempts is a Protocol method addition; existing AZ-303 conformance tests pass (the method has a default impl that raises NotImplementedError if a future implementation forgets it — but the AZ-305 impl provides the SQL version).

Reliability

• The retry loop is bounded by BOTH max_in_call_retries AND max_per_tile_attempts; neither alone can produce unbounded behaviour.
• The decorator does NOT swallow exceptions from inner; only outcome=partial results are eligible for retry.
• The injected Clock.sleep makes retry timing deterministic in tests.

Unit Tests

AC Ref	What to Test	Required Outcome
AC-1	Inner success	Pass-through; zero retries
AC-2	Partial → retry → success	One Clock.sleep(2.0); retry_count=1; final outcome=success
AC-3	Per-tile attempts hits 5	update_voting_status called; FDR + ERROR log emitted
AC-4	Persistent partial across 4 attempts	Clock.sleep(2.0), (4.0), (8.0); retry_count=3; final partial
AC-5	Cap=10 with high attempt number	Clock.sleep(10.0) not Clock.sleep(64.0)
AC-6	Import VotingStatus.UPLOAD_GIVEUP	Present; tile_metadata_store.md v1.1.0
AC-7	Concurrent increment_upload_attempts on different tiles	No deadlock; correct counts
AC-8	Apply 0002 migration	Column added; default 0; rollback drops
AC-9	isinstance check	True
AC-10	Config bypass	Bare impl; debug log
AC-11	Pass-through methods	Delegated unchanged
AC-12	Inner raises	Re-raised; zero retries
AC-13	Two-call scenario across operator re-invocations	First call: 30 acked / 20 rejected; second call: only 20 POSTed
NFR-perf-overhead	Microbench decorator with success-on-first	≤ 5 ms overhead

Constraints

• The decorator MUST be a drop-in for TileUploader; the composition root selects via config.c11.disable_retry_decorator only.
• The retry budget is per-call (in-call) and per-tile (across calls); neither budget alone fully bounds — both are required.
• increment_upload_attempts is the ONLY method that mutates upload_attempts; consumers do NOT directly UPDATE the column. This is a contract invariant; code-review treats direct UPDATEs as Architecture finding (High).
• The UPLOAD_GIVEUP voting status is a HUMAN-decision boundary — automated promotion back to pending is forbidden in this task. An out-of-band SQL UPDATE by the operator is the documented recovery path.
• The migration 0002 is APPEND-ONLY relative to 0001; it does NOT alter existing column types.
• This task introduces no new third-party dependencies.

Risks & Mitigation

Risk 1: AZ-303 contract bump cascades to other consumers

• Risk: Adding increment_upload_attempts to the Protocol forces every existing C6 consumer (C2 VPR, C2.5 ReRanker, C3 CrossDomainMatcher, C10 CacheProvisioner, C12 OperatorTooling) to re-confirm conformance.
• Mitigation: The new method is OPTIONAL via a Protocol default impl that raises NotImplementedError; consumers that don't call it are unaffected. The conformance test verifies only that AZ-305's impl provides it.

Risk 2: Backoff cap interacts badly with operator workflows

• Risk: A 60-second cap means the operator may walk away during retries; the visible CLI hangs.
• Mitigation: The decorator emits an INFO log per retry attempt with attempt_number, sleep_s, remaining_pending_count; C12's CLI surfaces this so the operator sees progress. Cap is configurable.

Risk 3: UPLOAD_GIVEUP tiles accumulating without operator visibility

• Risk: A subtle data corruption in C6 causes 100% of tiles to hit UPLOAD_GIVEUP; the operator notices only when they manually inspect C6.
• Mitigation: Each UPLOAD_GIVEUP event emits FDR kind="c11.upload.giveup" AND ERROR log; C12's CLI summary surfaces the count post-upload-run. This task adds NO direct UI; C12's task list will include surfacing.

Risk 4: Clock.sleep blocking on KeyboardInterrupt

• Risk: A long backoff (60s) blocks the process; Ctrl+C aborts mid-sleep but might leave state inconsistent.
• Mitigation: The decorator uses the injected Clock which is the same singleton as AZ-307/AZ-308; KeyboardInterrupt propagates upward and AZ-319's try/finally still runs key_manager.end_session(); the decorator's own state is just the retry counter (in-memory; no on-disk side effects between retries).

Runtime Completeness

• Named capability: bounded in-call retry on partial-success uploads, per-tile retry budget with UPLOAD_GIVEUP terminal state, operator-friendly next_retry_at_s hint (description.md § 5, C11-IT-05).
• Production code that must exist: real IdempotentRetryTileUploader decorator, real increment_upload_attempts SQL, real migration 0002, real VotingStatus.UPLOAD_GIVEUP enum value, real composition-root wiring with the bypass flag.
• Allowed external stubs: tests MAY use a fake inner (mock TileUploader implementing the Protocol with scripted responses), fake Clock, fake tile_metadata_store (already provided by AZ-303 conformance fakes); production wiring uses real all the way down.
• Unacceptable substitutes: a recursive Python implementation of the retry loop (stack-explosion risk; bounded iteration is required); skipping the per-tile budget (lets one bad tile poison every retry); silently moving tiles to UPLOAD_GIVEUP without FDR (loses safety officer surface); modifying AZ-304's 0001 migration in place (breaks deployment idempotence — migrations are append-only).