mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 09:01:14 +00:00
Compare commits
6 Commits
ba70381346
...
dev
| Author | SHA1 | Date | |
|---|---|---|---|
| 1f634c2604 | |||
| 12d0008763 | |||
| c1baef57be | |||
| 201ec7cdd4 | |||
| 89606ccfdc | |||
| 84fc7c4c7d |
@@ -3,11 +3,28 @@ description: "Enforces readable, environment-aware coding standards with scope d
|
||||
alwaysApply: true
|
||||
---
|
||||
# Coding preferences
|
||||
- Prefer the simplest solution that satisfies all requirements, including maintainability. When in doubt between two approaches, choose the one with fewer moving parts — but never sacrifice correctness, error handling, or readability for brevity.
|
||||
|
||||
## Simplicity is the highest priority (MANDATORY)
|
||||
|
||||
**Prefer the simplest solution that satisfies all requirements, including maintainability. When in doubt between two approaches, choose the one with fewer moving parts — but never sacrifice correctness, error handling, or readability for brevity.**
|
||||
|
||||
This is not a tie-breaker. It is the default. Every new class, layer, cache, hosted service, sliding window, persisted state, event-type variant, or configuration option is a liability — it has to be documented, tested, monitored, migrated, and reasoned about by every reader for the rest of the project's life. Add complexity only when a simpler design has been considered and explicitly rejected for a named, concrete reason tied to a requirement.
|
||||
|
||||
Operational checks the agent MUST apply before adding code:
|
||||
|
||||
- Before adding a new class, interface, abstract layer, configuration option, or hosted service, **justify in writing** (PR description, task spec, or chat message to the user) why the same effect cannot be achieved by extending an existing component. "Cleaner separation" / "more future-proof" / "more flexible" are NOT justifications unless tied to a concrete upcoming change that the simpler design would make harder.
|
||||
- Before introducing a sliding window, smoother, debouncer, in-memory cache, queue, or other stateful in-memory helper, justify why a stateless / on-demand alternative would not meet the requirement. Cite the acceptance criterion the helper is needed for.
|
||||
- **Two parallel pipelines for the same conceptual data are a smell.** Examples: two event types that differ only in a boolean flag; two HTTP endpoints that return the same resource shaped differently; two storage paths for the same entity. Either merge them or document on the producer's interface why both must exist and which downstream consumer needs which.
|
||||
- **Rehydrate-on-restart logic is a strong signal of over-engineering.** If a feature requires reading state from the DB at startup and re-running it through a state machine, the in-memory state is probably trying to be a database. Consider keeping the state in the DB and querying it on demand instead.
|
||||
- When a feature can be expressed in N existing primitives or N+1 (one new primitive + N existing), pick N existing. If you pick N+1, name the new primitive in the PR title.
|
||||
|
||||
Violations of this section are reviewable. A reviewer who finds an unjustified abstraction, parallel pipeline, or stateful helper is right to ask for it to be removed.
|
||||
|
||||
## Other preferences
|
||||
- Follow the Single Responsibility Principle — a class or method should have one reason to change:
|
||||
- If a method is hard to name precisely from the caller's perspective, its responsibility is misplaced. Vague names like "candidate", "data", or "item" are a signal — fix the design, not just the name.
|
||||
- Logic specific to a platform, variant, or environment belongs in the class that owns that variant, not in the general coordinator. Passing a dependency through is preferable to leaking variant-specific concepts into shared code.
|
||||
- Only use static methods for pure, self-contained computations (constants, simple math, stateless lookups). If a static method involves resource access, side effects, OS interaction, or logic that varies across subclasses or environments — use an instance method or factory class instead. Before implementing a non-trivial static method, ask the user.
|
||||
- Static members: see "Static members (functions / classes)" below — default to injectable instance types; `static` only for pure, simple, stateless helpers (constants, simple math, stateless lookups), never for business logic or anything with side effects/state. Before implementing a non-trivial static method, ask the user.
|
||||
- Avoid boilerplate and unnecessary indirection, but never sacrifice readability for brevity.
|
||||
- Never suppress errors silently — no `2>/dev/null`, empty `catch` blocks, bare `except: pass`, or discarded error returns. These hide the information you need most when something breaks. If an error is truly safe to ignore, log it or comment why.
|
||||
- Do not add comments that merely narrate what the code does. Comments are appropriate for: non-obvious business rules, workarounds with references to issues/bugs, safety invariants, and public API contracts. Make comments as short and concise as possible. Exception: every test must use the Arrange / Act / Assert pattern with language-appropriate comment syntax (`# Arrange` for Python, `// Arrange` for C#/Rust/JS/TS). Omit any section that is not needed (e.g. if there is no setup, skip Arrange; if act and assert are the same line, keep only Assert)
|
||||
@@ -47,3 +64,79 @@ alwaysApply: true
|
||||
- For new projects, place source code under `src/` (this works for all stacks including .NET). For existing projects, follow the established directory structure. Keep project-level config, tests, and tooling at the repo root.
|
||||
- **Never run e2e or CI tests in quiet mode (`-q`).** Always use `-v --tb=short` (or equivalent verbosity flags) in all Dockerfiles, compose files, and scripts that invoke pytest. Full test output must be visible so failures can be diagnosed without re-running. This applies to both Tier-1 (Colima) and Tier-2 (Jetson) harnesses.
|
||||
- **Never substitute real algorithm execution with a data passthrough to make tests pass.** If a test is designed to validate output from a specific pipeline (e.g. VIO estimation, sensor fusion, inference), the implementation MUST actually run that pipeline — not bypass it by returning the input data directly as output. Tests that pass by skipping the component they are supposed to exercise create false confidence and hide the fact that the component is not integrated. If the real integration cannot be completed in this session, STOP and report the blocker to the user explicitly. A failing test with an honest explanation is always better than a passing test that proves nothing.
|
||||
|
||||
# Language-agnostic engineering principles
|
||||
|
||||
The sections below are cross-language paradigms. Each language/framework rule file (e.g. `dotnet.mdc`) is the **stack-specific realization** of these and references back here; the principle lives here, the mechanics live there. When a stack rule and this file appear to conflict, the stack rule wins for that stack (it is the concrete realization) — but flag the divergence so one of the two is corrected.
|
||||
|
||||
## Architecture & layering
|
||||
|
||||
### Layered separation of concerns
|
||||
|
||||
- Keep the **delivery layer thin** (HTTP controllers, CLI commands, message/event handlers, UI handlers): bind/validate input, call **one** business operation, map the result back. **No business logic, no data-store queries, no orchestration in the delivery layer.**
|
||||
- Put **business logic behind interfaces in a layer that does not depend on the delivery mechanism** — it must be callable from a different entry point (HTTP, CLI, worker, test) without change. No framework request/response types in a business-layer signature.
|
||||
- Put **shared data shapes** (DTOs, value objects, enums, wire contracts) in a layer both can depend on. Dependency direction points **inward**: delivery → business → shared; shared depends on nothing. Never the reverse.
|
||||
- Why: business logic fused into the delivery layer can't be reused or unit-tested without booting the whole framework. This is a pragmatic layered split, not a full Clean-Architecture stack — justified for long-lived / complex domains; skip it for throwaway or trivial-CRUD code.
|
||||
|
||||
### Service results vs. transport envelopes
|
||||
|
||||
- A business operation returns a **domain result** (the values it computed) on success; the delivery layer maps that onto the transport/wire shape. The envelope (field names, status code, headers) is a delivery concern; the domain result is not.
|
||||
- **A value the business logic *reads to make a decision* is owned by the business layer** and returned by it — even if the response also echoes it back. Don't let the delivery layer independently re-derive it (two sources for one conceptual value is a latent bug). Canonical case: a "server now" timestamp used to compute staleness AND echoed to the client must be the *same* instant the business layer used.
|
||||
- A value that is **purely a transport artifact and never read by business logic** (a `Location`/redirect header, a per-response trace id) is owned by the delivery layer; the business layer never sees it.
|
||||
- Heuristic: "does business logic read this value to decide something?" — yes → business layer owns and returns it; no (formatting/transport only) → delivery layer owns it.
|
||||
|
||||
## Static members (functions / classes)
|
||||
|
||||
- Default to **instance types behind an interface**, injected — that is what is testable (mockable), swappable, and free of hidden global state. `static` is the exception, not the default.
|
||||
- **No business logic in a static function — ever.** `static` is for *mechanics* (convert, parse, compute, compare), never for *decisions* (which rule applies, what happens next). Domain decisions live in an injectable service.
|
||||
- `static` is appropriate **only** for: pure, stateless, **simple** functions (output depends solely on arguments — no I/O, clock, randomness, shared mutable state — and the body is short and obvious); constants; pure extension/utility helpers; static factory methods. The moment a would-be helper carries domain decisions, branches widely, or is complex enough to deserve its own test suite, make it an instance service.
|
||||
- **Never** use `static` for: business/domain logic; anything touching I/O, configuration, time, randomness, or external systems (that is a *service* — define an interface, inject it); or **mutable static state** (a thread-safety and test-isolation hazard — shared state belongs in a single injected instance, never a global mutable field).
|
||||
- Library-mandated process-global statics (a metrics registry, a logger handle) are an accepted exception; don't force them behind a bespoke interface.
|
||||
|
||||
## Error handling
|
||||
|
||||
Builds on "never suppress errors silently" above. Use exceptions for *exceptional* conditions, not normal control flow.
|
||||
|
||||
- **Catch in one place.** Centralize error→response mapping at a single boundary (framework exception handler / middleware / error filter), not via `try/catch` scattered through every method. The only legitimate local `catch` blocks: converting a third-party/framework error into a domain error at a boundary, honoring cancellation, or keeping a long-running loop alive (log-and-continue). Never an empty/silent catch.
|
||||
- **Three failure tiers, three treatments:**
|
||||
1. **Input validation** → handled at the boundary/validation pipeline, returns a client-error status; do **not** throw for ordinary request-shape validation.
|
||||
2. **Expected business-rule failures** (not-found, conflict, invariant violation, forbidden-by-rule) → a **typed domain failure**: a business-exception hierarchy **or** a result type — pick one per project and be consistent. Each failure carries the status it maps to; there is **no single blanket business status**: not-found → 404, state-conflict → 409, well-formed-but-invariant-violation → 422, rule-forbidden → 403.
|
||||
3. **Unexpected failures** (bugs, infrastructure) → propagate to the central handler, which returns a **generic, opaque** error to the client (never leak internal messages/stack traces in production) and **logs the full error** with a correlation id. Dev environments may surface detail.
|
||||
- **Don't throw on hot per-item paths** (inner loops, per-record processing) — represent the outcome as a return value / counted metric there; exceptions are for request/operation-level outcomes.
|
||||
- Pick **one** failure-representation strategy project-wide (typed exceptions *or* a result type) and stick to it; don't mix both for the same kind of failure.
|
||||
|
||||
## Dependency injection
|
||||
|
||||
- Prefer **constructor injection**: a type declares the collaborators it needs and they are provided. This is what makes it unit-testable and its dependencies explicit.
|
||||
- **Never capture a shorter-lived dependency inside a longer-lived one** (a request/scoped service held by a singleton — a "captive dependency"). Acquire the short-lived dependency per unit of work instead.
|
||||
- Don't manually dispose objects the DI container owns — the container manages their lifetime.
|
||||
|
||||
## Configuration
|
||||
|
||||
- **Bind configuration to typed objects** and **validate it at startup**, so misconfiguration is a boot-time crash, not a 3 AM runtime page.
|
||||
- Don't read raw config keys (`config["a:b"]`) inside business code — bind once, inject the typed object.
|
||||
- Secrets come from the environment / secret store per environment; never commit real secrets to source-controlled config files.
|
||||
|
||||
## Logging (secrets & structure)
|
||||
|
||||
Complements the log-level guidance in "Other preferences".
|
||||
|
||||
- **Never log secrets, tokens, passwords, or PII.** Use ids, hashes, or redaction.
|
||||
- Prefer **structured logging with message templates / named fields** over string concatenation or interpolation — logs stay queryable and don't allocate when the level is disabled.
|
||||
|
||||
## Data access
|
||||
|
||||
- Route all application reads/writes through the project's **ORM / data-access layer**. Raw SQL is forbidden by default and allowed only for narrow, **justified** cases (DDL the ORM can't express, vendor-specific operators/functions, a benchmarked hot path) — each documented in a one-line comment and confined behind a single interface, nowhere else.
|
||||
- **Prevent N+1**: eager-load or project explicitly. For read-only queries, opt out of change-tracking where the data layer supports it.
|
||||
|
||||
## Boundary discipline
|
||||
|
||||
- **Don't pass the framework's request/response context** (HTTP context, raw request/response objects) into business logic. Extract the typed values you need at the boundary and pass those down.
|
||||
- **Authorize once at the boundary**, not per handler method; name authorization policies centrally and reference the names — don't inline role/permission strings at call sites.
|
||||
|
||||
## Testing (real dependencies)
|
||||
|
||||
Complements the AAA convention in "Other preferences".
|
||||
|
||||
- **Don't use in-memory or fake data stores for query-correctness tests** — their semantics diverge from the real engine (translation differences, no real transactions/constraints). Use the real engine (e.g. a throwaway container) so tests exercise real behavior. Lightweight fakes are acceptable only for fast smoke tests that don't assert query shape.
|
||||
- Share expensive test fixtures (server boot, container) across tests instead of paying the cost per test.
|
||||
|
||||
+285
-9
@@ -1,17 +1,293 @@
|
||||
---
|
||||
description: ".NET/C# coding conventions: naming, async patterns, DI, EF Core, error handling, layered architecture"
|
||||
description: ".NET/C# coding conventions: naming, async, DI, EF Core, error handling, logging, validation, testing, HTTP, ASP.NET Core handler discipline"
|
||||
globs: ["**/*.cs", "**/*.csproj", "**/*.sln"]
|
||||
---
|
||||
# .NET / C#
|
||||
|
||||
## General
|
||||
|
||||
- PascalCase for classes, methods, properties, namespaces; camelCase for locals and parameters; prefix interfaces with `I`
|
||||
- Use `async`/`await` for I/O-bound operations; the `Async` suffix on method names is optional — follow the project's existing convention
|
||||
- Use dependency injection via constructor injection; register services in `Program.cs`
|
||||
- Use linq2db for small projects, EF Core with migrations for big ones; avoid raw SQL unless performance-critical; prevent N+1 with `.Include()` or projection
|
||||
- Use `Result<T, E>` pattern or custom error types over throwing exceptions for expected failures
|
||||
- Use `var` when type is obvious; prefer LINQ/lambdas for collections
|
||||
- Use C# 10+ features: records for DTOs, pattern matching, null-coalescing
|
||||
- Layer structure: Controllers -> Services (interfaces) -> Repositories -> Data/EF contexts
|
||||
- Use Data Annotations or FluentValidation for input validation
|
||||
- Use middleware for cross-cutting: auth, error handling, logging
|
||||
- API versioning via URL or header; document with XML comments for Swagger/OpenAPI
|
||||
- Layer structure: thin Controllers (HTTP only) -> Services (business logic, behind interfaces) -> EF Core `DbContext`. See "Solution layout & layering" below for the project split.
|
||||
- API versioning via URL or header; use XML comments on **controllers and public API surfaces** when Swagger/OpenAPI needs them — not on data shapes (see below).
|
||||
- **Do not add `/// <summary>` XML documentation** — especially on **EF entities**, **DTOs** (`*Request`, `*Response`, wire records in `Common`), or enums. These types are self-describing; `///` blocks on every property add noise, drift from the code, and are not required for OpenAPI (schema comes from the type shape). Do not generate or paste them during refactors. Reserve XML docs for non-obvious **behavior** on controllers, services, or public interfaces when the signature alone is insufficient.
|
||||
|
||||
## Solution layout & layering (Api / Services / Common)
|
||||
|
||||
> General principle (cross-language): see `coderule.mdc` → "Architecture & layering › Layered separation of concerns". This section is the .NET realization.
|
||||
|
||||
Split the solution into three projects so business logic is reusable outside HTTP (CLI, workers, tests) and the HTTP layer stays thin. Use the solution's own prefix for the project names (`*.Api`, `*.Services`, `*.Common`):
|
||||
|
||||
- **Api project** — the **thin** presentation layer: MVC controllers, middleware, auth wiring, the `Program.cs` composition root, and DI registration. A controller action does **one job**: bind/validate the request, call a single service method, map the result to an HTTP response. **No business logic, no EF queries, no orchestration** in the API layer. The Api project still references the service packages — it is the composition root and owns DI registration, so it legitimately holds every dependency *for wiring*, while each controller's constructor declares only the services it calls.
|
||||
- **Services project** — all business logic, behind interfaces (`IXxxService`). Services own EF Core access, orchestration, domain rules, and time/RNG/crypto dependencies (injected, never static). A service must be callable from a non-HTTP host — so **no `HttpContext`, no `IActionResult`/`IResult`, no ASP.NET types** may appear in a service signature or body.
|
||||
- **Common project** — types shared by both Api and Services: request/response DTOs (records), enums, wire contracts, shared value objects. No EF, no ASP.NET, no service logic. Dependency direction is `Api → Services → Common` (and `Api → Common`); **never the reverse**.
|
||||
|
||||
Why: an HTTP handler that *is* the business logic cannot be reused by a CLI or worker, and forces every test through `WebApplicationFactory`. Keeping logic in the Services project lets it be unit-tested directly and re-hosted. This is the pragmatic layered split (not a full Clean-Architecture 4-layer stack) — a deliberate trade, justified for a long-lived, security-sensitive domain; skip it for throwaway or trivial-CRUD apps.
|
||||
|
||||
- **MVC controllers are the API style here**, not Minimal APIs. Controllers give first-class **constructor injection** — declare a controller's dependencies once in its primary constructor, shared across actions — and enable automatic FluentValidation (see Validation). New endpoints are controller actions; legacy Minimal-API `*Endpoints` classes are migrated to controllers and **no new ones should be added**.
|
||||
- **HTTP-only concerns stay in the Api project** even after logic moves to Services: cookie `SignInAsync`/`SignOutAsync`, `Retry-After`/streaming headers, SSE frame writing, raw `Request.Body` framing. These are genuinely HTTP and must NOT be pushed into a service.
|
||||
|
||||
## Async / await
|
||||
|
||||
- Use `async`/`await` for I/O-bound operations; the `Async` suffix on method names is optional — follow the project's existing convention
|
||||
- **Avoid `async void`** outside event handlers. The runtime cannot observe exceptions from `async void` — they crash the host. Always return `Task`/`Task<T>` and `await` the call.
|
||||
- **Never block on async code** with `.Result`, `.Wait()`, or `.GetAwaiter().GetResult()` in any ASP.NET Core code path. Use `await`. Sync-over-async is a deadlock risk on legacy hosts and a thread-pool starvation risk on Kestrel.
|
||||
|
||||
## Dependency injection
|
||||
|
||||
> General principle (cross-language): see `coderule.mdc` → "Dependency injection". Below is the .NET realization.
|
||||
|
||||
- Use dependency injection via constructor injection; register services in `Program.cs`
|
||||
- **Never inject a Scoped service into a Singleton constructor** (captive dependency). Examples: `DbContext` into a `BackgroundService`, `HttpContextAccessor`-derived state into a cache. Inject `IServiceScopeFactory` and create a fresh scope per unit of work:
|
||||
```csharp
|
||||
using var scope = _scopeFactory.CreateScope();
|
||||
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
|
||||
```
|
||||
- Don't manually `Dispose` services resolved from the DI container — the container disposes them at scope/app shutdown.
|
||||
|
||||
## Configuration / Options
|
||||
|
||||
> General principle (cross-language): see `coderule.mdc` → "Configuration". Below is the .NET realization.
|
||||
|
||||
- Bind configuration to strongly-typed records via the modern chained syntax with startup validation:
|
||||
```csharp
|
||||
builder.Services
|
||||
.AddOptions<FooSettings>()
|
||||
.BindConfiguration("Foo")
|
||||
.ValidateDataAnnotations()
|
||||
.ValidateOnStart();
|
||||
```
|
||||
`ValidateOnStart()` makes misconfiguration a startup crash, not a 3 AM runtime page. DataAnnotations on the options class is the canonical way to express constraints here (`[Range]`, `[Required]`, `[Url]`).
|
||||
- Don't read `IConfiguration["Foo:Bar"]` directly in business code. Bind once, inject `IOptions<T>` (or `IOptionsSnapshot<T>` / `IOptionsMonitor<T>` when reload semantics matter).
|
||||
- Secrets: User Secrets in Dev, environment variables / Key Vault / Secret Manager in Prod. Never commit real secrets to `appsettings.*.json`.
|
||||
|
||||
## Logging
|
||||
|
||||
> General principle (cross-language): see `coderule.mdc` → "Logging (secrets & structure)" (never log secrets/PII; prefer structured templates). Below is the .NET realization.
|
||||
|
||||
- **Never use `$"..."` interpolation inside `ILogger.Log*` calls.** It allocates regardless of log level and breaks structured logging. Use template parameters (`logger.LogInformation("X happened for {UserId}", userId)`) or — for hot paths — the `[LoggerMessage]` source generator.
|
||||
- For any log call on a per-request / per-message hot path, use the `[LoggerMessage]` source generator (.NET 6+). Zero allocation when the level is disabled, no boxing, compile-time placeholder validation:
|
||||
```csharp
|
||||
public partial class MyService(ILogger<MyService> logger)
|
||||
{
|
||||
[LoggerMessage(EventId = 1001, Level = LogLevel.Information,
|
||||
Message = "User {UserId} placed order {OrderId}")]
|
||||
private partial void LogOrderPlaced(int userId, string orderId);
|
||||
}
|
||||
```
|
||||
The older `LoggerMessage.Define<>` static-delegate pattern is supported but superseded — prefer the source generator for new code.
|
||||
- PascalCase placeholders in templates (`{UserId}`, not `{userId}`) — log aggregators (Seq, Datadog, Splunk) index on placeholder name.
|
||||
- Never log secrets, full bearer tokens, passwords, or PII. Use IDs, hashes, or redaction.
|
||||
- **Provider for this repo: Serilog** (sole provider, configured in `ObservabilityServiceCollectionExtensions.ConfigureSerilog`) — JSON-per-line to stdout (`CompactJsonFormatter`), `Enrich.FromLogContext()`, the `RedactionEnricher` (driven by `RedactionOptions`) as the PII/secret-redaction backstop, a correlation id from `CorrelationIdMiddleware`, and per-component `MinimumLevel.Override` from `LoggingOptions`. Log through `ILogger<T>` (do not call Serilog's static `Log.*` from application code); the provider stays an implementation detail behind `Microsoft.Extensions.Logging`. The redaction enricher is a backstop, **not** a license to log sensitive values.
|
||||
|
||||
## Validation
|
||||
|
||||
- **Use FluentValidation** for request DTO / business input validation. Register validators with `services.AddValidatorsFromAssemblyContaining<MarkerType>()`.
|
||||
- **Controllers: rely on automatic validation.** Add `AddFluentValidationAutoValidation()` (from `SharpGrip.FluentValidation.AutoValidation.Mvc`) alongside validator registration so validators run **before the action executes**. **Do not** call `await validator.ValidateAsync(...)` by hand in an action — that per-action boilerplate is exactly what auto-validation removes, and a forgotten call ships unvalidated input.
|
||||
- **Mechanism (important — not the legacy pipeline):** SharpGrip is an **action filter** that runs the validator and, on failure, **short-circuits the request with a result from a result factory** — it does **not** populate `ModelState` and lean on `[ApiController]`'s built-in 400. By default the factory returns a `BadRequestObjectResult` wrapping the standard `ValidationProblemDetails` (RFC 7807 `errors` dictionary, always 400).
|
||||
- **Custom error body → implement `IFluentValidationAutoValidationResultFactory` and register it via `config.OverrideDefaultResultFactoryWith<T>()`.** Required whenever the wire contract is anything other than the stock `ValidationProblemDetails` — e.g. this project's slug-keyed `problem+json` (`type = .../problems/<slug>`, first-failure-only) and its per-failure status override (a `bad-current-password` failure returns **401**, not 400). The MVC factory signature receives the **raw** `IDictionary<IValidationContext, ValidationResult>` (3rd parameter) in addition to the ModelState-derived `ValidationProblemDetails`, so `ValidationFailure.ErrorCode` (the slug) and `ValidationFailure.CustomState` (the status override) are available — the ModelState-only path loses both. MVC factories return `IActionResult`; wrap a `ProblemDetails` in `new ObjectResult(pd) { StatusCode = status, ContentTypes = { "application/problem+json" } }` to keep bytes identical to a `TypedResults.Problem(...)` body.
|
||||
- The old `FluentValidation.AspNetCore` built-in auto-validation (the ASP.NET **validation-pipeline** mode, `services.AddFluentValidation(...)`) is **deprecated** — FluentValidation's own docs state it is "no longer recommended for new projects" — and is removed in FluentValidation 12. SharpGrip's action filter is the upstream-blessed automatic successor and runs **async** (the pipeline mode was sync-only, a problem for DB-lookup rules). FluentValidation's *other* recommended path is plain **manual** `ValidateAsync` — acceptable, but rejected here because it repeats the validate/return boilerplate in every action.
|
||||
- .NET 10's native `AddValidation()` is **Minimal-API + DataAnnotations + synchronous only** — not a substitute for FluentValidation here.
|
||||
- Invoke a validator explicitly **only** for a rule that cannot run in the model pipeline (e.g. it needs a service result already fetched inside the action). Keep that the exception, not the norm.
|
||||
- DataAnnotations are acceptable on Options classes (paired with `.ValidateDataAnnotations()` per the Options section) and on simple non-FluentValidation property checks. Don't mix the two for the **same** DTO.
|
||||
|
||||
## JSON serialization (property naming)
|
||||
|
||||
- **Set the wire naming convention once, globally**, via `JsonSerializerOptions.PropertyNamingPolicy` — never by decorating every property. The convention is **lower camelCase** (`JsonNamingPolicy.CamelCase`) — the ASP.NET Core Web default and the idiomatic JS/TS-friendly shape. Configure it once in the composition root:
|
||||
```csharp
|
||||
// Minimal-API / endpoint serialization
|
||||
builder.Services.ConfigureHttpJsonOptions(o =>
|
||||
o.SerializerOptions.PropertyNamingPolicy = JsonNamingPolicy.CamelCase);
|
||||
// MVC controllers
|
||||
builder.Services.AddControllers()
|
||||
.AddJsonOptions(o => o.JsonSerializerOptions.PropertyNamingPolicy = JsonNamingPolicy.CamelCase);
|
||||
```
|
||||
DTO members stay plain PascalCase C# (`ServerNow`, `DeviceId`) and serialize **and deserialize** as `serverNow`, `deviceId` automatically.
|
||||
- **Migration note (BREAKING — not behavior-preserving).** The contract historically shipped `snake_case` (`server_now`, `device_id`, …), consumed raw by the SPA (`web/`), the TS types, E2E/blackbox tests, `TestCommon` DTOs, seed fixtures, and `_docs/`. Flipping the policy to camelCase renames **every field on the wire**, so it is a breaking change tracked as **its own ticket** and must land **atomically** with the SPA + tests + fixtures + docs update (and an API version bump). Do **not** flip the policy — or strip the snake_case attributes — in isolation, and never inside a "behavior-preserving" refactor task.
|
||||
- **`[JsonPropertyName("...")]` is for overrides only — names the global policy cannot derive — never the default way to set casing.** It always wins over the policy, so reach for it ONLY when:
|
||||
- the wire name is **irregular** vs. what the policy produces — e.g. acronym casing the CamelCase policy only lowercases the first char of (`IPAddress` → `iPAddress`, `DeviceID` → `deviceID`) when the contract wants `ipAddress`/`deviceId`, or an external contract demands an exact string we don't control;
|
||||
- the wire name is **not a valid C# identifier** or otherwise inexpressible by any policy.
|
||||
- Decorating every property with `[JsonPropertyName("...")]` to emulate a global policy is a **code-review-fail signal**: it is noise, it drifts, and it silently shadows the policy. If a whole DTO's attributes merely restate what the policy would produce, delete them and rely on the policy.
|
||||
- Enum string values use a `JsonStringEnumConverter`; keep its naming policy consistent with the property policy.
|
||||
- Grounding: Microsoft's System.Text.Json docs recommend the global `PropertyNamingPolicy` for project-wide conventions and reserve `[JsonPropertyName]` for exact-string overrides (it takes highest precedence and overrides the policy).
|
||||
|
||||
## Error handling
|
||||
|
||||
> General principle (cross-language): see `coderule.mdc` → "Error handling". This section is the .NET realization (the three-tier model, central handler, opaque-500, and status mapping all originate there).
|
||||
|
||||
This project uses a **business-exception model with one central handler** — *not* `Result<T,E>` and *not* per-method `try/catch`. Three failure tiers, three treatments:
|
||||
|
||||
1. **Input validation** — handled by the **auto-validation action filter, never by throwing.** FluentValidation auto-validation (see Validation) short-circuits the request before the action runs and returns the `400` (slug-keyed `problem+json` via the custom result factory). Do **not** raise a `ValidationException` for request-shape validation.
|
||||
2. **Business-rule violations** (expected, part of the API contract: not-found, conflict, invariant violation, forbidden-by-rule) — the service **throws a `BusinessException` subtype**. Services express failure by throwing; they do **not** return error-wrapper values and do **not** catch their own business exceptions.
|
||||
3. **Unexpected failures** (bugs — NRE, invariant breaks; infrastructure — DB unreachable, network) — thrown by the framework/runtime and left to **propagate** to the central handler.
|
||||
|
||||
### Business exception hierarchy
|
||||
|
||||
- A single abstract base — `abstract class BusinessException : Exception` — carries the HTTP mapping data: an `int Status` and a stable `string Slug` (and optional extension members). Every expected, contract-level failure is a concrete subtype that fixes its own status; **there is no single blanket business status code**:
|
||||
- not-found → `404`
|
||||
- state conflict (duplicate key, concurrent edit, illegal state transition) → `409`
|
||||
- well-formed request that violates a business invariant → `422`
|
||||
- forbidden by a business rule (not auth-scheme denial) → `403`
|
||||
- The `Slug`/`Status`/title **must reuse the existing `FleetViewerProblems` slug catalog** (`Common/Problems/`) so the `application/problem+json` wire contract (`type` URI, `title`, `status`, any `code` extension) stays byte-identical to what blackbox tests pin. The catalog stays the single source of truth for the error contract; the exception types reference it.
|
||||
- Choose `422` vs `409` by meaning, never interchangeably: `422` = the request is well-formed but the business invariant rejects it; `409` = it conflicts with the resource's current state.
|
||||
|
||||
### Central handler (catch in exactly one place)
|
||||
|
||||
- Register **one** `IExceptionHandler` via `builder.Services.AddExceptionHandler<...>()` + `AddProblemDetails()` + `app.UseExceptionHandler()`. It maps:
|
||||
- `BusinessException` → `ProblemDetails` built from its `Status` + `FleetViewerProblems.TypePrefix + Slug` (+ extensions). **Do NOT log these as errors** — they are expected 4xx contract outcomes; at most a `Debug`/`Information` line. Logging them at `Error` pollutes the error rate and pages on-call for normal client mistakes.
|
||||
- **everything else (unexpected)** → `500` `ProblemDetails` with a **fixed, opaque production body** — `title: "Unexpected error"`, `detail: "An unexpected error occurred. Our team has been notified."` — and **log the full exception to Serilog at `Error`** (`logger.LogError(ex, ...)`) with the correlation id, so the log entry correlates to the client's response. The body must **never** carry the exception message, stack trace, or any internal detail (information-disclosure risk). In `Development` only, it is acceptable to surface `ex.Message`/stack in the body to aid debugging — gate that on `IHostEnvironment.IsDevelopment()`.
|
||||
- **No per-method `try/catch` for error mapping.** A handler/controller does not catch business exceptions to turn them into responses — that is the central handler's only job. Legitimate local `catch` blocks remain only for: converting a third-party/framework exception into a `BusinessException` at a boundary, honoring `OperationCanceledException`, or keeping a background loop alive (catch-log-continue). Never an empty/silent catch (see `coderule.mdc`).
|
||||
- **Do not throw on hot per-item paths** (e.g. ingest per-record processing): exceptions are for request-level outcomes, not inner loops — return/skip with a counted metric there.
|
||||
- API error responses are always `ProblemDetails` (RFC 7807) with a stable slug `type` when the failure is part of the contract.
|
||||
|
||||
## HttpClient
|
||||
|
||||
- **Never `new HttpClient()` per request** (sockets enter `TIME_WAIT` for ~240s; you exhaust the ephemeral port range under load).
|
||||
- **Never use a naive `static HttpClient`** either (handlers don't rotate, DNS changes are missed).
|
||||
- Register via `IHttpClientFactory` — typed or named clients:
|
||||
```csharp
|
||||
builder.Services.AddHttpClient<MyApiClient>(c => c.BaseAddress = new Uri("https://api.example.com"));
|
||||
```
|
||||
- **Don't capture a typed `HttpClient` in a singleton.** Typed clients are Transient; capturing one in a singleton defeats handler rotation. Inject `IHttpClientFactory` into the singleton and call `CreateClient(name)` per operation, **or** configure `SocketsHttpHandler.PooledConnectionLifetime` so DNS refreshes at the socket level instead of the factory level.
|
||||
|
||||
## Modern C# / nullable reference types
|
||||
|
||||
- Enable nullable reference types (`<Nullable>enable</Nullable>`) on every new project.
|
||||
- **Don't paper over NRT warnings with `!`** (null-forgiving operator). Prefer:
|
||||
- `required` members (C# 11) for properties the caller must initialize via object initializer.
|
||||
- Constructor parameters for invariants established at construction.
|
||||
- `[NotNullWhen(true)]` / `[NotNull]` / `[MaybeNull]` attributes for `Try*` patterns.
|
||||
- Use `ArgumentNullException.ThrowIfNull(x)` at the top of any public method taking a reference-type argument. NRTs are design-time only; library entry points still need runtime guards.
|
||||
|
||||
## Static classes and static members
|
||||
|
||||
> General principle (cross-language): see `coderule.mdc` → "Static members (functions / classes)". Below is the .NET realization plus framework-specific exemptions.
|
||||
|
||||
Default to **instance classes behind an interface, registered in DI and constructor-injected.** That is what makes a unit testable (mockable), swappable, and free of hidden global state. `static` is the exception, not the default — reach for it only when the alternative below clearly applies.
|
||||
|
||||
**No business logic in a static method — ever.** `static` is for *mechanics* (convert, parse, compute, compare), never for *decisions* (what the system should do, which rule applies, what happens next). Domain logic lives in a service.
|
||||
|
||||
- **`static` is appropriate ONLY for:**
|
||||
- **Pure, stateless, and SIMPLE functions** — output depends solely on the arguments; no I/O, no clock, no `Random`/`Guid.NewGuid`, no DB/file/network, no mutable shared state; **and** the body is short and obvious (math, encoding/decoding, parsing, formatting, a small predicate). Simplicity — not purity alone — is the bar: the moment a would-be helper carries domain decisions, branches across many cases, or is complex enough to deserve its own unit-test suite, it stops being a "helper." Make it an **instance service behind an interface** so it is injectable, mockable by its collaborators, and discoverable. A complicated *pure* function still belongs in a service.
|
||||
- **Extension methods** over framework or domain types, when the body is pure and simple (e.g. claim/identity readers, enum⇄wire mappers).
|
||||
- **Constants / well-known values** (a `static class` holding `const`s).
|
||||
- **Static factory methods** on a type (private ctor + `public static Create(...)` returning a fully-formed instance) — an accepted construction pattern, distinct from a static *service*.
|
||||
- **Never use `static` for:**
|
||||
- **Business / domain logic of any kind**, even if currently it looks "pure." Decisions belong in a tested, injectable service.
|
||||
- A helper that touches I/O, configuration, time, randomness, or any external system — that is a *service*. Define an interface, make it an instance class, inject it. A static method that reaches a DB/clock/file cannot be mocked and forces brittle integration-style tests.
|
||||
- **Mutable static fields of any kind.** Global mutable state is a thread-safety and test-isolation hazard. A cache or in-memory state store belongs in a DI **singleton behind an interface**, never a `static Dictionary`.
|
||||
- Avoiding `new`/DI "ceremony." DI registration is one line and buys testability; saving it is never a reason to go static.
|
||||
- **Controllers are instance classes (constructor DI), not static.** A controller is `[ApiController] public sealed class XxxController(IXxxService svc) : ControllerBase { ... }` — dependencies are constructor-injected, actions are thin, and the type is never `static`. This is the standard for all new HTTP code (see "Solution layout & layering").
|
||||
- **Transitional exemption — legacy Minimal-API endpoint classes.** Existing `internal static class XxxEndpoints` exposing `MapXxxEndpoints(this RouteGroupBuilder group)` + `static` handler methods are the idiomatic *Minimal-API* pattern (no static state; deps are per-request method parameters; testable via `WebApplicationFactory`) and are **not** a static-class violation **while they exist**. Where the codebase has chosen controllers, migrate them and do **not** add new ones; until migrated, keep handler bodies thin with logic in injected services.
|
||||
- The static-OK rule also covers framework callback types that the runtime instantiates or invokes by convention — `AuthenticationHandler<TOptions>`, middleware `InvokeAsync`, `CookieAuthenticationEvents`, route predicates. They legitimately receive `HttpContext`/framework primitives and are not "static-class" or "HttpContext-discipline" violations.
|
||||
- **Library-mandated process-global statics are an accepted exception.** Some libraries are *designed* around a process-global, thread-safe static registry — e.g. a metrics library's `static readonly` counter/gauge collectors, or a `static` logger handle. Those `static readonly` fields are not the "mutable static state" this rule bans; do not force them behind a bespoke interface. A stateless utility over the system CSPRNG is likewise acceptable as `static` (folding it behind an interface for consistency with sibling generators is a fine choice, not a requirement).
|
||||
|
||||
## Data access (EF Core)
|
||||
|
||||
> General principle (cross-language): see `coderule.mdc` → "Data access" (single ORM path, justify raw SQL, prevent N+1). Below is the EF Core realization.
|
||||
|
||||
- **Use the project ORM (EF Core for this repo) as the ONLY data-access path for application reads/writes.** Raw SQL via `CommandText`, `FromSqlRaw`, `FromSqlInterpolated`, `ExecuteSqlRaw`, `ExecuteSqlInterpolated`, or `NpgsqlCommand`/`NpgsqlConnection.CreateCommand()` is **forbidden by default** in endpoint, service, and repository code. Reaching for raw SQL because "it's simpler" or "EF generates ugly SQL" is not a valid reason — write the LINQ query, profile if you must, and only then justify a workaround.
|
||||
- Narrow exceptions (each requires a 1-line comment in the code naming the EF limitation being worked around):
|
||||
- **DDL the ORM cannot express** — `CREATE EXTENSION`, vendor enum-cast DEFAULT (`HasDefaultValueSql("'active'::device_state")`). Confine to migrations or to one-shot `IHostedService.StartAsync` bootstrap hooks.
|
||||
- **Vendor-specific operators / functions** (e.g., TimescaleDB `time_bucket`, `make_interval(secs => ...)`, hypertable functions, PostGIS `ST_*`). Wrap each operator in a single repository method behind an interface; nowhere else in the codebase touches raw SQL for that operator. Prefer EF Core function mapping (`HasDbFunction` + `[DbFunction]`) before falling back to `FromSqlInterpolated`.
|
||||
- **Benchmarked hot path** where EF demonstrably generates a worse plan than hand-rolled SQL. Requires a `BenchmarkDotNet` file checked in next to the workaround proving the gap. "We think it's faster" is not a benchmark.
|
||||
- Prevent N+1 with `.Include()` / projection / explicit `.Select()`. New raw-SQL sites that do not fit one of the three exceptions MUST be flagged in code review as **High** severity (Maintainability / Architecture). Reviewers reject the PR until the SQL is either replaced with LINQ or moved behind a justified repository method with the required comment.
|
||||
- **`AsNoTracking()` on every read-only query.** The change tracker costs ~50% more memory and 2.9–5.2× more time on typical reads; you pay it for nothing on `GET` endpoints, reports, lookups. For read-heavy services, set `QueryTrackingBehavior.NoTracking` as the DbContext default and opt **in** to tracking with `.AsTracking()` on update paths.
|
||||
|
||||
## ASP.NET Core handler discipline (controllers)
|
||||
|
||||
> General principle (cross-language): see `coderule.mdc` → "Boundary discipline" (don't leak request/response context into business logic; authorize once at the boundary). Below is the ASP.NET Core realization.
|
||||
|
||||
These rules keep controller actions and services free of framework primitives that hide dependencies, defeat unit testing, and bypass the auth/binding pipelines the framework already gives you. (They also apply to the legacy Minimal-API handlers still being migrated.)
|
||||
|
||||
### `HttpContext` discipline
|
||||
|
||||
- **Do not pass `HttpContext`, `HttpRequest`, `HttpResponse`, or `IHttpContextAccessor` into services or repositories.** Extract the values you need (headers, route values, body, `ClaimsPrincipal`) inside the handler and pass them down as typed parameters.
|
||||
- Take `HttpContext` (or `HttpRequest`/`HttpResponse`) as a handler parameter **only** when no binding source can express the requirement. Concrete examples that justify it:
|
||||
- Custom body framing or streaming (you read `Request.Body`/`BodyReader` yourself).
|
||||
- Multiple discriminated payload shapes on one URL that cannot be one DTO.
|
||||
- Pre-allocation size caps that must reject **before** the body materializes into objects.
|
||||
- Writing a custom response envelope that doesn't fit `Results.*`/`TypedResults.*`.
|
||||
Document the reason with a `//` comment on the parameter or above the method.
|
||||
- Prefer **separate endpoints/methods** over discriminated payload shapes on one URL. Only fuse them when splitting would duplicate the majority of the validation logic — otherwise you trade testability for one fewer route registration, which is rarely worth it.
|
||||
- Default to specific binding sources: `[FromBody]`, `[FromQuery]`, `[FromHeader]`, `[FromRoute]`, `[FromServices]`, `ClaimsPrincipal user`, `CancellationToken cancellationToken`. Each of those is documented, testable, and integrates with OpenAPI.
|
||||
|
||||
### JSON deserialization
|
||||
|
||||
- **Default to `[FromBody]` + a typed `record`/DTO.** The framework calls `JsonSerializer.DeserializeAsync` for you, validates `Content-Type`, surfaces `BadHttpRequestException` on malformed input, and produces OpenAPI metadata.
|
||||
- Direct `JsonDocument` / `Utf8JsonReader` parsing of `Request.Body` is allowed **only** when typed deserialization cannot express the required validation. Allowed reasons:
|
||||
- **Typed slug-keyed error envelopes** that the standard binder cannot produce (e.g., per-field problem+json with a stable `type` URI).
|
||||
- **Pre-allocation size caps** that must reject `batch-too-large` before the array materializes.
|
||||
- **Shape discrimination at parse time** when the alternative is a single fat DTO + runtime branching.
|
||||
Each site needs a one-line comment naming which exception applies.
|
||||
- Reading raw `Request.Body` for plain typed JSON content is a code-review-fail signal in the absence of one of the named exceptions.
|
||||
|
||||
### Custom authentication schemes
|
||||
|
||||
- Custom bearer/token/API-key schemes go through **`AuthenticationHandler<TOptions>`** registered via `AddAuthentication().AddScheme<TOptions, THandler>(name, …)`. Apply `.RequireAuthorization(new AuthorizeAttribute { AuthenticationSchemes = name })` or `[Authorize(AuthenticationSchemes = name)]` on the endpoint.
|
||||
- **Do not read `Authorization` / cookie / API-key headers manually inside a handler that is `.AllowAnonymous()`.** That bypasses the auth pipeline, makes the auth logic unreusable for any second endpoint, and forces tests to reach the logic via reflection.
|
||||
- If you need a custom 401/403 body envelope (e.g. typed `application/problem+json` with a slug), override `HandleChallengeAsync` / `HandleForbiddenAsync` in the scheme handler — not by bypassing the pipeline.
|
||||
- In the endpoint, take `ClaimsPrincipal user` as a parameter and read identity from claims (`user.FindFirstValue(...)`). The auth handler is responsible for putting the right claims on the principal.
|
||||
|
||||
### Authorization (declare-once at the boundary)
|
||||
|
||||
- Authorize at the **boundary, once** — not per action. In MVC, put `[Authorize(Policy = "...")]` on the **controller class** (or a shared base controller); every action inherits it. Override on a single action with a narrower `[Authorize(Policy = ...)]` / `[AllowAnonymous]` only where it genuinely differs.
|
||||
- The Minimal-API equivalent is `group.MapGroup("/...").RequireAuthorization(policy)` on the **route group**. Both compile to the **same authorization metadata** — the group-level fluent call and the class-level attribute are equally correct and equally DRY. Per-method attributes / per-endpoint `RequireAuthorization` are for intentional per-route overrides only.
|
||||
- Name policies centrally (a single constants holder) and reference the constant — never inline role strings at the call site.
|
||||
|
||||
### Current-user / identity access
|
||||
|
||||
- **Inject `ClaimsPrincipal` directly into handlers for current-user identity; read it through the shared `ClaimsPrincipalExtensions` (`GetUserId()`, `GetSessionId()`, `GetDeviceId()`).** Do **not** wrap identity access in an `ICurrentUser` / `ICurrentUserProvider` service by default.
|
||||
- Why `ClaimsPrincipal` is the right seam here (not an over-coupling):
|
||||
- It is a **data-driven seam whose producer is the auth handler** — the cookie scheme, `DeviceBearerAuthenticationHandler`, or any future JWT all populate the *same* `ClaimsPrincipal`. The handler is already decoupled from *how* identity was obtained.
|
||||
- It is **available for free** in the HTTP layer — `ControllerBase.User` in a controller action (or a `ClaimsPrincipal user` parameter in a legacy Minimal-API handler), sourced from `HttpContext.User`; no `IHttpContextAccessor`, no scoped registration, no lifetime caveat. Identity stays in the `Api` layer: a controller reads `User`, extracts the IDs it needs via `ClaimsPrincipalExtensions`, and passes **plain values** (`Guid userId`) into the service — `ClaimsPrincipal` does not cross into the Services layer.
|
||||
- It is **testable without an interface**: `ClaimsPrincipal` is `new`-able with arbitrary claims and its behaviour (`IsInRole`, `FindFirst`, the extensions) is fully driven by those claims. Construct a real principal with test claims — preferable to a mocked `IPrincipal`, which can diverge from real claim-matching semantics. (In this repo, handlers are exercised over HTTP via `WebApplicationFactory` with a real login, so identity is never substituted anyway.)
|
||||
- The `ClaimsPrincipalExtensions` already provide the domain-friendly, centralized read surface that a provider's properties would duplicate.
|
||||
- A current-user provider adds a scoped `IHttpContextAccessor`-backed service — exactly the captive-dependency shape the DI section warns about — to replace a free, already-abstracted, already-testable binding. That fails the "simplicity is the highest priority" bar unless one of the concrete triggers below holds.
|
||||
- **Introduce an `ICurrentUser` abstraction ONLY when a named trigger appears:**
|
||||
1. **Identity is needed outside an HTTP request** — background job, message consumer, worker thread — where `ClaimsPrincipal` cannot be bound from the pipeline. A provider with swappable impls (HTTP-backed vs job-context) earns its keep.
|
||||
2. **The domain layer must consume identity** and you do not want `System.Security.Claims` types leaking into domain code — expose a domain-pure `ICurrentUser` value instead.
|
||||
3. **You need richer-than-claims current-user data** (a loaded `User` entity, tenant, permission set) resolved and cached per request.
|
||||
When introduced: back the HTTP implementation with `IHttpContextAccessor`, register it **Scoped**, never capture it in a singleton, and keep `ClaimsPrincipalExtensions` as the implementation detail it delegates to.
|
||||
|
||||
### Response shapes
|
||||
|
||||
**Controllers (the standard here): default to `ActionResult<T>`.** It mixes the success type `T` with `ActionResult` error shapes, participates in MVC's configured output formatters / content negotiation, and is the most reliable for OpenAPI:
|
||||
- Annotate with `[ProducesResponseType]`; the `Type` can be **omitted for the success code** (`[ProducesResponseType(StatusCodes.Status200OK)]`) — it is inferred from `T`. Add one attribute per additional status code (`404`, `409`, …).
|
||||
- Return the value directly (`return product;` — implicit cast to `200 OK`) or a `ControllerBase` helper for other shapes (`NotFound()`, `Conflict()`, `BadRequest(error)`, `CreatedAtAction(...)`).
|
||||
- The auto-validation action filter already produces the `400` for invalid input before the action runs (see Validation) — don't hand-write that path.
|
||||
- Keep the action **thin**: it maps the service's **success value** onto the success shape (`return product;` → `200`, `CreatedAtAction(...)` → `201`) and does not compute the business decision itself. **Expected failures are not mapped here** — the service throws a `BusinessException` subtype and the central `IExceptionHandler` produces the `ProblemDetails` (see Error handling). So a controller action has essentially no error branches: happy path in, success shape out.
|
||||
- `TypedResults` / `Results<T1, T2, …>` / `IResult` **are** usable in controllers, but they are the *Minimal-API* idiom and they **bypass MVC's configured output formatters / content negotiation** (they write the response directly — Microsoft Learn: "Does not leverage the configured Formatters"). Prefer `ActionResult<T>` in a controller; reach for `IResult` only for a deliberately format-agnostic raw response.
|
||||
|
||||
**Legacy Minimal-API endpoints (until migrated): default to `TypedResults.*`** over `Results.*`. `TypedResults` returns concrete types (`Ok<T>`, `NotFound`, `BadRequest<T>`) that carry OpenAPI metadata and are unit-testable without casting. For handlers that return more than one shape, declare the return type as `Results<T1, T2, ...>` — the compiler enforces every branch returns a declared type and the OpenAPI generator reads the union, so no `Produces`/`ProducesResponseType` attributes are needed:
|
||||
```csharp
|
||||
app.MapGet("/items/{id}", Results<Ok<Item>, NotFound> (int id) =>
|
||||
item is not null ? TypedResults.Ok(item) : TypedResults.NotFound());
|
||||
```
|
||||
Don't mix `Results.*` and `TypedResults.*` in the same handler — you lose the metadata.
|
||||
|
||||
### Service results vs. wire envelopes
|
||||
|
||||
> General principle (cross-language): see `coderule.mdc` → "Architecture & layering › Service results vs. transport envelopes". Below is the .NET realization.
|
||||
|
||||
- A service returns a **domain result** — a record of the values it computed (`IReadOnlyList<LiveDevice>`, a small snapshot record) on success, and **throws a `BusinessException` subtype** on an expected failure (see Error handling); it does not return error-wrapper values. The **controller maps the success value onto the wire DTO**. The response envelope (the `*Response` record, its field names, the HTTP status) is an **Api-layer concern**; the domain result is not, and ASP.NET / wire types must not appear in a service signature (see "Solution layout & layering").
|
||||
- **A value that the response echoes to the client but that the service ALSO used to compute the result is owned by the service** — it returns that value alongside the data; the controller must NOT independently re-derive it. Two clocks/sources for the same conceptual value is a latent bug.
|
||||
- Canonical case: a "server now" timestamp that a projection uses to decide freshness/staleness (which devices are dropped, what color each gets) **and** is echoed so the client renders relative ages consistently. If the controller stamped its own `DateTimeOffset.UtcNow`, it would diverge from the instant the service filtered against — a boundary bug.
|
||||
- Pattern: the service injects `TimeProvider`, captures the instant **once**, uses it, and returns it inside a domain result — e.g. `LiveSnapshot(DateTimeOffset CapturedAt, IReadOnlyList<LiveDevice> Devices)`. The controller returns `ActionResult<LiveStateResponse>`, mapping `CapturedAt → server_now`. The envelope name and JSON shape stay in the Api layer; the *instant* originates in the Services layer where it is consumed.
|
||||
- The opposite case: a value that is **purely an HTTP/transport artifact and is never consumed by domain logic** (a `Location` header, a per-response correlation id minted for tracing) is owned by the **Api layer** and the service never sees it.
|
||||
- Heuristic: ask "does the business logic *read* this value to make a decision?" If yes → it lives in the service and is returned. If it is only *formatting/transport* → it lives in the controller.
|
||||
|
||||
## Testing
|
||||
|
||||
> General principle (cross-language): see `coderule.mdc` → "Testing (real dependencies)" (real engine over fakes for query-correctness; share expensive fixtures). Below is the .NET realization.
|
||||
|
||||
- **xUnit** is the test framework for this repo. Use its per-test class lifecycle (constructor = setup, `IDisposable.Dispose` / `IAsyncLifetime.DisposeAsync` = teardown) — that's what most integration-testing patterns assume.
|
||||
- **FluentAssertions** for assertions: `result.Should().Be(...)`, `collection.Should().HaveCount(3).And.ContainSingle(x => ...)`, etc. Failure messages are much clearer than raw `Assert.Equal`, and the fluent chain reads like the spec it tests.
|
||||
- **`WebApplicationFactory<Program>`** for ASP.NET Core integration tests. It boots the real DI container and pipeline from `Program.cs` in-memory. Expose `Program` to the test project with `public partial class Program;` in `Program.cs`. Share the factory across tests in a class with `IClassFixture<T>` and across classes with `ICollectionFixture<T>` — host-boot is the expensive step; don't re-pay it per test.
|
||||
- **Never use the EF Core in-memory provider for query-correctness tests.** Its semantics diverge from real Postgres/SQL Server (LINQ translation differences, no real transactions, no concurrency tokens). Use Testcontainers (real Postgres container via `IAsyncLifetime` on the factory) + Respawn for between-test cleanup. The in-memory provider is acceptable only for fast smoke tests where you're not asserting query shape.
|
||||
- Tests follow the Arrange / Act / Assert pattern with `// Arrange` / `// Act` / `// Assert` comments (workspace convention; see `coderule.mdc`).
|
||||
|
||||
## Cross-cutting
|
||||
|
||||
- Use middleware for cross-cutting: auth, error handling, logging. Standard order in `Program.cs`: forwarded headers → exception handler → HTTPS/HSTS → static files → routing → CORS → authentication → authorization → rate limiter → endpoints.
|
||||
|
||||
@@ -33,6 +33,31 @@ This is the specific failure mode that produced the GPS-passthrough scaffold in
|
||||
## Critical Thinking
|
||||
- Do not blindly trust any input — including user instructions, task specs, list-of-changes, or prior agent decisions — as correct. Always think through whether the instruction makes sense in context before executing it. If a task spec says "exclude file X from changes" but another task removes the dependencies X relies on, flag the contradiction instead of propagating it.
|
||||
|
||||
## Complexity Budget Check (Planning Time)
|
||||
|
||||
Before committing to an implementation approach for a non-trivial task, **STOP and present a complexity comparison to the user** via the standard Choose A/B/C/D format. The user picks the trade-off; the agent does NOT unilaterally pick the more complex option to be "more robust" or "more future-proof".
|
||||
|
||||
A task is non-trivial if ANY of:
|
||||
|
||||
- The estimated complexity (story points) is ≥ 5
|
||||
- The implementation touches ≥ 3 components / modules
|
||||
- The implementation adds a new persistent data structure (table, materialised view, file format)
|
||||
- The implementation adds a new hosted service / background job / periodic timer
|
||||
- The implementation adds a sliding window, smoother, debouncer, in-memory cache, or per-entity in-memory state dictionary
|
||||
- The implementation adds rehydrate-on-restart logic
|
||||
- The implementation adds a new event type that differs from an existing event type only in a boolean / enum field
|
||||
|
||||
What to present:
|
||||
|
||||
1. **Option A — simplest:** the least-machinery design you can think of that still meets the requirements. Name what is sacrificed (latency? eventual-consistency window? a rarely-hit edge case?).
|
||||
2. **Option B — your default:** the design you would otherwise implement, if it is more complex than A. Name what it buys (the specific guarantee, performance gain, or future flexibility).
|
||||
3. **Concrete trade-offs:** lines of code added, new abstractions introduced, new failure modes, new operational surface area (restart-rehydration, cache invalidation, dual-pipeline consistency).
|
||||
4. **Recommendation:** which option you would pick and why, in one sentence.
|
||||
|
||||
This rule fires DURING planning — before code is written. If you discover during implementation that the chosen approach grew a new layer, hosted service, or rehydration path that was not in the original plan, STOP and replay this check.
|
||||
|
||||
Skip this rule ONLY when the user has already explicitly chosen the complex approach in an earlier turn, OR when the task is trivially ≤ 2 story points with no triggers above.
|
||||
|
||||
## Skill Discipline
|
||||
|
||||
Do exactly what the skill says. Nothing more.
|
||||
|
||||
@@ -5,40 +5,3 @@
|
||||
- When a task requires changes in another repository (e.g., admin API, flights, UI), **document** the required changes in the task's implementation notes or a dedicated cross-repo doc — do not implement them.
|
||||
- The mock API at `e2e/mocks/mock_api/` may be updated to reflect the expected contract of external services, but this is a test mock — not the real implementation.
|
||||
- If a task is entirely scoped to another repository, mark it as out-of-scope for this workspace and note the target repository.
|
||||
|
||||
## Exception — Adding Task Specs to Sibling Repos
|
||||
|
||||
The ONLY permitted form of writing into a sibling repository is **creating task-spec markdown files** (and updating the matching `_dependencies_table.md`) in that repo's `_docs/02_tasks/todo/` directory, and ONLY when the user explicitly asks for it in the current turn.
|
||||
|
||||
- "Explicit" means the user names the action (e.g. "add the md files to satellite-provider", "create the task spec there", "mirror it into their repo"). Inference from context is NOT enough — ask first.
|
||||
- Mirror the sibling repo's existing template (read ONE of their `done/` task files to learn the format — this is process documentation, not source code).
|
||||
- NEVER commit or push in the sibling repo unless the user separately and explicitly authorizes it. Default is "write to disk, leave for their review".
|
||||
- Update `_dependencies_table.md` to keep it consistent with the new task files.
|
||||
- The exception covers task specs ONLY. It does NOT extend to source code, CI/compose files, README, design docs, scripts, env templates, or any other file type in the sibling repo.
|
||||
- Each task-spec md must point back to the Jira ticket (which is the source of truth) and reference where the work was discovered (originating ticket in this repo).
|
||||
|
||||
## External Systems Are Black Boxes
|
||||
|
||||
External systems (sibling repos, third-party services, parent-suite services like `satellite-provider`) are treated as **black boxes** governed by their published **contract** (OpenAPI spec, contracts/*.md, public schemas, env-var docs).
|
||||
|
||||
- Treat the contract as the ONLY source of truth about an external system. The contract is what you may rely on; the implementation is what you may NOT rely on.
|
||||
- Do NOT investigate, grep, read, browse, or reason about an external system's internal source, internal directory layout, internal database schema, internal config files, persistent volumes, cache contents, log formats, deployment scripts, or any other implementation detail — even when the sibling repo is right there on disk and you could.
|
||||
- The ONE acceptable use of an external repo's source files is to READ ITS CONTRACT (e.g., `../satellite-provider/_docs/02_document/contracts/api/*.md`, an `openapi.yaml`, a `.proto`, a published schema). The contract may live in the sibling repo because that's where the producer documents it — that's fine. Anything OUTSIDE the contract directory is off-limits.
|
||||
- When the external system fails (returns errors, returns malformed data, is unreachable, contradicts its contract): STOP and report it to the user with the exact symptom (status code, error message, missing field, timeout). Do NOT diagnose why by reading the external system's internals. The producer team owns its own diagnosis. The signal is the symptom.
|
||||
- "It works" / "it doesn't work" is the only thing you may conclude about an external system. "It works this way because of X internal mechanism" is forbidden.
|
||||
|
||||
## Why
|
||||
|
||||
- Internals drift; contracts are stable. Reasoning that depends on internals breaks when the producer refactors.
|
||||
- Investigating internals trains the wrong mental model — agents start "fixing" cross-repo bugs by adapting consumer code to producer quirks instead of flagging the contract gap.
|
||||
- The producer team is the authority on its own system. Bypassing them creates two competing diagnoses and erodes the contract boundary.
|
||||
- Time spent reading external internals is time NOT spent on the actual scope.
|
||||
|
||||
## Concrete examples
|
||||
|
||||
- ✅ Reading `../satellite-provider/_docs/02_document/contracts/api/tile-inventory.md` to learn the inventory POST schema.
|
||||
- ❌ Reading `../satellite-provider/SatelliteProvider.Api/Program.cs` to learn what the inventory endpoint does internally.
|
||||
- ❌ Listing `../satellite-provider/tiles/` to see what tiles are cached.
|
||||
- ❌ Reading `../satellite-provider/.env` to figure out what env vars it expects (read the producer's published `.env.example` or contract doc instead).
|
||||
- ✅ Reporting "satellite-provider returns 500 when I POST a 1-tile inventory for (z=15, x=19308, y=11420)".
|
||||
- ❌ Reporting "satellite-provider returns 500 because its `TileService.GetInventoryAsync` throws when the Postgres `tiles` table is empty".
|
||||
|
||||
@@ -33,8 +33,8 @@ A first-time run executes Phase A then Phase B; every subsequent invocation re-e
|
||||
| 13 | Update Docs | document/SKILL.md (task mode) | Task Steps 0–5 |
|
||||
| 14 | Security Audit | security/SKILL.md | Phase 1–5 (optional) |
|
||||
| 15 | Performance Test | test-run/SKILL.md (perf mode) | Steps 1–5 (optional) |
|
||||
| 16 | Deploy | deploy/SKILL.md | Step 1–7 |
|
||||
| 16.5 | Release | release/SKILL.md | Phase 1–6 |
|
||||
| 16 | Deploy | deploy/SKILL.md | Step 1–7 (optional) |
|
||||
| 16.5 | Release | release/SKILL.md | Phase 1–6 (optional — only if Step 16 completed) |
|
||||
| 17 | Retrospective | retrospective/SKILL.md (cycle-end mode) | Steps 1–4 |
|
||||
|
||||
After Step 17, the feature cycle completes and the flow loops back to Step 9 with `state.cycle + 1` — see "Re-Entry After Completion" below.
|
||||
@@ -276,24 +276,32 @@ State-driven: reached by auto-chain from Step 14 (completed or skipped).
|
||||
Action: Apply the **Optional Skill Gate** (`protocols.md` → "Optional Skill Gate") with:
|
||||
- question: `Run performance/load tests before deploy?`
|
||||
- option-a-label: `Run performance tests (recommended for latency-sensitive or high-load systems)`
|
||||
- option-b-label: `Skip — proceed directly to deploy`
|
||||
- option-b-label: `Skip — proceed to deploy choice`
|
||||
- recommendation: `A or B — base on whether acceptance criteria include latency, throughput, or load requirements`
|
||||
- target-skill: `.cursor/skills/test-run/SKILL.md` in **perf mode** (the skill handles runner detection, threshold comparison, and its own A/B/C gate on threshold failures)
|
||||
- next-step: Step 16 (Deploy)
|
||||
|
||||
---
|
||||
|
||||
**Step 16 — Deploy**
|
||||
**Step 16 — Deploy (optional)**
|
||||
State-driven: reached by auto-chain from Step 15 (completed or skipped).
|
||||
|
||||
Action: Read and execute `.cursor/skills/deploy/SKILL.md`.
|
||||
Action: Apply the **Optional Skill Gate** (`protocols.md` → "Optional Skill Gate") with:
|
||||
- question: `Run deploy planning or refresh deploy artifacts for this cycle?`
|
||||
- option-a-label: `Run deploy — update scripts/procedures for this release`
|
||||
- option-b-label: `Skip — keep developing; deploy when ready for production`
|
||||
- recommendation: `B during active feature work; A when this cycle should ship`
|
||||
- target-skill: `.cursor/skills/deploy/SKILL.md`
|
||||
- next-step: Step 16.5 (Release) — only when Step 16 was completed; otherwise Step 17 (Retrospective)
|
||||
|
||||
After the deploy skill completes successfully, mark Step 16 as `completed` and auto-chain to Step 16.5 (Release).
|
||||
On **skip**: mark Step 16 and Step 16.5 as `skipped`; auto-chain to Step 17 (Retrospective in cycle-end mode).
|
||||
|
||||
On **complete**: mark Step 16 `completed` and auto-chain to Step 16.5 (Release).
|
||||
|
||||
---
|
||||
|
||||
**Step 16.5 — Release**
|
||||
State-driven: reached by auto-chain from Step 16, for the current `state.cycle`.
|
||||
**Step 16.5 — Release (optional)**
|
||||
State-driven: reached by auto-chain from Step 16 **only when Step 16 status is `completed`**, for the current `state.cycle`. If Step 16 was `skipped`, Step 16.5 is `skipped` and `/release` is not invoked.
|
||||
|
||||
Action: Read and execute `.cursor/skills/release/SKILL.md`. The release skill owns its own user interaction (Phase 1 pre-release gate, Phase 2 strategy select, Phase 6 escalation). Autodev does NOT add a wrapping A/B/C gate. Pass cycle context (`cycle: state.cycle`).
|
||||
|
||||
@@ -307,7 +315,7 @@ After the release skill exits, route on the verdict:
|
||||
---
|
||||
|
||||
**Step 17 — Retrospective**
|
||||
State-driven: reached by auto-chain from Step 16.5 with a `Released`, `Released-with-override`, or `Rolled-Back` verdict, for the current `state.cycle`.
|
||||
State-driven: reached by auto-chain from Step 16.5 (any verdict) OR from Step 16/16.5 both `skipped`, for the current `state.cycle`.
|
||||
|
||||
Action: Read and execute `.cursor/skills/retrospective/SKILL.md`. Mode selection:
|
||||
|
||||
@@ -318,15 +326,15 @@ Pass cycle context (`cycle: state.cycle`) so the retro report and LESSONS.md ent
|
||||
|
||||
After retrospective completes:
|
||||
|
||||
- If Step 16.5 verdict was `Released` or `Released-with-override` → mark Step 17 as `completed` and enter "Re-Entry After Completion" evaluation (loop back to Step 9 for cycle N+1).
|
||||
- If Step 16.5 verdict was `Released` or `Released-with-override`, OR Step 16.5 was `skipped` → mark Step 17 as `completed` and enter "Re-Entry After Completion" evaluation (loop back to Step 9 for cycle N+1).
|
||||
- If Step 16.5 verdict was `Rolled-Back` → mark Step 17 as `completed` but do NOT loop back. Surface the incident retro path and STOP.
|
||||
|
||||
---
|
||||
|
||||
**Re-Entry After Completion**
|
||||
State-driven: `state.step == done` OR Step 17 (Retrospective) is completed for `state.cycle` AND Step 16.5 verdict was `Released` or `Released-with-override`. A `Rolled-Back` cycle does NOT trigger Re-Entry — the user must explicitly invoke `/autodev` again.
|
||||
State-driven: `state.step == done` OR Step 17 (Retrospective) is completed for `state.cycle` AND (Step 16.5 verdict was `Released` or `Released-with-override` OR Step 16.5 was `skipped`). A `Rolled-Back` cycle does NOT trigger Re-Entry — the user must explicitly invoke `/autodev` again.
|
||||
|
||||
Action: The project completed a full cycle. Before incrementing the cycle counter, run the **Previous-Cycle Retro Existence Gate** below. If the gate passes (or its `state.cycle == 1` early exit applies), print the status banner and automatically loop back to New Task — do NOT ask the user for confirmation:
|
||||
Action: The project completed a full cycle. Print the status banner and automatically loop back to New Task — do NOT ask the user for confirmation:
|
||||
|
||||
```
|
||||
══════════════════════════════════════
|
||||
@@ -339,59 +347,7 @@ Action: The project completed a full cycle. Before incrementing the cycle counte
|
||||
|
||||
Set `step: 9`, `status: not_started`, and **increment `cycle`** (`cycle: state.cycle + 1`) in the state file, then auto-chain to Step 9 (New Task). Reset `sub_step` to `phase: 0, name: awaiting-invocation, detail: ""` and `retry_count: 0`.
|
||||
|
||||
Note: the loop (Steps 9 → 17 → 9) ensures every feature cycle includes: New Task → Implement → Run Tests → Test-Spec Sync → Update Docs → Security → Performance → Deploy → Release → Retrospective. The cycle only completes (and loops back to Step 9) on a `Released` or `Released-with-override` verdict; rolled-back or aborted releases stop the cycle.
|
||||
|
||||
---
|
||||
|
||||
**Previous-Cycle Retro Existence Gate** (AZ-900, codifies LESSONS 2026-05-26 [process])
|
||||
|
||||
Trigger: run this gate at the start of Re-Entry After Completion, BEFORE the `cycle: state.cycle + 1` increment in the state file.
|
||||
|
||||
Early-exit: if `state.cycle == 1`, the gate is **skipped** — cycle 1 has no previous cycle whose retro could exist. (A greenfield → existing-code transition on first entry to Phase B falls in this branch.)
|
||||
|
||||
Otherwise (`state.cycle >= 2`):
|
||||
|
||||
1. **Compute the date range for the cycle just completing.**
|
||||
- `cycle_start = ` modification date of the latest `_docs/03_implementation/implementation_report_*_cycle{state.cycle-1}.md` file. If no implementation report exists for the previous cycle (e.g. cycle was rolled back at Step 16.5), use the modification date of the latest `_docs/06_metrics/retro_*.md` file as a lower bound, or fall back to "yesterday" if neither exists.
|
||||
- `cycle_end = ` today (the date at which the gate runs).
|
||||
2. **Glob for the retro file**: `_docs/06_metrics/retro_*.md`, parse the `YYYY-MM-DD` portion of each filename, and check whether **any** file's date lies in the inclusive range `[cycle_start, cycle_end]`.
|
||||
3. **If at least one retro file is in range** → gate PASSES → continue with the cycle increment.
|
||||
4. **If no retro file is in range** → gate BLOCKS → play the notification sound per `.cursor/rules/human-attention-sound.mdc` and present the Choose block below.
|
||||
|
||||
```
|
||||
══════════════════════════════════════
|
||||
RETRO MISSING for cycle <state.cycle>
|
||||
══════════════════════════════════════
|
||||
No `_docs/06_metrics/retro_*.md` file dated
|
||||
within [<cycle_start>, <cycle_end>] was found.
|
||||
Per LESSONS 2026-05-26 [process], the cycle
|
||||
must close with a retro before cycle <state.cycle+1>
|
||||
can start.
|
||||
══════════════════════════════════════
|
||||
A) Author the missing retro now (invoke
|
||||
.cursor/skills/retrospective/SKILL.md in
|
||||
cycle-end mode against cycle <state.cycle>,
|
||||
then re-run this gate)
|
||||
B) Stub a backfilled retro and proceed (file
|
||||
a leftover entry under
|
||||
_docs/_process_leftovers/<YYYY-MM-DD>_retro_backfill_cycle<N>.md
|
||||
naming what data is missing; create
|
||||
_docs/06_metrics/retro_<today>_backfill_cycle<N>.md
|
||||
with the available data; then continue
|
||||
the cycle increment)
|
||||
C) Abort and ask the user
|
||||
══════════════════════════════════════
|
||||
Recommendation: A — a real retro keeps
|
||||
LESSONS.md honest; B is a last resort when
|
||||
cycle data is genuinely unrecoverable.
|
||||
══════════════════════════════════════
|
||||
```
|
||||
|
||||
- **On A** → invoke `.cursor/skills/retrospective/SKILL.md` in cycle-end mode with `cycle: state.cycle`. When it completes successfully, re-run the gate from step 1; on PASS, continue. If retrospective itself fails, follow standard Failure Handling (`protocols.md`).
|
||||
- **On B** → create the leftover entry and stub retro as documented in the option label, then continue with the cycle increment. Surface this in the Status Summary footer of the next session via the leftovers folder.
|
||||
- **On C** → STOP. Do not increment `cycle`. Leave `state.step == done` so the user re-invokes `/autodev` after writing the retro by hand.
|
||||
|
||||
Gate scope: this gate fires ONLY in `existing-code` flow. `greenfield` has no cycle counter (single Done step). `meta-repo` has no cycle counter (its cadence is `monorepo-status` re-runs, not feature cycles).
|
||||
Note: the loop (Steps 9 → 17 → 9) covers: New Task → Implement → Run Tests → Test-Spec Sync → Update Docs → Security → Performance → Deploy (optional) → Release (optional) → Retrospective. The cycle completes (and loops back to Step 9) on a `Released` or `Released-with-override` verdict, or when deploy/release were skipped; rolled-back or aborted releases stop the cycle.
|
||||
|
||||
## Auto-Chain Rules
|
||||
|
||||
@@ -418,13 +374,14 @@ Gate scope: this gate fires ONLY in `existing-code` flow. `greenfield` has no cy
|
||||
| Test-Spec Sync (12, done or skipped) | Auto-chain → Update Docs (13) |
|
||||
| Update Docs (13) | Auto-chain → Security Audit choice (14) |
|
||||
| Security Audit (14, done or skipped) | Auto-chain → Performance Test choice (15) |
|
||||
| Performance Test (15, done or skipped) | Auto-chain → Deploy (16) |
|
||||
| Deploy (16) | Auto-chain → Release (16.5) |
|
||||
| Performance Test (15, done or skipped) | Auto-chain → Deploy choice (16) |
|
||||
| Deploy (16, completed) | Auto-chain → Release (16.5) |
|
||||
| Deploy (16, skipped) | Mark 16.5 `skipped` → auto-chain → Retrospective (17, cycle-end mode) |
|
||||
| Release (16.5, verdict Released) | Auto-chain → Retrospective (17, cycle-end mode) |
|
||||
| Release (16.5, verdict Released-with-override) | Auto-chain → Retrospective (17, **incident mode**) |
|
||||
| Release (16.5, verdict Rolled-Back) | Auto-chain → Retrospective (17, **incident mode**); cycle does NOT loop back |
|
||||
| Release (16.5, verdict Aborted) | STOP — surface abort reason; do not auto-chain |
|
||||
| Retrospective (17, after Released / Released-with-override) | **Cycle complete** — loop back to New Task (9) with incremented cycle counter |
|
||||
| Retrospective (17, after Released / Released-with-override / deploy skipped) | **Cycle complete** — loop back to New Task (9) with incremented cycle counter |
|
||||
| Retrospective (17, after Rolled-Back) | Cycle remains incomplete — STOP and surface incident retro path |
|
||||
|
||||
## Status Summary — Step List
|
||||
@@ -464,7 +421,7 @@ Flow-specific slot values:
|
||||
| 16.5 | Release | `DONE (Released | Released-with-override | Rolled-Back | Aborted)` |
|
||||
| 17 | Retrospective | — |
|
||||
|
||||
All rows accept the shared state tokens (`DONE`, `IN PROGRESS`, `NOT STARTED`, `FAILED (retry N/3)`); rows 2, 4, 8, 12, 13, 14, 15 additionally accept `SKIPPED`.
|
||||
All rows accept the shared state tokens (`DONE`, `IN PROGRESS`, `NOT STARTED`, `FAILED (retry N/3)`); rows 2, 4, 8, 12, 13, 14, 15, 16, 16.5 additionally accept `SKIPPED`.
|
||||
|
||||
Row rendering format (renders with a phase separator between Step 8 and Step 9):
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Greenfield Workflow
|
||||
|
||||
Workflow for new projects built from scratch. Flows linearly: Problem → Research → Plan → UI Design (if applicable) → Test Spec → Decompose → Implement + Product Completeness Gate → Code Testability Revision → Decompose Tests → Implement Tests → Run Tests → Test-Spec Sync → Update Docs → Security Audit (optional) → Performance Test (optional) → Deploy → Release → Retrospective.
|
||||
Workflow for new projects built from scratch. Flows linearly: Problem → Research → Plan → UI Design (if applicable) → Test Spec → Decompose → Implement + Product Completeness Gate → Code Testability Revision → Decompose Tests → Implement Tests → Run Tests → Test-Spec Sync → Update Docs → Security Audit (optional) → Performance Test (optional) → Deploy (optional) → Release (optional, only if Deploy ran) → Retrospective.
|
||||
|
||||
## Step Reference Table
|
||||
|
||||
@@ -21,8 +21,8 @@ Workflow for new projects built from scratch. Flows linearly: Problem → Resear
|
||||
| 13 | Update Docs | document/SKILL.md (task mode) | Task Steps 0–5 |
|
||||
| 14 | Security Audit | security/SKILL.md | Phase 1–5 (optional) |
|
||||
| 15 | Performance Test | test-run/SKILL.md (perf mode) | Steps 1–5 (optional) |
|
||||
| 16 | Deploy | deploy/SKILL.md | Step 1–7 |
|
||||
| 16.5 | Release | release/SKILL.md | Phase 1–6 |
|
||||
| 16 | Deploy | deploy/SKILL.md | Step 1–7 (optional) |
|
||||
| 16.5 | Release | release/SKILL.md | Phase 1–6 (optional — only if Step 16 completed) |
|
||||
| 17 | Retrospective | retrospective/SKILL.md (cycle-end mode) | Steps 1–4 |
|
||||
|
||||
## Detection Rules
|
||||
@@ -280,17 +280,25 @@ Action: Apply the **Optional Skill Gate** (`protocols.md` → "Optional Skill Ga
|
||||
|
||||
---
|
||||
|
||||
**Step 16 — Deploy**
|
||||
**Step 16 — Deploy (optional)**
|
||||
State-driven: reached by auto-chain from Step 15 (after Step 15 is completed or skipped).
|
||||
|
||||
Action: Read and execute `.cursor/skills/deploy/SKILL.md`.
|
||||
Action: Apply the **Optional Skill Gate** (`protocols.md` → "Optional Skill Gate") with:
|
||||
- question: `Run deploy planning (scripts, procedures, compose overlays) now?`
|
||||
- option-a-label: `Run deploy — produce/update deploy artifacts and scripts`
|
||||
- option-b-label: `Skip — continue development; deploy when ready for production`
|
||||
- recommendation: `B when the product is not ready to ship; A when targeting a release soon`
|
||||
- target-skill: `.cursor/skills/deploy/SKILL.md`
|
||||
- next-step: Step 16.5 (Release) — only when Step 16 was completed; otherwise Step 17 (Retrospective)
|
||||
|
||||
After the deploy skill completes successfully, mark Step 16 as `completed` and auto-chain to Step 16.5 (Release).
|
||||
On **skip**: mark Step 16 and Step 16.5 as `skipped`; record in the release report (if one exists) or `_docs/_autodev_state.md` `sub_step.detail` that deploy/release were deferred; auto-chain to Step 17 (Retrospective in cycle-end mode).
|
||||
|
||||
On **complete**: mark Step 16 `completed` and auto-chain to Step 16.5 (Release).
|
||||
|
||||
---
|
||||
|
||||
**Step 16.5 — Release**
|
||||
State-driven: reached by auto-chain from Step 16.
|
||||
**Step 16.5 — Release (optional)**
|
||||
State-driven: reached by auto-chain from Step 16 **only when Step 16 status is `completed`**. If Step 16 was `skipped`, Step 16.5 is also `skipped` and the flow does not invoke `/release`.
|
||||
|
||||
Action: Read and execute `.cursor/skills/release/SKILL.md`. The release skill is responsible for selecting the target environment, executing the deploy artifacts, smoke-testing, watching the rollout, and producing a definitive verdict (`Released`, `Released-with-override`, `Rolled-Back`, or `Aborted`).
|
||||
|
||||
@@ -306,7 +314,7 @@ After the release skill exits:
|
||||
---
|
||||
|
||||
**Step 17 — Retrospective**
|
||||
State-driven: reached by auto-chain from Step 16.5 with a `Released` or `Released-with-override` verdict, OR from a `Rolled-Back` verdict (in incident mode).
|
||||
State-driven: reached by auto-chain from Step 16.5 (any verdict) OR from Step 16/16.5 both `skipped` (cycle-end mode — note deploy/release deferred in the retro report).
|
||||
|
||||
Action: Read and execute `.cursor/skills/retrospective/SKILL.md`. Mode selection:
|
||||
|
||||
@@ -320,7 +328,7 @@ After retrospective completes, mark Step 17 as `completed` and enter "Done" eval
|
||||
---
|
||||
|
||||
**Done**
|
||||
State-driven: reached by auto-chain from Step 17. (Sanity check: `_docs/04_deploy/` should contain all expected artifacts — containerization.md, ci_cd_pipeline.md, environment_strategy.md, observability.md, deployment_procedures.md, deploy_scripts.md. `_docs/04_release/` should contain at least one `release_<version>_<env>_<timestamp>.md` with a `Released` verdict — or the user has explicitly chosen to handle release outside autodev.)
|
||||
State-driven: reached by auto-chain from Step 17. (Sanity check: if Step 16 was `completed`, `_docs/04_deploy/` should contain the expected deploy artifacts. If Step 16.5 was `completed`, `_docs/04_release/` should contain a release report with a definitive verdict. Skipped deploy/release is valid — no release report required.)
|
||||
|
||||
Action: Report project completion with summary. Then **rewrite the state file** so the next `/autodev` invocation enters the feature-cycle loop in the existing-code flow:
|
||||
|
||||
@@ -358,8 +366,9 @@ On the next invocation, Flow Resolution rule 1 reads `flow: existing-code` and r
|
||||
| Test-Spec Sync (12, done or skipped) | Auto-chain → Update Docs (13) |
|
||||
| Update Docs (13, done or skipped) | Auto-chain → Security Audit choice (14) |
|
||||
| Security Audit (14, done or skipped) | Auto-chain → Performance Test choice (15) |
|
||||
| Performance Test (15, done or skipped) | Auto-chain → Deploy (16) |
|
||||
| Deploy (16) | Auto-chain → Release (16.5) |
|
||||
| Performance Test (15, done or skipped) | Auto-chain → Deploy choice (16) |
|
||||
| Deploy (16, completed) | Auto-chain → Release (16.5) |
|
||||
| Deploy (16, skipped) | Mark 16.5 `skipped` → auto-chain → Retrospective (17, cycle-end mode) |
|
||||
| Release (16.5, verdict Released) | Auto-chain → Retrospective (17, cycle-end mode) |
|
||||
| Release (16.5, verdict Released-with-override) | Auto-chain → Retrospective (17, **incident mode**) |
|
||||
| Release (16.5, verdict Rolled-Back) | Auto-chain → Retrospective (17, **incident mode**); do NOT enter Done |
|
||||
@@ -391,7 +400,7 @@ Flow name: `greenfield`. Render using the banner template in `protocols.md` →
|
||||
| 16.5 | Release | `DONE (Released | Released-with-override | Rolled-Back | Aborted)` |
|
||||
| 17 | Retrospective | — |
|
||||
|
||||
All rows also accept the shared state tokens (`DONE`, `IN PROGRESS`, `NOT STARTED`, `FAILED (retry N/3)`); rows 4, 12, 13, 14, 15 additionally accept `SKIPPED`.
|
||||
All rows also accept the shared state tokens (`DONE`, `IN PROGRESS`, `NOT STARTED`, `FAILED (retry N/3)`); rows 4, 12, 13, 14, 15, 16, 16.5 additionally accept `SKIPPED`.
|
||||
|
||||
Row rendering format (step-number column is right-padded to 2 characters for alignment):
|
||||
|
||||
|
||||
@@ -146,10 +146,6 @@ A **session boundary** is a transition that explicitly breaks auto-chain. Which
|
||||
|
||||
**Invariant**: a flow row without the `Session boundary` marker auto-chains unconditionally. Missing marker = missing boundary.
|
||||
|
||||
**Cross-reference — content gates that can also stop auto-chain.** Some flow files declare additional gates that block a transition even when the row would otherwise auto-chain. These are NOT session boundaries (they don't end the conversation); they BLOCK the next step until a content prerequisite is satisfied. The orchestrator must respect them in the same place it respects session boundaries — between completing one step and starting the next. Currently declared:
|
||||
|
||||
- `existing-code` Re-Entry After Completion → **Previous-Cycle Retro Existence Gate** (AZ-900) — blocks the `cycle: state.cycle + 1` increment if no `_docs/06_metrics/retro_*.md` file dated within the closing cycle's range exists. Skipped when `state.cycle == 1`. Presents an A/B/C choice (author now / stub-and-leftover / abort) when triggered. Full spec: `.cursor/skills/autodev/flows/existing-code.md` § "Previous-Cycle Retro Existence Gate".
|
||||
|
||||
### Orchestrator mechanism at a boundary
|
||||
|
||||
1. Update the state file: mark the current step `completed`; set the next step with `status: not_started`; reset `sub_step: {phase: 0, name: awaiting-invocation, detail: ""}`; keep `retry_count: 0`.
|
||||
|
||||
@@ -274,8 +274,8 @@ source repo
|
||||
|
||||
Two consequences for the architecture:
|
||||
|
||||
1. **C11 read contract adapted to the v1.0.0 inventory shape (AZ-777 Phase 1)** — `POST /api/satellite/tiles/inventory` + `GET /tiles/{z}/{x}/{y}` replace the historical `GET /api/satellite/tiles?bbox=…&zoom=…` shape. The bbox-driven `download_tiles_for_area` entry point and its DTOs are unchanged at the call-site level; the contract adaptation is internal to `HttpTileDownloader`. Auth is JWT Bearer (`SATELLITE_PROVIDER_API_KEY`) over TLS; `SATELLITE_PROVIDER_TLS_INSECURE=1` is a documented dev-only knob for self-signed certs.
|
||||
2. **Route-driven seeding (Epic AZ-835 — C11's third interface, `SatelliteProviderRouteClient`)** — the operator can now submit a tlog-derived `RouteSpec` (waypoints + region size; produced by `replay_input.tlog_route.extract_route_from_tlog` — AZ-836; canonical DTO at `_types/route.py` per AZ-845) via `POST /api/satellite/route` and have `satellite-provider` materialise just the corridor tiles, polling `GET /api/satellite/route/{id}` until `mapsReady=true`. This is ~100× more tile-efficient than the bbox path on long, narrow flights. Pre-emptive validation mirrors the AZ-809 `CreateRouteRequestValidator` bounds. The route-driven path is exercised today by the cycle-3 e2e fixture `operator_pre_flight_setup` (AZ-839) and the orchestrator test `test_az835_e2e_real_flight.py` (AZ-840); the C12 production CLI binding is a future-cycle integration.
|
||||
1. **C11 read contract adapted to the v1.0.0 inventory shape (AZ-777 Phase 1)** — `POST /api/satellite/tiles/inventory` + `GET /tiles/{z}/{x}/{y}` replace the historical `GET /api/satellite/tiles?bbox=…&zoom=…` shape. The bbox-driven `download_tiles_for_area` entry point and its DTOs are unchanged at the call-site level; the contract adaptation is internal to `HttpTileDownloader`. Auth is JWT Bearer (`SATELLITE_PROVIDER_API_KEY`) over TLS; `SATELLITE_PROVIDER_TLS_INSECURE=1` is a documented dev-only knob for self-signed certs. **Proposed successor (ADR-013 / AZ-976)**: gRPC `satellite.v1.RouteTileDelivery.DeliverRouteTiles` server-streaming with client tile catalog — see `tile_provision_grpc.md`; supersedes the never-shipped inventory REST endpoint.
|
||||
2. **Route-driven seeding (Epic AZ-835 / AZ-969)** — the operator submits a tlog-derived `RouteSpec` (produced by `replay_input.tlog_route.extract_route_from_tlog` — AZ-836) via C12 `seed-cache-from-tlog` (AZ-974) or the F11 `replay_api` demo job (AZ-973). E2E fixture `operator_pre_flight_setup` wraps the same production `operator_replay.cache_seed` module.
|
||||
|
||||
**Imagery source license attribution (cycle 3)**: the Jetson `satellite-provider` instance downloads from the **Google Maps satellite layer** (`lyrs=s`), governed by Google Maps Platform Terms of Service. Dev/research use only; production deployment requires either a Google Maps Platform licensing review or migration to a true CC-BY satellite source on the parent-suite side (parent-suite ticket TBD). Operator-side seed scripts (`tests/fixtures/derkachi_c6/seed_region.py`, `seed_route.py`) propagate the "Imagery © Google" attribution.
|
||||
|
||||
@@ -292,11 +292,17 @@ Cycle 4 rebuilt the replay-mode operator-input surface around a single canonical
|
||||
| **AZ-894** (CSV adapter) | New primary path | `csv_replay_input.CsvReplayInputAdapter` consumes a paired `(video, CSV)` where the CSV's `Time` column is the canonical clock for every IMU/GPS sample. Gated `BUILD_CSV_REPLAY_ADAPTER=ON` in airborne and research binaries; OFF in operator-orchestrator. |
|
||||
| **AZ-895** (auto-sync deprecation) | Removed legacy | `replay_input.auto_sync` (AZ-405) reduced to a no-op stub that raises on first call; `tlog_video_adapter.py` reduced to a deprecated stub whose `open()` raises immediately. The legacy `--time-offset-ms` / `--skip-auto-sync` / `--auto-trim` CLI flags accepted-with-warning, ignored. Hard removal tracked in AZ-908 (cycle 5+ backlog). |
|
||||
| **AZ-896** (CSV format spec) | Contract | `_docs/02_document/contracts/replay/csv_replay_format.md` documents the CSV row schema, the row-0-alignment-with-video-frame-0 invariant, and an example `data_imu.csv` shipped under the same path. |
|
||||
| **AZ-897** (operator UI) | Cycle-5+ follow-up | First operator-facing UI surface — a React + Tailwind single-page form that uploads a paired `(video, CSV)`, links to AZ-896's format docs + example CSV, and tails the verdict from the headless `gps-denied-replay` invocation. Not on cycle-4 critical path; flagged here so the CSV format stays UI-friendly. |
|
||||
| **AZ-897** (operator UI) | Cycle 5 — Epic AZ-969 | Dual-timeline `(video, tlog)` alignment UI in `../ui`; uploads raw tlog, calls `replay_api` preview/align/demo endpoints; displays map + verdict. Spec: `../ui/_docs/02_tasks/todo/AZ-897_operator_replay_sync_ui.md`. |
|
||||
|
||||
The architectural rationale is captured in **Invariant 14** of the replay protocol (`_docs/02_document/contracts/replay/replay_protocol.md`): the system runs as a single edge process on a single device; there must be exactly one wall/monotonic clock authoritative for timestamps that cross component boundaries. In live mode that clock is the C8 inbound `FcAdapter`'s FC-boot-relative timestamp; in replay mode (after cycle 4) it is the CSV row's `Time` column. The previous design's two-clock surface (Jetson monotonic at C1 VIO emission, FC-boot at C8 IMU window arrival) produced the AZ-848 regression and is retired with the auto-sync deprecation.
|
||||
|
||||
The legacy `TlogReplayFcAdapter` is retained for two audit-only paths — offline FDR analysis from `tools/` and a one-shot `gps-denied-tlog-to-csv` migration utility that exports legacy tlog inputs to the canonical CSV. Neither path runs from the airborne composition root after cycle 4.
|
||||
The legacy `TlogReplayFcAdapter` is retained for audit paths — offline FDR analysis and `gps-denied-tlog-to-csv` export (AZ-972). Runtime replay uses the CSV adapter after operator alignment (F11 / Epic AZ-969).
|
||||
|
||||
### Demo replay operator flow (cycle 5 — Epic AZ-969)
|
||||
|
||||
F11 in `system-flows.md` is the **primary product demo**, not an e2e-test concern. Raw operator inputs are `(video, tlog, calibration)`; alignment produces an AZ-896 CSV on a single canonical clock; route-driven cache seeding uses `extract_route_from_tlog` via C12 / `replay_api` production modules (AZ-974, AZ-973). Backend children: AZ-970 (preview API), AZ-971 (alignment refine), AZ-972 (CSV export), AZ-973 (orchestration), AZ-974 (C12 seed CLI), AZ-975 (docs). UI: AZ-897 in `../ui`.
|
||||
|
||||
The cycle-4 `(video, CSV)` upload bypass (AZ-959) remains for operators who already have an aligned CSV; it is not the default demo entry.
|
||||
|
||||
### `satellite-provider` upload contract (per D-PROJ-2 carryforward)
|
||||
|
||||
@@ -782,3 +788,31 @@ When C5 ships a second strategy — `eskf` (ESKF baseline, AZ-588) — the subst
|
||||
- `_docs/02_document/components/06_c4_pose/description.md` gains an "Enabled flag" sub-section that points at this ADR; the rest of the component contract is unchanged.
|
||||
- The unit-test surface at `tests/unit/runtime_root/test_az776_open_loop_eskf_composition.py` owns the seven invariants AZ-776 introduces: `C4PoseConfig.enabled` default-true, AC-1 (open-loop ESKF composes without C4), AC-2 (default GTSAM profile still includes C4), AC-3a + AC-3b (the two forbidden pairings raise `CompositionError`), and the two `pre_constructed` behaviours (`c5_isam2_graph_handle` omitted when C4 disabled, present when C4 enabled). The full suite passes in ~4 s.
|
||||
- The composition root's contract surface in `runtime_root/__init__.py` gains one public helper (`CompositionError` was already public; the new `skip_slugs` parameter to `_compose` is module-private). No public CLI flag is added — operators set `c4_pose.enabled = false` in YAML.
|
||||
|
||||
### ADR-013 — gRPC server-streaming tile provision for operator pre-flight (AZ-976)
|
||||
|
||||
**Context**: Operator-side cache build (C11/C12 ↔ `satellite-provider`) is off the hot airborne path but dominates time-to-ready when a corridor has thousands of tiles. The current REST shape (`POST /route` + poll + planned `POST /inventory` + N× `GET /tiles/{z}/{x}/{y}`) multiplies round-trips and cannot overlap "tiles already on SP disk" with "tiles still downloading from Google Maps". The inventory POST was specified in AZ-777 but never shipped in satellite-provider; Jetson smoke tests 404 on it today. Both codebases are owned by the same team (.NET satellite-provider, Python gps-denied operator tooling), so a typed streaming contract is feasible without a browser client.
|
||||
|
||||
**Decision**:
|
||||
|
||||
1. **We will add `satellite.v1.RouteTileDelivery.DeliverRouteTiles`** — unary request (`RouteSpec` + `client_tiles`), server-streaming `RouteTileEvent` (manifest → batches → progress → complete | error) — as the primary operator-side pre-flight transport (Epic AZ-976). Proto: `tile_provision.proto`; human contract: `tile_provision_grpc.md`.
|
||||
2. **The request carries `RouteSpec.route_id` (idempotent UUID) plus `ClientTileRecord[]`.** satellite-provider omits tiles when the client catalog already has equal-or-better resolution and equal-or-newer `captured_at` (lower m/px = better).
|
||||
3. **First stream event is `RouteManifest`** (`total_candidates`, `skipped_by_client`, `to_deliver`); then `TileBatch` messages with inline JPEGs. Server sends on-disk hits before externally fetched tiles (wire-agnostic ordering; `TilePayload.route_priority` hints along-route order).
|
||||
4. **ADR-004 boundary is preserved**: only C11/C12 on the operator workstation import gRPC stubs.
|
||||
|
||||
**Alternatives considered**:
|
||||
|
||||
| Alternative | Rejected because |
|
||||
|-------------|------------------|
|
||||
| REST `POST /inventory` + parallel GET | Never implemented in satellite-provider; still N+1 HTTP; no overlap of cached vs in-flight fetch |
|
||||
| SSE over HTTPS | Weaker typing; both sides are service binaries, not browsers — gRPC + protobuf is the better fit |
|
||||
| ZeroMQ between products | Poor fit across WAN/NAT; better kept **inside** satellite-provider's fetch workers |
|
||||
| In-flight streaming to UAV | Violates RESTRICT-SAT-1 / ADR-004; wrong reliability model for the aircraft |
|
||||
|
||||
**Consequences**:
|
||||
|
||||
- Epic AZ-976 decomposes: AZ-977 (SP gRPC server), AZ-978 (C11 client + C12 wiring), AZ-979 (Jetson benchmark + flip default).
|
||||
- REST `route_client` + `HttpTileDownloader` remain as fallback until AZ-979 benchmark promotes gRPC.
|
||||
- Finished C6 is still staged onto the Jetson via USB/rsync before flight — this ADR optimizes operator wait time, not in-air link dependency.
|
||||
|
||||
**Evidence**: `_docs/02_document/contracts/c11_tilemanager/tile_provision.proto`, `tile_provision_grpc.md`, `_docs/02_tasks/todo/AZ-976_grpc_tile_provision_epic.md`.
|
||||
@@ -0,0 +1,95 @@
|
||||
syntax = "proto3";
|
||||
|
||||
package satellite.v1;
|
||||
|
||||
import "google/protobuf/timestamp.proto";
|
||||
|
||||
option csharp_namespace = "Satellite.V1";
|
||||
|
||||
service RouteTileDelivery {
|
||||
rpc DeliverRouteTiles(DeliverRouteTilesRequest) returns (stream RouteTileEvent);
|
||||
}
|
||||
|
||||
message DeliverRouteTilesRequest {
|
||||
RouteSpec route = 1;
|
||||
repeated ClientTileRecord client_tiles = 2;
|
||||
}
|
||||
|
||||
message RouteSpec {
|
||||
string route_id = 1;
|
||||
repeated Waypoint waypoints = 2;
|
||||
double region_size_meters = 3;
|
||||
int32 zoom = 4;
|
||||
repeated GeofencePolygon geofences = 5;
|
||||
bool include_geofence_tiles = 6;
|
||||
}
|
||||
|
||||
message Waypoint {
|
||||
double lat = 1;
|
||||
double lon = 2;
|
||||
}
|
||||
|
||||
message GeofencePolygon {
|
||||
repeated Waypoint vertices = 1;
|
||||
}
|
||||
|
||||
message ClientTileRecord {
|
||||
int32 z = 1;
|
||||
int32 x = 2;
|
||||
int32 y = 3;
|
||||
double resolution_m_per_px = 4;
|
||||
google.protobuf.Timestamp captured_at = 5;
|
||||
optional string source = 6;
|
||||
bytes content_sha256 = 7;
|
||||
}
|
||||
|
||||
message RouteTileEvent {
|
||||
oneof payload {
|
||||
RouteManifest manifest = 1;
|
||||
TileBatch batch = 2;
|
||||
ProgressUpdate progress = 3;
|
||||
DeliveryComplete complete = 4;
|
||||
DeliveryError error = 5;
|
||||
}
|
||||
}
|
||||
|
||||
message RouteManifest {
|
||||
uint32 total_candidates = 1;
|
||||
uint32 skipped_by_client = 2;
|
||||
uint32 to_deliver = 3;
|
||||
}
|
||||
|
||||
message TileBatch {
|
||||
uint32 batch_seq = 1;
|
||||
repeated TilePayload tiles = 2;
|
||||
}
|
||||
|
||||
message TilePayload {
|
||||
int32 z = 1;
|
||||
int32 x = 2;
|
||||
int32 y = 3;
|
||||
double resolution_m_per_px = 4;
|
||||
google.protobuf.Timestamp captured_at = 5;
|
||||
string source = 6;
|
||||
bytes jpeg = 7;
|
||||
bytes content_sha256 = 8;
|
||||
uint32 route_priority = 9;
|
||||
}
|
||||
|
||||
message ProgressUpdate {
|
||||
uint32 delivered = 1;
|
||||
uint32 total = 2;
|
||||
uint32 downloading = 3;
|
||||
}
|
||||
|
||||
message DeliveryComplete {
|
||||
uint32 delivered = 1;
|
||||
uint32 skipped_client = 2;
|
||||
uint32 skipped_server_filter = 3;
|
||||
}
|
||||
|
||||
message DeliveryError {
|
||||
string code = 1;
|
||||
string message = 2;
|
||||
bool retryable = 3;
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
# Contract: RouteTileDelivery (gRPC)
|
||||
|
||||
**Component**: c11_tilemanager (consumer), satellite-provider (producer)
|
||||
**Epic**: AZ-976
|
||||
**ADR**: ADR-013 (architecture.md)
|
||||
**Proto**: `tile_provision.proto` — `package satellite.v1`
|
||||
**Version**: 0.3.0
|
||||
**Status**: proposed
|
||||
**Last Updated**: 2026-06-19
|
||||
|
||||
## Purpose
|
||||
|
||||
Operator-side **pre-flight cache provisioning**. Client sends route + onboard tile catalog once; server streams `RouteTileEvent` messages until `DeliveryComplete` or `DeliveryError`.
|
||||
|
||||
satellite-provider does **not** receive `flight_id` — that is a C6 bookkeeping concern on the gps-denied side only (`route_id` is the wire correlation id).
|
||||
|
||||
C11/C12 on the **operator workstation** only. ADR-004: airborne image must not import stubs or open this channel.
|
||||
|
||||
## RPC
|
||||
|
||||
```protobuf
|
||||
service RouteTileDelivery {
|
||||
rpc DeliverRouteTiles(DeliverRouteTilesRequest) returns (stream RouteTileEvent);
|
||||
}
|
||||
```
|
||||
|
||||
| Concern | Rule |
|
||||
|---------|------|
|
||||
| Auth | gRPC metadata `authorization: Bearer <JWT>` |
|
||||
| TLS | Required in production; `SATELLITE_PROVIDER_TLS_INSECURE=1` dev knob |
|
||||
| Idempotency | `RouteSpec.route_id` (UUID string) |
|
||||
| Resume | Client persists last acked `batch_seq` per `route_id` locally (not on wire) |
|
||||
|
||||
## Request
|
||||
|
||||
### `DeliverRouteTilesRequest`
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `route` | Corridor geometry + single zoom |
|
||||
| `client_tiles` | Onboard inventory snapshot (route intersection only) |
|
||||
|
||||
### `RouteSpec`
|
||||
|
||||
| Field | Maps from gps-denied |
|
||||
|-------|----------------------|
|
||||
| `route_id` | Client-generated UUID per provision job |
|
||||
| `waypoints` | `replay_input.tlog_route.RouteSpec.waypoints` |
|
||||
| `region_size_meters` | `RouteSpec.suggested_region_size_meters` |
|
||||
| `zoom` | Single slippy zoom level (confirmed sufficient) |
|
||||
| `geofences` | Optional inclusion polygons |
|
||||
| `include_geofence_tiles` | Union geofence tiles with corridor grid |
|
||||
|
||||
### `ClientTileRecord`
|
||||
|
||||
Canonical key: **`(z, x, y)`**. `source` is informational only — **not** used in skip logic.
|
||||
|
||||
| Field | C6 mapping |
|
||||
|-------|------------|
|
||||
| `resolution_m_per_px` | RESTRICT-SAT-4 (lower = better) |
|
||||
| `captured_at` | `TileMetadata.capture_timestamp` |
|
||||
| `content_sha256` | `TileMetadata.content_sha256_hex` (raw 32 bytes) |
|
||||
|
||||
## Server skip rule (client catalog)
|
||||
|
||||
For each server candidate tile, **omit from stream** when `client_tiles` has matching `(z,x,y)` and **any** of:
|
||||
|
||||
1. `client.content_sha256` is non-empty and **equals** server payload hash → skip (byte-identical)
|
||||
2. `client.resolution_m_per_px <= server.resolution_m_per_px` **and** `client.captured_at >= server.captured_at` → skip (metadata-sufficient)
|
||||
|
||||
`source` is **not** compared.
|
||||
|
||||
`RouteManifest.skipped_by_client` counts tiles removed by this rule.
|
||||
|
||||
## Sector — not on this wire
|
||||
|
||||
**Sector** (`active_conflict` vs `stable_rear`) controls **how stale a tile may be before C6 rejects it on write** (AC-NEW-6 freshness). It is an operator decision about the geographic area, not something satellite-provider needs to deliver tiles.
|
||||
|
||||
| Layer | Who applies sector |
|
||||
|-------|-------------------|
|
||||
| satellite-provider | Does not need sector — streams tiles by route geometry |
|
||||
| C11 client write | Reads sector from **C11/C12 config** (same as today) when calling C6 freshness gate |
|
||||
|
||||
No `SectorClass` field on the gRPC request.
|
||||
|
||||
## Response stream: `RouteTileEvent`
|
||||
|
||||
Typical sequence:
|
||||
|
||||
1. **`RouteManifest`** — `total_candidates`, `skipped_by_client`, `to_deliver`
|
||||
2. **`TileBatch`** — monotonic `batch_seq`; on-disk hits first, then freshly fetched
|
||||
3. **`ProgressUpdate`** — optional
|
||||
4. **`DeliveryComplete`** or **`DeliveryError`**
|
||||
|
||||
### `DeliveryComplete` counters
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| `delivered` | Tiles actually sent in `TileBatch` streams |
|
||||
| `skipped_client` | Same as manifest `skipped_by_client` (echo for client verify) |
|
||||
| `skipped_server_filter` | Tiles SP required but **did not send** after client dedup — see below |
|
||||
|
||||
#### `skipped_server_filter` — what counts
|
||||
|
||||
Tiles that entered the post-client-dedup work queue but never appeared in a batch:
|
||||
|
||||
| Reason | Example |
|
||||
|--------|---------|
|
||||
| **Fetch failed** | External imagery provider 404/timeout after retries |
|
||||
| **Below SP min resolution** | SP refuses to store/serve below its configured floor |
|
||||
| **Geometry clip** | Tile dropped after server-side corridor/geofence validation |
|
||||
| **Operational cap** | Job hit max-tiles / rate limit (if SP enforces) |
|
||||
|
||||
Tiles skipped by the **client catalog rule** are **not** included here (they are `skipped_client`).
|
||||
|
||||
If SP has no server-side filters in v1, `skipped_server_filter` may be **0**; the field is reserved for observability.
|
||||
|
||||
### `TilePayload`
|
||||
|
||||
| Field | Notes |
|
||||
|-------|-------|
|
||||
| `content_sha256` | 32-byte SHA-256 of `jpeg`; matches C6 DB invariant |
|
||||
| `route_priority` | Lower = earlier along route |
|
||||
|
||||
## Client write path (gps-denied)
|
||||
|
||||
`RouteTileDeliveryClient` (C11):
|
||||
|
||||
- Assigns C6 `flight_id` from operator context locally (not from SP)
|
||||
- Applies RESTRICT-SAT-4, **sector-based freshness**, AZ-308 budget, download journal
|
||||
- Resumes via persisted `route_id` + `batch_seq`
|
||||
|
||||
## Migration
|
||||
|
||||
REST `route_client` + `HttpTileDownloader` remain fallback until AZ-979 benchmark.
|
||||
|
||||
## Change log
|
||||
|
||||
| Version | Date | Change |
|
||||
|---------|------|--------|
|
||||
| 0.3.0 | 2026-06-19 | `ClientTileRecord.content_sha256`; sequential field nums on `TilePayload`; sector/flight_id off wire; skip rule + `skipped_server_filter` defined |
|
||||
| 0.2.0 | 2026-06-19 | `satellite.v1.RouteTileDelivery` + `RouteTileEvent` oneof |
|
||||
| 0.1.0 | 2026-06-19 | Initial draft (superseded) |
|
||||
@@ -289,7 +289,9 @@ The two **invalid** cells (`true` + `eskf` and `false` + `gtsam_isam2`) raise `C
|
||||
|
||||
**Sub-invariant 14.c (auto-sync deprecation — AZ-895)**: the `replay_input.auto_sync` module (AZ-405) is reduced to a deprecated no-op stub that raises `ReplayInputAdapterError("auto-sync removed; supply --imu CSV instead")` from every public entry point. The CLI flags `--time-offset-ms`, `--skip-auto-sync`, and `--auto-trim` are accepted with a deprecation warning and ignored. The justification: with a single canonical clock at the CSV row level (14.a), there is no second clock to align against — the operator authors the CSV with the correct row-0 alignment, and the fixture verifies row 0's `Time == 0`. Hard removal of the deprecated surface is tracked in AZ-908; this cycle ships only the stub + warnings to preserve source-compat for any downstream caller built against AZ-405's pre-deprecation shape.
|
||||
|
||||
**Sub-invariant 14.d (operator-facing UI — AZ-897, future cycle)**: the cycle-4 deliverable is the headless `gps-denied-replay --video X --imu Y` shape. An operator-facing web UI (single-page React + Tailwind form that uploads a paired `(video, CSV)` and tails the verdict) is tracked separately in AZ-897 and is NOT on the critical path of the CSV redesign; this sub-invariant exists only to record that the format spec (AZ-896) and the CSV adapter (AZ-894) MUST stay UI-friendly (CSV example, format docs link, clear error messages on row-0-misalignment) so AZ-897 lands without contract drift.
|
||||
**Sub-invariant 14.d (operator-facing UI — AZ-897, superseded by Invariant 15)**: retained for historical cycle-4 CSV-only upload spec. Default demo entry is now F11 / AZ-969.
|
||||
|
||||
15. **Operator demo replay path (cycle 5 — AZ-969 / F11)**: the default product demo accepts raw `(video, tlog, calibration)` from the suite UI. Alignment is operator-visible (dual timeline bars + explicit refine); the backend exports an AZ-896 CSV whose `Time` column is the single canonical replay clock (Invariant 14.a). Steps: preview timelines (AZ-970) → coarse align + refine (AZ-897, AZ-971) → export CSV (AZ-972) → seed corridor cache from tlog GPS (AZ-974) → run `gps-denied-replay` (AZ-973) → map + verdict. The `(video, pre-authored CSV)` bypass (AZ-959) is optional, not default. E2E tests MUST use the same orchestration modules as production — no parallel test-only graph. AZ-908 (hard removal of alignment stubs) is deferred until AZ-971 ships.
|
||||
|
||||
## Producer / Consumer Split
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@
|
||||
| F8 | Companion reboot recovery | Companion process restart while FC remains armed | C8 (FC IMU pose ingest), C5, C10 (warm-cache verify), C13 | Medium |
|
||||
| F9 | GCS telemetry stream | Per-frame estimate available + GCS link healthy | C5, C8, [[QGroundControl]] | Medium |
|
||||
| F10 | Post-landing tile upload | Operator triggers C12 `PostLandingUploadOrchestrator`; orchestrator confirms `flight_footer.clean_shutdown == True` and invokes C11 `TileUploader` | C12 `PostLandingUploadOrchestrator` (operator-side; reads FDR footer), C11 `TileUploader` (operator-side), C6 (read), [[`satellite-provider`]] (D-PROJ-2 endpoint, planned) | High |
|
||||
| F11 | Demo replay validation (operator) | Operator uploads `(video, tlog, calibration)` in suite UI; aligns timelines; runs full GPS-denied replay verdict | [[`suite/ui`]] (AZ-897), `replay_api` (AZ-973), `replay_input` (AZ-970–972), C12 `seed-cache-from-tlog` (AZ-974), C11 route seed, C10, airborne replay (`config.mode=replay`) | High |
|
||||
|
||||
## Flow Dependencies
|
||||
|
||||
@@ -34,6 +35,7 @@
|
||||
| F8 | F1 + F2 (warm cache survives reboot via content-hash verify) | F3 (resumes once warm), F5 (degraded mode if recovery fails) |
|
||||
| F9 | F3 | n/a (read-only outbound) |
|
||||
| F10 | F4 (locally-saved tiles), C13 `flight_footer` written on clean shutdown, parent-suite D-PROJ-2 endpoint availability | F1 of the next flight (uploaded tiles enter the basemap once promoted to `trusted`) |
|
||||
| F11 | F1 route-driven variant (AZ-974) OR warm cache; E-DEMO-REPLAY (AZ-265) | F1 (corridor cache), replay JSONL + map artifacts consumed by suite UI |
|
||||
|
||||
**Cross-cutting**: F13 FDR-write is not a flow per se — every flow above has an FDR write side-effect. AC-NEW-3 requires every payload class (estimate, IMU, MAVLink, mid-flight tile, system health, failed-tile thumbnail) to be present; rollover is logged, never silent.
|
||||
|
||||
@@ -53,7 +55,7 @@ This flow is offline and not time-critical. **Only Phase 0 reaches `flights` RES
|
||||
|
||||
#### Phase 1 variant — route-driven seeding (cycle 3 — Epic AZ-835 / AZ-836 + AZ-838 + AZ-839)
|
||||
|
||||
A tlog-driven alternative to bbox download lets the operator (or the post-flight replay harness) pre-commit the cache to the precise corridor the drone actually flew. The path is exercised today by the e2e fixture `tests/e2e/replay/conftest.py::operator_pre_flight_setup` (AZ-839) and the orchestrator test `tests/e2e/replay/test_az835_e2e_real_flight.py` (AZ-840); the C12 production CLI binding for this variant is deferred to a future cycle.
|
||||
A tlog-driven alternative to bbox download lets the operator pre-commit the cache to the precise corridor the drone actually flew. **Production bindings** (Epic AZ-969): C12 `seed-cache-from-tlog` (AZ-974) and the `replay_api` demo job (AZ-973) call the same `operator_replay.cache_seed` module. The e2e fixture `operator_pre_flight_setup` (AZ-839) is a thin wrapper over that production path — not a parallel implementation.
|
||||
|
||||
Phase-1 sub-steps in the route-driven variant (replaces the bbox download for that invocation):
|
||||
|
||||
@@ -1083,6 +1085,96 @@ flowchart TD
|
||||
|
||||
---
|
||||
|
||||
## Flow F11: Demo replay validation (operator)
|
||||
|
||||
### Description
|
||||
|
||||
Post-flight **product demo** and **validation** flow. The operator uploads a nav-camera video and ArduPilot `.tlog` through the suite UI (AZ-897), visually aligns the two recordings on dual timeline bars, and runs the same airborne GPS-denied pipeline used in live flight — against a corridor cache seeded from the tlog GPS trace. Output: per-tick estimated positions (JSONL), accuracy map, and PASS/FAIL verdict against tlog ground truth (AZ-696 AC-3).
|
||||
|
||||
This is **not** a test-harness shortcut. E2E tests (AZ-840) call the same `replay_api` orchestration (AZ-973) and `operator_replay.cache_seed` (AZ-974) as the UI.
|
||||
|
||||
**Phases** (sequenced by `replay_api` demo job or manual CLI equivalents):
|
||||
|
||||
1. **Preview** (AZ-970) — parse tlog IMU2 activity + video metadata for UI timelines.
|
||||
2. **Align** (AZ-897 + AZ-971) — operator coarse offset; backend refine via optical-flow + IMU cross-correlation.
|
||||
3. **Export** (AZ-972) — write AZ-896 canonical CSV with `Time=0` at aligned video frame 0 (single canonical clock for replay).
|
||||
4. **Seed cache** (AZ-974) — `extract_route_from_tlog` → `SatelliteProviderRouteClient.seed_route` → tile download → FAISS build (F1 route-driven variant).
|
||||
5. **Replay** — `gps-denied-replay --video … --imu aligned.csv` with `config.mode=replay`; C1–C5 identical to live.
|
||||
6. **Verdict** — horizontal-error distribution + map artifact returned to UI.
|
||||
|
||||
Advanced bypass: operator may upload a pre-aligned `(video, CSV)` per AZ-959 without steps 1–3.
|
||||
|
||||
### Preconditions
|
||||
|
||||
- Operator workstation runs `replay_api` (docker-compose or native) with network to `satellite-provider`.
|
||||
- Camera calibration JSON for the flight's nav camera.
|
||||
- Tlog contains `SCALED_IMU2` (or `RAW_IMU`) and `GLOBAL_POSITION_INT` / `GPS_RAW_INT`.
|
||||
- Video covers the active flight segment after alignment.
|
||||
|
||||
### Sequence Diagram
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Operator
|
||||
participant UI as [[suite/ui]] AZ-897
|
||||
participant API as replay_api AZ-973
|
||||
participant Align as replay_input alignment AZ-971
|
||||
participant Export as tlog_to_csv AZ-972
|
||||
participant Seed as operator_replay cache_seed AZ-974
|
||||
participant Sat as [[satellite-provider]]
|
||||
participant Replay as gps-denied-replay
|
||||
participant Pipeline as C1..C5 replay mode
|
||||
|
||||
Operator->>UI: upload video + tlog + calibration
|
||||
UI->>API: POST /replay/preview
|
||||
API-->>UI: video metadata + IMU2 activity timeline
|
||||
Operator->>UI: drag video bar / refine
|
||||
UI->>API: POST /replay/align/refine
|
||||
API->>Align: refine_video_offset
|
||||
Align-->>UI: refined_offset_ms + confidence
|
||||
Operator->>UI: Run demo
|
||||
UI->>API: POST /replay/demo
|
||||
API->>Export: export_aligned_csv
|
||||
API->>Seed: extract_route + seed_route + FAISS
|
||||
Seed->>Sat: POST /api/satellite/route
|
||||
Sat-->>Seed: mapsReady
|
||||
API->>Replay: subprocess --video --imu
|
||||
Replay->>Pipeline: per-frame loop
|
||||
Pipeline-->>API: results.jsonl
|
||||
API-->>UI: map URL + verdict report
|
||||
```
|
||||
|
||||
### Data flow
|
||||
|
||||
| Step | From | To | Data | Format |
|
||||
|------|------|----|------|--------|
|
||||
| 1 | UI | replay_api | video + tlog multipart | HTTP |
|
||||
| 2 | replay_api | UI | timeline preview JSON | JSON |
|
||||
| 3 | UI | replay_api | `video_offset_ms` | JSON |
|
||||
| 4 | replay_api | disk | aligned `data_imu.csv` | AZ-896 CSV |
|
||||
| 5 | replay_api | satellite-provider | `RouteSpec` waypoints | JSON POST |
|
||||
| 6 | replay_api | airborne binary | video + CSV + cache config | subprocess |
|
||||
| 7 | replay_api | UI | JSONL path, map URL, verdict md | JSON job result |
|
||||
|
||||
### Error scenarios
|
||||
|
||||
| Error | Detection | Recovery |
|
||||
|-------|-----------|----------|
|
||||
| Missing IMU in tlog | preview 422 | Operator message; cannot align |
|
||||
| Refine hard-fail (< 95 % frame match) | align/refine response | Operator adjusts bar or aborts |
|
||||
| Route seed terminal failure | `RouteTerminalFailureError` | Job failed; operator retries |
|
||||
| ESKF divergence (no cache) | replay exit ≠ 0 | Ensure step 4 completed; check AZ-963 |
|
||||
|
||||
### Performance expectations
|
||||
|
||||
| Metric | Target | Notes |
|
||||
|--------|--------|-------|
|
||||
| Preview latency | p95 < 5 s | tlog parse + video probe |
|
||||
| Full demo (Derkachi) | ≤ 15 min cold | matches AZ-835 AC-7 |
|
||||
| Warm cache reuse | ≤ 30 s seed skip | named volume / cache_root reuse |
|
||||
|
||||
---
|
||||
|
||||
## Cross-cutting: FDR write side-effect
|
||||
|
||||
Every flow above produces FDR records (per AC-NEW-3). The cross-cutting rules are:
|
||||
|
||||
@@ -203,6 +203,17 @@ are all declared and documented below under **Cycle Check**.
|
||||
| AZ-951 | OKVIS2 v2 upstream patch: expose 6×6 pose covariance accessor (+ ADR for pin deviation) | 3 | AZ-332; AZ-592 | AZ-254 |
|
||||
| AZ-952 | OKVIS2 v2 upstream patch: expose tracking-stats accessor (counts + parallax + MRE) | 3 | AZ-332; AZ-592; AZ-951 (SOFT) | AZ-254 |
|
||||
| AZ-959 | replay_api: extend POST /replay to accept (video, csv) multipart for AZ-897 UI | 3 | AZ-701; AZ-894; AZ-896 | (none) |
|
||||
| AZ-969 | Demo replay operator flow (Epic) — F11 tlog+video align → cache seed → verdict | 21 (epic) | AZ-894; AZ-836; AZ-838; AZ-701; AZ-959 | AZ-897 |
|
||||
| AZ-970 | Tlog/video timeline preview API (AZ-969 C1) | 3 | AZ-697; AZ-836 | AZ-897; AZ-971 |
|
||||
| AZ-971 | Alignment library restore + refine (AZ-969 C2) | 5 | AZ-405 (historical) | AZ-972; AZ-973 |
|
||||
| AZ-972 | Aligned CSV export from tlog + offset (AZ-969 C3) | 3 | AZ-896; AZ-697; AZ-971; AZ-836 | AZ-973 |
|
||||
| AZ-973 | replay_api demo orchestration endpoints (AZ-969 C4) | 5 | AZ-970; AZ-971; AZ-972; AZ-974 (soft); AZ-960; AZ-701 | AZ-897 |
|
||||
| AZ-974 | C12 seed-cache-from-tlog production CLI (AZ-969 C5) | 3 | AZ-836; AZ-838; AZ-839; AZ-326 | AZ-973 (soft) |
|
||||
| AZ-975 | System design docs F11 + Invariant 15 (AZ-969 C6) | 2 | AZ-969 | (none) |
|
||||
| AZ-976 | gRPC streaming tile provision epic (ADR-013) | Epic ~13 | AZ-838; AZ-316; ADR-004 | AZ-977; AZ-978; AZ-979 |
|
||||
| AZ-977 | satellite-provider TileProvision gRPC service (AZ-976 C1) | 5 | AZ-976 | AZ-978 |
|
||||
| AZ-978 | C11 GrpcTileProvisionClient + C12 wiring (AZ-976 C2) | 5 | AZ-977; AZ-836; AZ-838; AZ-974 (soft) | AZ-979 |
|
||||
| AZ-979 | gRPC tile provision Jetson e2e + benchmark (AZ-976 C3) | 3 | AZ-977; AZ-978 | (none) |
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Replay: hard removal of deprecated auto-sync surface (AZ-895 follow-up)
|
||||
|
||||
> **BLOCKED by Epic AZ-969 (2026-06-19).** AZ-971 restores alignment kernels as operator-driven refine behind `replay_input/alignment.py`. Do not delete alignment logic until AZ-969 ships. AZ-908 scope shrinks to: remove deprecated CLI flags and `auto_sync.py` stub re-exports only — **not** the new alignment module.
|
||||
|
||||
**Task**: AZ-908_replay_auto_sync_hard_removal
|
||||
**Name**: Cycle-5+ cleanup that physically removes the auto-sync surface AZ-895 deprecated
|
||||
**Description**: Follow-up to AZ-895 (cycle 4). AZ-895 made the auto_sync surface a no-op and deprecated the CLI flags (`--time-offset-ms`, `--skip-auto-sync`, `--auto-trim`) with one-cycle warnings, but left the call sites, config fields, and interface DTOs intact for backward compat. AZ-908 completes the removal in cycle 5+ after a one-cycle deprecation window has passed.
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# Operator replay sync UI (relocated)
|
||||
|
||||
**Task**: AZ-897_operator_replay_sync_ui
|
||||
**Tracker**: AZ-897
|
||||
**Repo**: `../ui` (Azaion suite front-end)
|
||||
|
||||
Authoritative spec: `ui/_docs/02_tasks/todo/AZ-897_operator_replay_sync_ui.md` (sibling repo at `../ui` relative to monorepo root).
|
||||
|
||||
Parent epic (backend): [AZ-969_demo_replay_operator_flow_epic.md](./AZ-969_demo_replay_operator_flow_epic.md)
|
||||
|
||||
Implement in the UI workspace. Backend blockers: AZ-970, AZ-973.
|
||||
@@ -0,0 +1,66 @@
|
||||
# Demo replay operator flow (Epic)
|
||||
|
||||
**Task**: AZ-969_demo_replay_operator_flow_epic
|
||||
**Name**: Demo replay operator flow — tlog + video alignment → cache seed → airborne replay verdict
|
||||
**Description**: Promote the demo replay path from an e2e-test harness concern to a first-class operator workflow (F11). Given raw `(video, tlog, calibration)`, the system lets the operator align timelines in the suite UI, exports a canonical aligned CSV, seeds the satellite corridor cache from the tlog, runs the airborne replay pipeline, and returns a map + accuracy verdict. Supersedes the cycle-4 `(video, CSV)` upload-only shortcut as the **default** demo entry; CSV upload remains an advanced bypass.
|
||||
**Complexity**: Epic — ~21 SP across 6 backend children + AZ-897 UI (5 SP in `../ui`)
|
||||
**Dependencies**: AZ-894 (CSV adapter — done), AZ-836 (route extractor — done), AZ-838 (route client — done), AZ-701 (replay_api — done), AZ-959 (CSV API path — done)
|
||||
**Component**: cross-cutting — `replay_input`, `replay_api`, `c12_operator_orchestrator`, `c11_tile_manager`
|
||||
**Tracker**: AZ-969 (https://denyspopov.atlassian.net/browse/AZ-969)
|
||||
**Originating directive**: user (2026-06-19) — demo flow must accept tlog + video with manual alignment UI; not test-only.
|
||||
|
||||
## Goal
|
||||
|
||||
An operator with no Python install completes the full GPS-denied validation demo from the suite UI: upload → align → run → read verdict. The same code path powers Tier-2 e2e (`test_az835_e2e_real_flight`) without a separate test-only fixture graph.
|
||||
|
||||
## Pipeline (7 steps — production, not test-only)
|
||||
|
||||
| # | Step | Owner | New? |
|
||||
|---|------|-------|------|
|
||||
| 1 | Preview timelines (video metadata + tlog IMU2 activity) | AZ-970 `replay_api` | **New** |
|
||||
| 2 | Operator coarse-align + backend refine offset | AZ-897 UI + AZ-971 | **New** |
|
||||
| 3 | Export aligned CSV (`Time` col = video frame 0) | AZ-972 | **New** |
|
||||
| 4 | Extract route + seed corridor tiles + FAISS | AZ-974 (promotes AZ-836/838 from e2e fixture) | **Wire production** |
|
||||
| 5 | Run `gps-denied-replay` on `(video, aligned_csv)` | existing CLI + AZ-973 orchestration | existing |
|
||||
| 6 | Render map + verdict report | AZ-960 path | done |
|
||||
| 7 | Display in UI | AZ-897 | **New** |
|
||||
|
||||
## Decomposition
|
||||
|
||||
| # | Ticket | Est | Repo | Depends |
|
||||
|---|--------|-----|------|---------|
|
||||
| C1 | AZ-970 — tlog/video preview API | 3 | onboard | — |
|
||||
| C2 | AZ-971 — alignment library restore + refine | 5 | onboard | AZ-970 (soft) |
|
||||
| C3 | AZ-972 — aligned CSV export | 3 | onboard | AZ-971 |
|
||||
| C4 | AZ-973 — replay_api demo orchestration endpoints | 5 | onboard | AZ-972, AZ-974 (soft) |
|
||||
| C5 | AZ-974 — C12 `seed-cache-from-tlog` production CLI | 3 | onboard | AZ-836, AZ-838 |
|
||||
| C6 | AZ-975 — system design docs (F11, protocol, architecture) | 2 | onboard | C1–C5 specs |
|
||||
| UI | AZ-897 — dual-timeline sync UI | 5 | `../ui` | AZ-970, AZ-973 |
|
||||
|
||||
**Total ~21 SP backend + 5 SP UI.**
|
||||
|
||||
## Architectural decisions
|
||||
|
||||
1. **Single canonical clock preserved** — alignment happens **before** replay; exported CSV's `Time` column is authoritative (Invariant 14.a unchanged). Tlog runtime parsing is not reintroduced into `compose_root`.
|
||||
2. **Alignment is operator-visible** — auto-sync (AZ-405) is restored as a **refinement kernel** behind explicit operator consent, not a silent default.
|
||||
3. **Route seeding leaves test fixtures** — `extract_route_from_tlog` becomes a C12/replay_api production step, not only `operator_pre_flight_setup`.
|
||||
4. **AZ-908 deferred** — hard removal of alignment stubs blocked until AZ-971 lands; stub module renamed, not deleted.
|
||||
|
||||
## Acceptance criteria (Epic-level)
|
||||
|
||||
- **AC-1**: F11 documented in `system-flows.md` with sequence diagram; `architecture.md` lists demo flow alongside F1–F10.
|
||||
- **AC-2**: `POST /replay/demo` runs steps 3–6 without manual CLI on docker-compose dev stack.
|
||||
- **AC-3**: AZ-897 UI completes Derkachi demo end-to-end against local `replay_api`.
|
||||
- **AC-4**: `tests/e2e/replay/test_az835_e2e_real_flight.py` refactored to call production orchestration API/helpers — no parallel test-only graph.
|
||||
- **AC-5**: Advanced `(video, csv)` upload still works (AZ-959 regression green).
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Replacing live FC adapter with tlog at runtime (F3 stays live MAVLink).
|
||||
- OKVIS2 / AZ-943 chain.
|
||||
- Removing CSV bypass path (AZ-908 remains backlog after this epic).
|
||||
|
||||
## Coordination
|
||||
|
||||
- **AZ-897** spec: `../ui/_docs/02_tasks/todo/AZ-897_operator_replay_sync_ui.md`
|
||||
- **AZ-908** backlog: amend — do not execute until AZ-969 ships
|
||||
@@ -0,0 +1,79 @@
|
||||
# Tlog/video timeline preview API
|
||||
|
||||
**Task**: AZ-970_tlog_timeline_preview_api
|
||||
**Name**: `replay_api` preview endpoint — video metadata + tlog IMU2 activity timeline for AZ-897 UI
|
||||
**Description**: First backend building block of Epic AZ-969. Exposes `POST /replay/preview` accepting `(video, tlog)` multipart and returning JSON the dual-bar UI needs: video duration/fps/frame count, tlog duration, active-flight segment bounds, and per-bin IMU2 activity energy for heatmap rendering. Pure read-only — no alignment, no replay.
|
||||
**Complexity**: 3 SP
|
||||
**Dependencies**: AZ-697 (`load_tlog_ground_truth` — done), AZ-836 (`_detect_active_segment` semantics — reuse via shared trim helper or import)
|
||||
**Blocks**: AZ-897 (UI), AZ-971 (soft — refine can ship without preview in isolation but UI cannot)
|
||||
**Component**: `replay_api` + new `replay_input/tlog_timeline.py`
|
||||
**Tracker**: AZ-970
|
||||
**Parent Epic**: AZ-969
|
||||
|
||||
## Public surface
|
||||
|
||||
```python
|
||||
# replay_input/tlog_timeline.py
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Imu2ActivityBin:
|
||||
t_ms: int # bin start, FC-boot-relative ms
|
||||
energy: float # 0..1 normalized IMU2 magnitude
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class TlogTimelinePreview:
|
||||
duration_ms: int
|
||||
active_segment: tuple[int, int] # (start_idx, end_idx) into GPS rows
|
||||
active_start_ms: int
|
||||
active_end_ms: int
|
||||
imu2_activity: tuple[Imu2ActivityBin, ...]
|
||||
has_scaled_imu2: bool
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class VideoTimelinePreview:
|
||||
duration_ms: int
|
||||
frame_count: int
|
||||
fps: float
|
||||
|
||||
def build_tlog_timeline_preview(tlog: Path, *, bin_width_ms: int = 100) -> TlogTimelinePreview: ...
|
||||
def build_video_timeline_preview(video: Path) -> VideoTimelinePreview: ...
|
||||
```
|
||||
|
||||
## HTTP
|
||||
|
||||
`POST /replay/preview` — multipart `video` + `tlog` (both required).
|
||||
|
||||
Response 200:
|
||||
```json
|
||||
{
|
||||
"video": { "duration_ms": 490000, "frame_count": 14700, "fps": 30.0 },
|
||||
"tlog": {
|
||||
"duration_ms": 520000,
|
||||
"active_segment": [120, 4980],
|
||||
"active_start_ms": 12000,
|
||||
"active_end_ms": 498000,
|
||||
"imu2_activity": [{ "t_ms": 0, "energy": 0.02 }, ...],
|
||||
"has_scaled_imu2": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Errors: 400 missing file; 422 tlog missing SCALED_IMU2/RAW_IMU; 422 unreadable video.
|
||||
|
||||
## Implementation notes
|
||||
|
||||
- IMU2 energy: RMS of `(xacc,yacc,zacc)` from SCALED_IMU2 messages, binned, min-max normalized over full tlog.
|
||||
- Reuse active-segment thresholds from `extract_route_from_tlog` defaults for consistency.
|
||||
- Video probe via OpenCV `cv2.VideoCapture` — lazy-import gated like existing replay paths.
|
||||
- Optional: persist upload to temp job dir (same storage as AZ-701) and return `preview_id` for subsequent refine/demo calls.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **AC-1**: Derkachi tlog returns ≥ 1 activity peak in active segment; pre-takeoff bins < 0.15 normalized energy.
|
||||
- **AC-2**: Derkachi video returns fps within 0.5 of ffprobe ground truth.
|
||||
- **AC-3**: Unit tests for binning + normalization without disk video (synthetic IMU samples).
|
||||
- **AC-4**: Integration test in `test_az701_replay_api.py` for happy path + missing IMU types.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Thumbnail strip generation (UI may request later; optional `GET /replay/preview/{id}/frames` follow-up).
|
||||
- Alignment refine (AZ-971).
|
||||
@@ -0,0 +1,59 @@
|
||||
# Alignment library restore + refine offset
|
||||
|
||||
**Task**: AZ-971_alignment_library_restore_refine
|
||||
**Name**: Restore `replay_input` alignment kernels (AZ-405) as operator-driven refine behind explicit offset
|
||||
**Description**: Second building block of Epic AZ-969. AZ-895 replaced `auto_sync.py` with raising stubs. Restore the pure compute kernels from pre-AZ-895 history (`_compute_tlog_takeoff_from_samples`, `_compute_video_onset_from_samples`, `validate_offset_or_fail`, `find_aligned_window` from AZ-698) into a new module `replay_input/alignment.py`. Public API: `refine_video_offset(tlog, video, manual_offset_ms) -> AlignmentResult` — takes the operator's coarse bar offset and returns refined offset + confidence + frame-window match %. No silent auto-run at upload.
|
||||
**Complexity**: 5 SP
|
||||
**Dependencies**: AZ-405 (historical implementation — restore from git), AZ-698 (`find_aligned_window` — optional cross-correlation pass)
|
||||
**Blocks**: AZ-972, AZ-973
|
||||
**Component**: `replay_input/alignment.py`
|
||||
**Tracker**: AZ-971
|
||||
**Parent Epic**: AZ-969
|
||||
|
||||
## Public surface
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AlignmentResult:
|
||||
manual_offset_ms: int
|
||||
refined_offset_ms: int
|
||||
confidence: float # 0..1
|
||||
frame_window_match_pct: float # AC-8 metric
|
||||
hard_fail: bool
|
||||
|
||||
def refine_video_offset(
|
||||
tlog: Path,
|
||||
video: Path,
|
||||
manual_offset_ms: int,
|
||||
*,
|
||||
target_fc_dialect: str = "ardupilot_plane",
|
||||
match_threshold_pct: float = 95.0,
|
||||
) -> AlignmentResult: ...
|
||||
```
|
||||
|
||||
Semantics: `refined_offset_ms` = best offset after cross-correlating IMU energy (from manual anchor ± 2 s window) with video optical-flow onset. If `frame_window_match_pct < match_threshold_pct`, set `hard_fail=True` but still return best offset (UI decides whether to proceed).
|
||||
|
||||
## Scope
|
||||
|
||||
1. New `replay_input/alignment.py` with restored kernels (not re-exported from deprecated `auto_sync.py`).
|
||||
2. `auto_sync.py` stubs updated to delegate to `alignment` with deprecation warning OR left as-is until AZ-908 post-AZ-969.
|
||||
3. Unit tests ported from AZ-405 / AZ-698 test matrix (synthetic fixtures).
|
||||
4. `POST /replay/align/refine` handler stub in AZ-973 may call this module — implement library here first.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **AC-1**: Derkachi fixture with known ground-truth offset: `refine_video_offset` within ± 200 ms of truth when manual offset within ± 2 s.
|
||||
- **AC-2**: Deliberately wrong manual offset (± 30 s) → `hard_fail=True`, `frame_window_match_pct < 50`.
|
||||
- **AC-3**: Deterministic: same inputs → same `refined_offset_ms` within 1 ms.
|
||||
- **AC-4**: Missing SCALED_IMU2 → `ReplayInputAdapterError` at entry, not deep in OpenCV.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Automatic alignment without manual seed (operator must drag bar first).
|
||||
- Re-enabling `TlogReplayFcAdapter` in `compose_root`.
|
||||
- AZ-908 hard removal.
|
||||
|
||||
## Notes
|
||||
|
||||
- Restore source from commit before AZ-895 stub landing; do not resurrect `ReplayInputAdapter.open()` tlog path.
|
||||
- Keep OpenCV lazy-import discipline from batch 60.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Aligned CSV export from tlog + video offset
|
||||
|
||||
**Task**: AZ-972_aligned_csv_export
|
||||
**Name**: Export AZ-896 canonical CSV from tlog trimmed and aligned to video frame 0
|
||||
**Description**: Third building block of Epic AZ-969. Given `(tlog, video_offset_ms, optional active_segment)`, stream-parse the tlog and write a CSV matching `csv_replay_format.md`: `Time` column starts at 0.0 s at the video frame that aligns to the chosen tlog instant; only rows inside the active flight segment are exported; IMU + GLOBAL_POSITION_INT columns populated at 10 Hz (resample if needed).
|
||||
**Complexity**: 3 SP
|
||||
**Dependencies**: AZ-896 (format spec — done), AZ-697 (`load_tlog_ground_truth` / IMU parse), AZ-971 (refined offset input), AZ-836 (active segment detection — reuse)
|
||||
**Blocks**: AZ-973
|
||||
**Component**: `replay_input/tlog_to_csv.py` + CLI `gps-denied-tlog-to-csv`
|
||||
**Tracker**: AZ-972
|
||||
**Parent Epic**: AZ-969
|
||||
|
||||
## Public surface
|
||||
|
||||
```python
|
||||
def export_aligned_csv(
|
||||
tlog: Path,
|
||||
output_csv: Path,
|
||||
*,
|
||||
video_offset_ms: int,
|
||||
active_segment: tuple[int, int] | None = None,
|
||||
min_takeoff_speed_m_s: float = 2.0,
|
||||
min_takeoff_altitude_agl_m: float = 5.0,
|
||||
) -> Path: ...
|
||||
```
|
||||
|
||||
CLI: `gps-denied-tlog-to-csv --tlog PATH --output PATH --video-offset-ms N [--active-segment START,END]`
|
||||
|
||||
## Alignment math
|
||||
|
||||
Let `tlog_anchor_ms` be the FC-boot-relative instant matching video `t=0` after applying `video_offset_ms` (positive = video starts before tlog anchor). For each exported row at tlog time `t_fc_ms`:
|
||||
|
||||
`Time = (t_fc_ms - tlog_anchor_ms) / 1000.0`
|
||||
|
||||
Only rows with `Time >= 0` and within active segment are emitted. First row MUST have `Time == 0` within one IMU sample period (Invariant 14.a / AZ-896).
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **AC-1**: Round-trip: export Derkachi with known offset → `load_csv_ground_truth` → 10 Hz monotonic `Time`.
|
||||
- **AC-2**: `gps-denied-replay --video derkachi.mp4 --imu exported.csv` starts without `ReplayInputAdapterError`.
|
||||
- **AC-3**: Row count matches active segment duration × 10 Hz ± 1 row.
|
||||
- **AC-4**: Unit test: schema header exact match to `example_data_imu.csv`.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- PX4 / non-ArduPilot dialects.
|
||||
- Magnetometer columns (optional in AZ-896).
|
||||
@@ -0,0 +1,47 @@
|
||||
# replay_api demo orchestration endpoints
|
||||
|
||||
**Task**: AZ-973_replay_api_demo_orchestration
|
||||
**Name**: `replay_api` align/refine/export/demo endpoints — production F11 orchestrator
|
||||
**Description**: Fourth building block of Epic AZ-969. Extends `replay_api` with the operator demo job lifecycle: refine offset, export aligned CSV, run full pipeline (export → route seed → subprocess replay → map render → verdict). Replaces the ad-hoc wiring in `tests/e2e/replay/conftest.py` and `_operator_pre_flight.py` as the canonical orchestration surface for demo runs.
|
||||
**Complexity**: 5 SP
|
||||
**Dependencies**: AZ-970, AZ-971, AZ-972, AZ-974 (soft — demo can use pre-seeded cache env override), AZ-960 (map — done), AZ-701 (job storage — done)
|
||||
**Blocks**: AZ-897 (UI)
|
||||
**Component**: `replay_api`
|
||||
**Tracker**: AZ-973
|
||||
**Parent Epic**: AZ-969
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|--------|------|---------|
|
||||
| POST | `/replay/preview` | AZ-970 (may land in same or prior batch) |
|
||||
| POST | `/replay/align/refine` | Body/json: `{ job_id, video_offset_ms }` → `AlignmentResult` |
|
||||
| POST | `/replay/align/export` | Returns aligned CSV bytes or `{ csv_path }` in job dir |
|
||||
| POST | `/replay/demo` | multipart: `video`, `tlog`, `calibration`, `video_offset_ms` → starts async job |
|
||||
| GET | `/jobs/{id}` | Extend status with `phase`: `queued`, `aligning`, `exporting_csv`, `seeding_cache`, `replaying`, `rendering_map`, `complete`, `failed` |
|
||||
|
||||
## Demo job pipeline (in-process or subprocess chain)
|
||||
|
||||
1. Validate uploads; persist to job dir.
|
||||
2. `refine_video_offset` (AZ-971) — log refined offset; fail job if `hard_fail` and `REPLAY_API_STRICT_ALIGN=1`.
|
||||
3. `export_aligned_csv` (AZ-972) → `{job}/work/data_imu.csv`.
|
||||
4. `extract_route_from_tlog` + `SatelliteProviderRouteClient.seed_route` + tile download + FAISS build (delegate to shared helper extracted from `tests/e2e/replay/_operator_pre_flight.py` — **move to** `src/gps_denied_onboard/operator_replay/cache_seed.py` or `replay_api/orchestrator.py`).
|
||||
5. Shell `gps-denied-replay --video ... --imu ... --output ...` with populated `GPS_DENIED_OPERATOR_CONFIG_PATH` / cache mount.
|
||||
6. `_maybe_render_map` + verdict report (AZ-960 / AZ-699 paths).
|
||||
|
||||
## Refactor requirement
|
||||
|
||||
Extract `populate_c6_from_route` from test module into production package importable by both `replay_api` and C12. E2e fixture becomes thin wrapper calling production orchestrator. Satisfies Epic AC-4.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **AC-1**: `POST /replay/demo` on Derkachi fixtures (docker-compose) reaches `phase=complete` with map URL + verdict markdown path in response.
|
||||
- **AC-2**: `GET /jobs/{id}` exposes phase transitions in order.
|
||||
- **AC-3**: Unit tests mock satellite-provider; no network in unit tier.
|
||||
- **AC-4**: `test_az835_e2e_real_flight` refactored to call production orchestrator helper (same code path as API).
|
||||
- **AC-5**: AZ-959 `(video, csv)` bypass unchanged.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- WebSocket progress streaming (poll-only for v1).
|
||||
- Authentication changes beyond AZ-701 bearer token.
|
||||
@@ -0,0 +1,45 @@
|
||||
# C12 production CLI — seed cache from tlog route
|
||||
|
||||
**Task**: AZ-974_c12_seed_cache_from_tlog
|
||||
**Name**: C12 `seed-cache-from-tlog` — production binding for route-driven cache build (AZ-836 + AZ-838)
|
||||
**Description**: Fifth building block of Epic AZ-969. Promotes `extract_route_from_tlog` + `SatelliteProviderRouteClient.seed_route` + C11 tile download + C10 FAISS build from the e2e-only `operator_pre_flight_setup` fixture into the C12 operator CLI. Operators and `replay_api` demo jobs invoke the same production module — not test `conftest.py`.
|
||||
**Complexity**: 3 SP
|
||||
**Dependencies**: AZ-836, AZ-838, AZ-839 (fixture reference impl), AZ-326 (C12 CLI — done)
|
||||
**Blocks**: AZ-973 (soft — demo can seed inline via shared module landed here)
|
||||
**Component**: `c12_operator_orchestrator` + extracted `operator_replay/cache_seed.py`
|
||||
**Tracker**: AZ-974
|
||||
**Parent Epic**: AZ-969
|
||||
|
||||
## CLI
|
||||
|
||||
```
|
||||
gps-denied-operator seed-cache-from-tlog \
|
||||
--tlog PATH \
|
||||
--cache-root PATH \
|
||||
[--max-waypoints 10] \
|
||||
[--region-size-meters 500]
|
||||
```
|
||||
|
||||
Exit 0 on `PopulatedC6Cache` written; exit 2 on `RouteValidationError` / `RouteExtractionError`; exit 1 on transient exhaustion.
|
||||
|
||||
## Shared module
|
||||
|
||||
Move core of `tests/e2e/replay/_operator_pre_flight.py::populate_c6_from_route` to:
|
||||
|
||||
`src/gps_denied_onboard/operator_replay/cache_seed.py`
|
||||
|
||||
Public: `populate_c6_from_route(route_spec, *, cache_root, config) -> PopulatedC6Cache`
|
||||
|
||||
Imported by: C12 CLI, `replay_api` orchestrator (AZ-973), thinned e2e fixture.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **AC-1**: CLI succeeds against mock/real satellite-provider in docker-compose test stack.
|
||||
- **AC-2**: Output matches `PopulatedC6Cache` shape from AZ-839.
|
||||
- **AC-3**: `system-flows.md` F11 Phase 1 references this CLI — not "deferred to future cycle".
|
||||
- **AC-4**: E2e fixture imports production module; no duplicate logic in `tests/`.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Bbox-driven F1 Phase 1 (unchanged).
|
||||
- Companion NVM push (separate C12 bring-up).
|
||||
@@ -0,0 +1,30 @@
|
||||
# System design — F11 demo replay operator flow docs
|
||||
|
||||
**Task**: AZ-975_demo_replay_system_design_docs
|
||||
**Name**: Document F11 demo replay operator flow in system-flows, architecture, replay_protocol
|
||||
**Description**: Sixth building block of Epic AZ-969. Capture the demo replay path as a first-class system flow (F11), update architecture and replay protocol invariants, amend F1 route-driven variant to reference production C12/replay_api bindings, and cross-link AZ-897 UI spec.
|
||||
**Complexity**: 2 SP
|
||||
**Dependencies**: AZ-969 epic spec (this lands with or immediately after child specs)
|
||||
**Blocks**: (none)
|
||||
**Component**: `_docs/02_document/`
|
||||
**Tracker**: AZ-975
|
||||
**Parent Epic**: AZ-969
|
||||
|
||||
## Modified files
|
||||
|
||||
1. `_docs/02_document/system-flows.md` — add F11 to inventory + full section (sequence, flowchart, data flow).
|
||||
2. `_docs/02_document/architecture.md` — replace cycle-4 AZ-897 row; add § "Demo replay operator flow (cycle 5 — AZ-969)".
|
||||
3. `_docs/02_document/contracts/replay/replay_protocol.md` — add **Invariant 15** (operator demo path); note AZ-908 deferred.
|
||||
4. `_docs/how_to_test.md` — align with tlog+video UI flow (user-facing intent).
|
||||
5. `_docs/02_tasks/_dependencies_table.md` — register AZ-969 children.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **AC-1**: F11 appears in flow inventory; depends on F1 route variant + replay mode.
|
||||
- **AC-2**: Invariant 15 documents: raw upload → align → export CSV → single clock replay.
|
||||
- **AC-3**: No doc claims route seeding is "test-only" or "deferred" without pointing at AZ-974.
|
||||
- **AC-4**: `../ui` AZ-897 spec cross-linked.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Jira bulk sync (process leftover).
|
||||
@@ -0,0 +1,54 @@
|
||||
# gRPC streaming tile provision (Epic)
|
||||
|
||||
**Task**: AZ-976_grpc_tile_provision_epic
|
||||
**Name**: gRPC streaming tile provision — route + local index in, batched tiles out
|
||||
**Description**: Replace operator-side REST pre-flight tile transfer (`route poll` + `inventory` + per-tile GET) with a single gRPC server-streaming RPC. satellite-provider streams cached tiles immediately while fetching missing tiles from external imagery; gps-denied sends a local tile index so SP skips tiles the client already has at equal-or-better quality and equal-or-newer capture time. Documented in ADR-013 and `tile_provision.proto`.
|
||||
**Complexity**: Epic — ~13 SP across 3 children (split repos)
|
||||
**Dependencies**: AZ-838 (route client — done), AZ-316 (tile downloader — done), ADR-004 (operator-only boundary)
|
||||
**Component**: cross-cutting — `c11_tile_manager`, `c12_operator_orchestrator`, satellite-provider (sibling repo)
|
||||
**Tracker**: pending
|
||||
**Originating directive**: user (2026-06-19) — speed up pre-flight cache fill; gRPC streaming with client-side dedup index.
|
||||
|
||||
## Goal
|
||||
|
||||
Minimize wall-clock from route submit → C6 cache complete on the operator workstation. Time-to-first-tile and total bytes on the wire both improve vs REST.
|
||||
|
||||
## Pipeline
|
||||
|
||||
| Step | Owner | Mechanism |
|
||||
|------|-------|-----------|
|
||||
| 1 | C12 | Build `Route` + collect `local_tiles` from C6 (route bbox intersection) |
|
||||
| 2 | C11 | `DeliverRouteTiles` gRPC call |
|
||||
| 3 | satellite-provider | Skip dedup → stream `CACHED` batches → fetch externals → stream `FRESHLY_FETCHED` batches |
|
||||
| 4 | C11 | Write batches to C6 (existing gates) |
|
||||
| 5 | Operator | Stage C6 volume to Jetson (USB/rsync) — unchanged |
|
||||
|
||||
## Decomposition
|
||||
|
||||
| # | Ticket | Est | Repo | Depends |
|
||||
|---|--------|-----|------|---------|
|
||||
| C1 | AZ-977 — satellite-provider `RouteTileDelivery` gRPC service | 5 | `../satellite-provider` | — |
|
||||
| C2 | AZ-978 — C11 `RouteTileDeliveryClient` + C12 integration | 5 | onboard | AZ-977 |
|
||||
| C3 | AZ-979 — Jetson e2e smoke + ADR/doc sync | 3 | onboard + SP | AZ-978 |
|
||||
|
||||
**Total ~13 SP.**
|
||||
|
||||
## Acceptance criteria (Epic-level)
|
||||
|
||||
- **AC-1**: ADR-013 accepted in `architecture.md`; `tile_provision.proto` + `tile_provision_grpc.md` published.
|
||||
- **AC-2**: Derkachi corridor provision completes over gRPC with fewer round-trips than REST baseline (measured in AZ-979 report).
|
||||
- **AC-3**: Client local index suppresses re-transfer when C6 already holds equal-or-better tile (unit test on skip rule).
|
||||
- **AC-4**: Airborne image build excludes gRPC provision stubs (ADR-004 regression test unchanged).
|
||||
- **AC-5**: REST `route_client` + `HttpTileDownloader` remain as fallback until AZ-979 marks gRPC primary.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- In-flight tile download on the UAV (RESTRICT-SAT-1)
|
||||
- Implementing REST `POST /api/satellite/tiles/inventory` (superseded by this epic)
|
||||
- Browser/Web UI transport (operator CLI / C12 first)
|
||||
|
||||
## References
|
||||
|
||||
- ADR-013 — `_docs/02_document/architecture.md`
|
||||
- Proto — `_docs/02_document/contracts/c11_tilemanager/tile_provision.proto`
|
||||
- Contract — `_docs/02_document/contracts/c11_tilemanager/tile_provision_grpc.md`
|
||||
@@ -0,0 +1,23 @@
|
||||
# satellite-provider TileProvision gRPC service
|
||||
|
||||
**Task**: AZ-977_sp_tile_provision_grpc_service
|
||||
**Epic**: AZ-976
|
||||
**Name**: Implement `RouteTileDelivery.DeliverRouteTiles` in satellite-provider
|
||||
**Description**: Add gRPC host implementing `satellite.v1.RouteTileDelivery` from `tile_provision.proto`. Emit `RouteManifest` first, stream `TileBatch` (cached tiles before external fetch), optional `ProgressUpdate`, then `DeliveryComplete` or `DeliveryError`. JWT via gRPC metadata.
|
||||
**Complexity**: 5 SP
|
||||
**Dependencies**: AZ-976 (proto contract)
|
||||
**Component**: satellite-provider (sibling repo)
|
||||
**Tracker**: pending
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **AC-1**: `DeliverRouteTiles` stream matches `tile_provision_grpc.md` event sequence.
|
||||
- **AC-2**: Skip rule omits tiles when client snapshot is equal-or-better resolution and equal-or-newer `captured_at`.
|
||||
- **AC-3**: `phase=CACHED` batches emit before external fetch completes for on-disk hits.
|
||||
- **AC-4**: gRPC + existing REST coexist behind feature flag until AZ-979 flips default.
|
||||
- **AC-5**: OpenAPI/gRPC reflection or grpcurl smoke documented in satellite-provider README.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- gps-denied Python client (AZ-978)
|
||||
- Post-landing ingest (D-PROJ-2)
|
||||
@@ -0,0 +1,22 @@
|
||||
# C11 RouteTileDeliveryClient
|
||||
|
||||
**Task**: AZ-978_c11_grpc_tile_provision_client
|
||||
**Name**: Python gRPC consumer for RouteTileDelivery + C12 wiring
|
||||
**Description**: Implement `RouteTileDeliveryClient` in `c11_tile_manager` using `grpcio` + stubs from `tile_provision.proto`. Map internal `RouteSpec` → `satellite.v1.RouteSpec`; build `client_tiles` from C6; consume `RouteTileEvent` oneof (manifest, batch, progress, complete, error). Wire from C12 seed path behind `c11.tile_provision.transport: grpc|rest`.
|
||||
**Complexity**: 5 SP
|
||||
**Dependencies**: AZ-977, AZ-974 (soft), AZ-836, AZ-838
|
||||
**Component**: c11_tile_manager, c12_operator_orchestrator
|
||||
**Tracker**: pending
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **AC-1**: Unit tests with fake server cover manifest-first ordering and `batch_seq` resume per `route_id`.
|
||||
- **AC-2**: `local_tiles` populated from C6 metadata query intersecting route corridor.
|
||||
- **AC-3**: RESTRICT-SAT-4 / freshness / budget gates unchanged — reject bad tiles even if SP sent them.
|
||||
- **AC-4**: Generated stubs not imported by airborne/runtime_root build (BUILD flag or package split).
|
||||
- **AC-5**: Config default `rest` until AZ-979 benchmark promotes `grpc`.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- satellite-provider server (AZ-977)
|
||||
- Jetson benchmark report (AZ-979)
|
||||
@@ -0,0 +1,21 @@
|
||||
# gRPC tile provision e2e + benchmark
|
||||
|
||||
**Task**: AZ-979_grpc_tile_provision_e2e_benchmark
|
||||
**Epic**: AZ-976
|
||||
**Name**: Jetson e2e smoke and REST vs gRPC benchmark for tile provision
|
||||
**Description**: Add Tier-2 smoke test calling `RouteTileDeliveryClient` against real satellite-provider on Jetson harness. Benchmark wall-clock and bytes transferred vs REST path on Derkachi corridor. Update `architecture.md` integration table to mark gRPC primary. Document resume behaviour after disconnect.
|
||||
**Complexity**: 3 SP
|
||||
**Dependencies**: AZ-978, AZ-977
|
||||
**Component**: tests/e2e, docs
|
||||
**Tracker**: pending
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- **AC-1**: `tests/e2e/satellite_provider/test_grpc_provision.py` passes on Jetson with `JETSON_SSH_ALIAS=jetson`.
|
||||
- **AC-2**: Benchmark report in `_docs/06_metrics/` with REST vs gRPC timings and byte counts.
|
||||
- **AC-3**: `docker-compose.test.jetson.yml` exposes gRPC port for satellite-provider.
|
||||
- **AC-4**: `c11.tile_provision.transport` default flipped to `grpc` after green benchmark.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Deprecating REST route_client in same ticket (follow-up after soak)
|
||||
@@ -0,0 +1,61 @@
|
||||
# Batch 05 — Cycle 4 Implementation Report
|
||||
|
||||
**Date:** 2026-09-06
|
||||
**Task:** AZ-963 — Fix Derkachi 60 s smoke regressions (ESKF divergence on CSV-only path)
|
||||
**Chosen option:** D (xfail with rationale) + E (investigate XPASS)
|
||||
|
||||
## Changes
|
||||
|
||||
### `tests/e2e/replay/test_derkachi_1min.py`
|
||||
|
||||
Added `@pytest.mark.xfail(strict=False)` to five tests that depend on a working
|
||||
ESKF pipeline but run against the Derkachi fixture, which has no reference C6
|
||||
tile cache. Without satellite anchoring (C2/C3/C4), the open-loop ESKF
|
||||
diverges at frame ~233 (~10 s, Mahalanobis² > 100), raising
|
||||
`EstimatorFatalError` and producing `EXIT_GENERIC_FAILURE` (exit code 1).
|
||||
|
||||
Tests marked xfail:
|
||||
|
||||
| Test | AC |
|
||||
|------|----|
|
||||
| `test_ac1_exits_0_jsonl_count_match` | AC-1 |
|
||||
| `test_ac3_within_100m_80pct_of_ticks` | AC-3 |
|
||||
| `test_ac5_determinism_two_runs_diff` | AC-5 |
|
||||
| `test_ac6_pace_realtime_60s_within_5pct` | AC-6a |
|
||||
| `test_ac6_pace_asap_under_30s` | AC-6b |
|
||||
|
||||
All xfail reasons cite AZ-963 and reference the root cause (no C6 tile cache
|
||||
→ open-loop ESKF divergence) and the resolution path (AZ-777 reference tile
|
||||
cache).
|
||||
|
||||
**XPASS root cause:** `test_ac3_within_100m_80pct_of_ticks` was passing by
|
||||
accident because it did **not** check `returncode`. Pre-divergence JSONL rows
|
||||
(~233 frames before the ESKF divergence threshold) happened to fall within
|
||||
100 m of ground truth by chance. Added `assert result.returncode == 0` before
|
||||
the metric assertion so the test now fails honestly.
|
||||
|
||||
### `tests/e2e/replay/README.md`
|
||||
|
||||
Updated AC matrix: AC-1/AC-3/AC-5/AC-6a/AC-6b now marked `xfail (AZ-963)`.
|
||||
Added AZ-777 to Follow-up work as the only resolution path for AZ-963.
|
||||
Updated Expected runtime notes.
|
||||
|
||||
## Test results
|
||||
|
||||
```
|
||||
tests/e2e/replay/test_derkachi_1min.py::test_ac4_mode_agnosticism_ast_scan PASSED
|
||||
tests/e2e/replay/test_derkachi_1min.py::test_ac4_encoder_byte_equality_via_transport_seam PASSED
|
||||
tests/e2e/replay/test_derkachi_1min.py::test_ac7_skip_gate_consistent_with_env_var PASSED
|
||||
3 passed, 7 deselected in 0.28s
|
||||
```
|
||||
|
||||
All unconditional (non-gated) tests pass. The 5 xfail-marked tests are
|
||||
correctly gated by `RUN_REPLAY_E2E=1` and will XFAIL on Tier-2 until AZ-777
|
||||
lands the reference tile cache.
|
||||
|
||||
## Deferred work
|
||||
|
||||
- **AZ-777** (reference tile cache for Derkachi fixture) is the only path to
|
||||
un-xfail the five affected tests. No other code changes are needed.
|
||||
- **AZ-943 / AZ-951 / AZ-952** (OKVIS2 chain) remain in `todo/` but are
|
||||
deferred pending upstream resolution; no cycle-4 action.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,317 @@
|
||||
[run-tests-jetson] minting fresh dev JWT via scripts/mint_dev_jwt.py
|
||||
[run-tests-jetson] using ssh alias: jetson
|
||||
[run-tests-jetson] remote dir: /home/jetson/gps-denied-onboard
|
||||
[run-tests-jetson] remote satprov: /home/jetson/satellite-provider
|
||||
[run-tests-jetson] compose file: docker-compose.test.jetson.yml
|
||||
[run-tests-jetson] ensure-dev-cert (local)
|
||||
[ensure-dev-cert] cert present at /Users/zxsanny/dev/azaion/gps-denied-onboard/satellite-provider/certs/api.pfx
|
||||
[run-tests-jetson] rsync gps-denied-onboard → jetson:/home/jetson/gps-denied-onboard/
|
||||
Number of files: 1927
|
||||
Number of files transferred: 2
|
||||
Total file size: 384584252 B
|
||||
Total transferred file size: 12082 B
|
||||
Unmatched data: 2815 B
|
||||
Matched data: 9267 B
|
||||
File list size: 136728 B
|
||||
File list generation time: 0.020 seconds
|
||||
File list transfer time: 0.041 seconds
|
||||
Total sent: 137905 B
|
||||
Total received: 172 B
|
||||
|
||||
sent 137905 bytes received 172 bytes 811740 bytes/sec
|
||||
total size is 384584252 speedup is 2785.29
|
||||
[run-tests-jetson] rsync satellite-provider → jetson:/home/jetson/satellite-provider/
|
||||
Number of files: 805
|
||||
Number of files transferred: 2
|
||||
Total file size: 4448030 B
|
||||
Total transferred file size: 19521 B
|
||||
Unmatched data: 3698 B
|
||||
Matched data: 15823 B
|
||||
File list size: 58214 B
|
||||
File list generation time: 0.012 seconds
|
||||
File list transfer time: 0.022 seconds
|
||||
Total sent: 59226 B
|
||||
Total received: 232 B
|
||||
|
||||
sent 59226 bytes received 232 bytes 475283 bytes/sec
|
||||
total size is 4448030 speedup is 74.81
|
||||
[run-tests-jetson] docker compose build e2e-runner (on Jetson)
|
||||
Image gps-denied-onboard/e2e-runner:jetson Building
|
||||
Image gps-denied-onboard/satellite-provider:dev Building
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 1.07kB done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile.jetson
|
||||
#2 transferring dockerfile: 37B
|
||||
#2 transferring dockerfile: 5.82kB done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/dustynv/l4t-pytorch:r36.4.0
|
||||
#3 DONE 0.5s
|
||||
|
||||
#4 [internal] load .dockerignore
|
||||
#4 transferring context: 383B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [1/8] FROM docker.io/dustynv/l4t-pytorch:r36.4.0@sha256:a05c85def9139c21014546451d3baab44052d7cabe854d937f163390bfd5201b
|
||||
#5 resolve docker.io/dustynv/l4t-pytorch:r36.4.0@sha256:a05c85def9139c21014546451d3baab44052d7cabe854d937f163390bfd5201b 0.0s done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [internal] load build context
|
||||
#6 transferring context: 24.56kB 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [4/8] COPY pyproject.toml README.md ./
|
||||
#7 CACHED
|
||||
|
||||
#8 [6/8] RUN rm -f /etc/pip.conf /root/.pip/pip.conf /root/.config/pip/pip.conf
|
||||
#8 CACHED
|
||||
|
||||
#9 [2/8] RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates build-essential libpq-dev libspatialindex-dev libpq5 libspatialindex-c6 libgl1 libglib2.0-0 python3-pip python3-venv && rm -rf /var/lib/apt/lists/*
|
||||
#9 CACHED
|
||||
|
||||
#10 [3/8] WORKDIR /opt
|
||||
#10 CACHED
|
||||
|
||||
#11 [5/8] COPY src ./src
|
||||
#11 CACHED
|
||||
|
||||
#12 [7/8] RUN pip3 install --no-cache-dir --break-system-packages --index-url https://pypi.org/simple --upgrade pip
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] RUN pip3 install --no-cache-dir --break-system-packages --index-url https://pypi.org/simple -e ".[dev]"
|
||||
#13 CACHED
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers 0.0s done
|
||||
#14 exporting manifest sha256:576a6cf55b8c565abc6f2c26b45b8119ef3924d343bfc7f6e2ee32c079230825 done
|
||||
#14 exporting config sha256:155e7d5a011ea9ab1493a930c71a9d0ed2874479d02f58ece9951c97207454cb done
|
||||
#14 exporting attestation manifest sha256:bdd66832b7a8d16539d3398081539fcbd31d568f6195ff15d5275bbc414d6db4 0.0s done
|
||||
#14 exporting manifest list sha256:6253d1aea7392182b2021241c4a4265ea5943e021f3b504de7a721e7e9271884 done
|
||||
#14 naming to docker.io/gps-denied-onboard/e2e-runner:jetson done
|
||||
#14 unpacking to docker.io/gps-denied-onboard/e2e-runner:jetson 0.0s done
|
||||
#14 DONE 0.2s
|
||||
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
Image gps-denied-onboard/e2e-runner:jetson Built
|
||||
[run-tests-jetson] docker compose up e2e-runner (on Jetson)
|
||||
Network gps-denied-onboard_default Creating
|
||||
Network gps-denied-onboard_default Created
|
||||
Container gps-denied-onboard-db-1 Creating
|
||||
Container gps-denied-e2e-satellite-provider-postgres Creating
|
||||
Container gps-denied-e2e-satellite-provider-postgres Created
|
||||
Container gps-denied-e2e-satellite-provider Creating
|
||||
Container gps-denied-onboard-db-1 Created
|
||||
Container gps-denied-e2e-satellite-provider Created
|
||||
Container gps-denied-onboard-e2e-runner-1 Creating
|
||||
Container gps-denied-onboard-e2e-runner-1 Created
|
||||
Attaching to gps-denied-e2e-satellite-provider, gps-denied-e2e-satellite-provider-postgres, db-1, e2e-runner-1
|
||||
Container gps-denied-e2e-satellite-provider-postgres Starting
|
||||
Container gps-denied-onboard-db-1 Starting
|
||||
Container gps-denied-onboard-db-1 Started
|
||||
Container gps-denied-e2e-satellite-provider-postgres Started
|
||||
Container gps-denied-e2e-satellite-provider-postgres Waiting
|
||||
db-1 |
|
||||
db-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
|
||||
db-1 |
|
||||
gps-denied-e2e-satellite-provider-postgres |
|
||||
gps-denied-e2e-satellite-provider-postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization
|
||||
gps-denied-e2e-satellite-provider-postgres |
|
||||
db-1 | 2026-06-20 08:14:12.259 UTC [1] LOG: starting PostgreSQL 16.14 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit
|
||||
db-1 | 2026-06-20 08:14:12.259 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
|
||||
db-1 | 2026-06-20 08:14:12.259 UTC [1] LOG: listening on IPv6 address "::", port 5432
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:14:12.261 UTC [1] LOG: starting PostgreSQL 16.14 (Debian 16.14-1.pgdg13+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
|
||||
db-1 | 2026-06-20 08:14:12.261 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:14:12.261 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:14:12.261 UTC [1] LOG: listening on IPv6 address "::", port 5432
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:14:12.263 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
|
||||
db-1 | 2026-06-20 08:14:12.268 UTC [29] LOG: database system was shut down at 2026-06-19 12:22:55 UTC
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:14:12.269 UTC [29] LOG: database system was shut down at 2026-06-19 12:22:56 UTC
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:14:12.278 UTC [1] LOG: database system is ready to accept connections
|
||||
db-1 | 2026-06-20 08:14:12.278 UTC [1] LOG: database system is ready to accept connections
|
||||
Container gps-denied-e2e-satellite-provider-postgres Healthy
|
||||
Container gps-denied-e2e-satellite-provider Starting
|
||||
Container gps-denied-e2e-satellite-provider Started
|
||||
Container gps-denied-onboard-db-1 Waiting
|
||||
Container gps-denied-e2e-satellite-provider Waiting
|
||||
Container gps-denied-onboard-db-1 Healthy
|
||||
gps-denied-e2e-satellite-provider | 2026-06-20 08:14:18 +00:00 [DBG] Master ConnectionString => Host=satellite-provider-postgres;Port=5432;Database=postgres;Username=postgres;Password=******
|
||||
gps-denied-e2e-satellite-provider | 2026-06-20 08:14:19 +00:00 [INF] Beginning database upgrade
|
||||
gps-denied-e2e-satellite-provider | 2026-06-20 08:14:19 +00:00 [INF] Checking whether journal table exists
|
||||
gps-denied-e2e-satellite-provider | 2026-06-20 08:14:19 +00:00 [INF] Fetching list of already executed scripts.
|
||||
gps-denied-e2e-satellite-provider | 2026-06-20 08:14:19 +00:00 [INF] No new scripts need to be executed - completing.
|
||||
gps-denied-e2e-satellite-provider | [08:14:19 INF] RegionRequestQueue created with capacity 1000
|
||||
gps-denied-e2e-satellite-provider | [08:14:19 INF] Region Processing Service started with 20 parallel workers
|
||||
gps-denied-e2e-satellite-provider | [08:14:19 INF] Route Processing Service started
|
||||
gps-denied-e2e-satellite-provider | [08:14:19 WRN] Overriding HTTP_PORTS '8080' and HTTPS_PORTS ''. Binding to values defined by URLS instead 'https://+:8080'.
|
||||
gps-denied-e2e-satellite-provider | [08:14:19 INF] Now listening on: https://[::]:8080
|
||||
gps-denied-e2e-satellite-provider | [08:14:19 INF] Application started. Press Ctrl+C to shut down.
|
||||
gps-denied-e2e-satellite-provider | [08:14:19 INF] Hosting environment: Development
|
||||
gps-denied-e2e-satellite-provider | [08:14:19 INF] Content root path: /app
|
||||
Container gps-denied-e2e-satellite-provider Healthy
|
||||
Container gps-denied-onboard-e2e-runner-1 Starting
|
||||
Container gps-denied-onboard-e2e-runner-1 Started
|
||||
e2e-runner-1 | ============================= test session starts ==============================
|
||||
e2e-runner-1 | platform linux -- Python 3.10.12, pytest-9.1.1, pluggy-1.6.0 -- /usr/bin/python3.10
|
||||
e2e-runner-1 | cachedir: .pytest_cache
|
||||
e2e-runner-1 | rootdir: /opt
|
||||
e2e-runner-1 | configfile: pyproject.toml
|
||||
e2e-runner-1 | plugins: cov-7.1.0, anyio-4.14.0, asyncio-1.4.0
|
||||
e2e-runner-1 | asyncio: mode=strict, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
|
||||
e2e-runner-1 | collecting ... collected 57 items
|
||||
e2e-runner-1 |
|
||||
e2e-runner-1 | tests/e2e/replay/test_az835_e2e_real_flight.py::test_az840_e2e_real_flight_orchestration SKIPPED [ 1%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_1min.py::test_ac1_exits_0_jsonl_count_match XFAIL [ 3%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_1min.py::test_ac2_jsonl_schema_match PASSED [ 5%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_1min.py::test_ac3_within_100m_80pct_of_ticks XFAIL [ 7%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_1min.py::test_ac4_mode_agnosticism_ast_scan PASSED [ 8%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_1min.py::test_ac4_encoder_byte_equality_via_transport_seam PASSED [ 10%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_1min.py::test_ac5_determinism_two_runs_diff XFAIL [ 12%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_1min.py::test_ac6_pace_realtime_60s_within_5pct XFAIL [ 14%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_1min.py::test_ac6_pace_asap_under_30s XFAIL [ 15%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_1min.py::test_ac7_skip_gate_consistent_with_env_var PASSED [ 17%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_1min.py::test_ac8_operator_workflow SKIPPED [ 19%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_derkachi_real_tlog.py::test_az699_real_flight_validation_emits_verdict_and_report SKIPPED [ 21%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_write_effective_replay_config_overlays_root_dir PASSED [ 22%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_write_effective_replay_config_creates_block_when_absent PASSED [ 24%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_write_effective_replay_config_malformed_yaml_fails PASSED [ 26%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_write_effective_replay_config_non_mapping_top_level_fails PASSED [ 28%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_read_calibration_acquisition_method_returns_field_when_present PASSED [ 29%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_read_calibration_acquisition_method_returns_unknown_on_missing PASSED [ 31%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_read_calibration_acquisition_method_returns_unknown_on_malformed PASSED [ 33%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_missing_tlog_fails_loud PASSED [ 35%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_missing_binary_fails_loud PASSED [ 36%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_replay_nonzero_exit_fails_loud PASSED [ 38%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_replay_timeout_fails_loud PASSED [ 40%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_replay_oserror_fails_loud PASSED [ 42%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_empty_jsonl_fails_loud PASSED [ 43%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_malformed_jsonl_fails_loud PASSED [ 45%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_ground_truth_loader_failure_fails_loud PASSED [ 47%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_happy_path_writes_report PASSED [ 49%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_writes_report_even_on_fail_verdict PASSED [ 50%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_ac9_l2_zero_at_same_point PASSED [ 52%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_ac9_l2_north_one_degree_111km PASSED [ 54%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_ac9_l2_known_pair_kharkiv_kyiv PASSED [ 56%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_ac9_l2_symmetric PASSED [ 57%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_match_percentage_all_within_threshold PASSED [ 59%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_match_percentage_none_within_threshold PASSED [ 61%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_match_percentage_empty_emissions_zero PASSED [ 63%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_match_percentage_empty_ground_truth_raises PASSED [ 64%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_parse_jsonl_round_trip PASSED [ 66%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_parse_jsonl_skips_trailing_blank PASSED [ 68%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_parse_jsonl_invalid_line_raises PASSED [ 70%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_capturing_transport_records_writes PASSED [ 71%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_capturing_transport_close_then_write_raises PASSED [ 73%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_helpers.py::test_capturing_transport_implements_protocol PASSED [ 75%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_populate_c6_from_route_returns_populated_cache PASSED [ 77%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_populate_c6_from_route_passes_sector_class_to_downloader PASSED [ 78%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_route_validation_error_propagates_unchanged PASSED [ 80%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_route_terminal_failure_propagates_unchanged PASSED [ 82%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_route_transient_error_retries_then_succeeds PASSED [ 84%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_route_transient_error_exhausted_propagates_last_attempt PASSED [ 85%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_descriptor_index_factory_index_unavailable_propagates PASSED [ 87%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_cleanup_removes_partial_sidecar_files_on_failure PASSED [ 89%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_cleanup_preserves_pre_existing_warm_cache PASSED [ 91%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_batcher_failure_propagates_and_cleans_up PASSED [ 92%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_driver.py::test_downloader_failure_propagates_and_cleans_up PASSED [ 94%]
|
||||
e2e-runner-1 | tests/e2e/replay/test_operator_pre_flight_integration.py::test_operator_pre_flight_setup_produces_populated_cache SKIPPED [ 96%]
|
||||
e2e-runner-1 | tests/e2e/satellite_provider/test_smoke.py::test_smoke_satellite_provider_inventory_contract FAILED [ 98%]
|
||||
e2e-runner-1 | tests/e2e/satellite_provider/test_smoke.py::test_smoke_c11_download_via_http_pipeline FAILED [100%]
|
||||
e2e-runner-1 |
|
||||
e2e-runner-1 | =================================== FAILURES ===================================
|
||||
e2e-runner-1 | _______________ test_smoke_satellite_provider_inventory_contract _______________
|
||||
e2e-runner-1 | tests/e2e/satellite_provider/test_smoke.py:189: in test_smoke_satellite_provider_inventory_contract
|
||||
e2e-runner-1 | assert response.status_code == 200, (
|
||||
e2e-runner-1 | E AssertionError: satellite-provider inventory POST returned 404: ''
|
||||
e2e-runner-1 | E assert 404 == 200
|
||||
e2e-runner-1 | E + where 404 = <Response [404 Not Found]>.status_code
|
||||
e2e-runner-1 | ----------------------------- Captured stdout call -----------------------------
|
||||
e2e-runner-1 | {"ts":"2026-06-20T08:15:44.848668Z","level":"INFO","component":"httpx","frame_id":null,"kind":"log.diag","msg":"HTTP Request: POST https://satellite-provider:8080/api/satellite/tiles/inventory \"HTTP/1.1 404 Not Found\"","kv":{},"exc":null}
|
||||
e2e-runner-1 | ------------------------------ Captured log call -------------------------------
|
||||
e2e-runner-1 | INFO httpx:_client.py:1025 HTTP Request: POST https://satellite-provider:8080/api/satellite/tiles/inventory "HTTP/1.1 404 Not Found"
|
||||
e2e-runner-1 | __________________ test_smoke_c11_download_via_http_pipeline ___________________
|
||||
e2e-runner-1 | tests/e2e/satellite_provider/test_smoke.py:301: in test_smoke_c11_download_via_http_pipeline
|
||||
e2e-runner-1 | report = downloader.download_tiles_for_area(request)
|
||||
e2e-runner-1 | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:543: in download_tiles_for_area
|
||||
e2e-runner-1 | summaries = self._enumerate_remote(request)
|
||||
e2e-runner-1 | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:636: in _enumerate_remote
|
||||
e2e-runner-1 | self._do_enumerate(
|
||||
e2e-runner-1 | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:678: in _do_enumerate
|
||||
e2e-runner-1 | summaries.extend(self._fetch_inventory_chunk(chunk))
|
||||
e2e-runner-1 | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:683: in _fetch_inventory_chunk
|
||||
e2e-runner-1 | response = self._send_post(
|
||||
e2e-runner-1 | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:878: in _send_post
|
||||
e2e-runner-1 | return self._send_request("POST", url, params=None, json_body=json_body, session=session)
|
||||
e2e-runner-1 | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:963: in _send_request
|
||||
e2e-runner-1 | raise SatelliteProviderError(
|
||||
e2e-runner-1 | E gps_denied_onboard.components.c11_tile_manager.errors.SatelliteProviderError: satellite-provider returned unexpected status 404 (expected 200)
|
||||
e2e-runner-1 | ----------------------------- Captured stdout call -----------------------------
|
||||
e2e-runner-1 | {"ts":"2026-06-20T08:15:44.866897Z","level":"INFO","component":"c11_tile_manager.tile_downloader","frame_id":null,"kind":"c11.download.session.start","msg":"Pre-flight tile download session started","kv":{"flight_id":"9346cdb7-a5b4-4d87-a47c-370415c297dd","request_hash":"46a59716a231eeab","bbox":[50.099,36.099,50.101,36.101],"zoom_levels":[15],"sector_class":"stable_rear","resume_from_journal":false,"tiles_already_completed":0},"exc":null}
|
||||
e2e-runner-1 | {"ts":"2026-06-20T08:15:44.883304Z","level":"INFO","component":"httpx","frame_id":null,"kind":"log.diag","msg":"HTTP Request: POST https://satellite-provider:8080/api/satellite/tiles/inventory \"HTTP/1.1 404 Not Found\"","kv":{},"exc":null}
|
||||
e2e-runner-1 | {"ts":"2026-06-20T08:15:44.884249Z","level":"ERROR","component":"c11_tile_manager.tile_downloader","frame_id":null,"kind":"c11.download.provider.failed","msg":"Download provider failed","kv":{"reason":"unexpected_status","http_status":404,"detail":"non-200","auth_header":"Bearer ***"},"exc":null}
|
||||
e2e-runner-1 | {"ts":"2026-06-20T08:15:44.888017Z","level":"INFO","component":"c11_tile_manager.tile_downloader","frame_id":null,"kind":"c11.download.session.end","msg":"Pre-flight tile download session ended","kv":{"flight_id":"9346cdb7-a5b4-4d87-a47c-370415c297dd","request_hash":"46a59716a231eeab","outcome":"failure","tiles_requested":0,"tiles_downloaded":0,"tiles_rejected_resolution":0,"tiles_rejected_freshness":0,"tiles_downgraded":0,"retry_count":0},"exc":null}
|
||||
e2e-runner-1 | ------------------------------ Captured log call -------------------------------
|
||||
e2e-runner-1 | INFO test_az777_smoke:tile_downloader.py:519 Pre-flight tile download session started
|
||||
e2e-runner-1 | INFO httpx:_client.py:1025 HTTP Request: POST https://satellite-provider:8080/api/satellite/tiles/inventory "HTTP/1.1 404 Not Found"
|
||||
e2e-runner-1 | ERROR test_az777_smoke:tile_downloader.py:994 Download provider failed
|
||||
e2e-runner-1 | INFO test_az777_smoke:tile_downloader.py:578 Pre-flight tile download session ended
|
||||
e2e-runner-1 | =============================== warnings summary ===============================
|
||||
e2e-runner-1 | ../usr/local/lib/python3.10/dist-packages/faiss/loader.py:44
|
||||
e2e-runner-1 | /usr/local/lib/python3.10/dist-packages/faiss/loader.py:44: DeprecationWarning:
|
||||
e2e-runner-1 |
|
||||
e2e-runner-1 | `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
|
||||
e2e-runner-1 | of the deprecation of `distutils` itself. It will be removed for
|
||||
e2e-runner-1 | Python >= 3.12. For older Python versions it will remain present.
|
||||
e2e-runner-1 | It is recommended to use `setuptools < 60.0` for those Python versions.
|
||||
e2e-runner-1 | For more details, see:
|
||||
e2e-runner-1 | https://numpy.org/devdocs/reference/distutils_status_migration.html
|
||||
e2e-runner-1 |
|
||||
e2e-runner-1 |
|
||||
e2e-runner-1 | import numpy.distutils.cpuinfo
|
||||
e2e-runner-1 |
|
||||
e2e-runner-1 | -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
e2e-runner-1 | =========================== short test summary info ============================
|
||||
e2e-runner-1 | SKIPPED [1] tests/e2e/replay/test_az835_e2e_real_flight.py:127: AZ-839 operator_pre_flight_setup: descriptor_dim resolver only supports c2_vpr.strategy='net_vlad'; got '<missing>' on backbone 'net_vlad'. See AZ-839 spec § Out of scope.
|
||||
e2e-runner-1 | SKIPPED [1] tests/e2e/replay/test_derkachi_1min.py:479: AC-8 (operator workflow rehearsal) blocked on the full D-PROJ-2 mock-suite-sat-service implementation — current tests/fixtures/mock-suite-sat-service/ is a bootstrap stub with only GET /healthz. Unskips when the mock implements tile-fetch + index-build endpoints.
|
||||
e2e-runner-1 | SKIPPED [1] tests/e2e/replay/test_derkachi_real_tlog.py:202: real tlog missing: /opt/_docs/00_problem/input_data/flight_derkachi/derkachi.tlog
|
||||
e2e-runner-1 | SKIPPED [1] tests/e2e/replay/test_operator_pre_flight_integration.py:22: AZ-839 operator_pre_flight_setup: descriptor_dim resolver only supports c2_vpr.strategy='net_vlad'; got '<missing>' on backbone 'net_vlad'. See AZ-839 spec § Out of scope.
|
||||
e2e-runner-1 | XFAIL tests/e2e/replay/test_derkachi_1min.py::test_ac1_exits_0_jsonl_count_match - AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands.
|
||||
e2e-runner-1 | XFAIL tests/e2e/replay/test_derkachi_1min.py::test_ac3_within_100m_80pct_of_ticks - AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands.
|
||||
e2e-runner-1 | XFAIL tests/e2e/replay/test_derkachi_1min.py::test_ac5_determinism_two_runs_diff - AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands.
|
||||
e2e-runner-1 | XFAIL tests/e2e/replay/test_derkachi_1min.py::test_ac6_pace_realtime_60s_within_5pct - AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands.
|
||||
e2e-runner-1 | XFAIL tests/e2e/replay/test_derkachi_1min.py::test_ac6_pace_asap_under_30s - AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands.
|
||||
e2e-runner-1 | FAILED tests/e2e/satellite_provider/test_smoke.py::test_smoke_satellite_provider_inventory_contract
|
||||
e2e-runner-1 | FAILED tests/e2e/satellite_provider/test_smoke.py::test_smoke_c11_download_via_http_pipeline
|
||||
e2e-runner-1 | === 2 failed, 46 passed, 4 skipped, 5 xfailed, 1 warning in 79.92s (0:01:19) ===
|
||||
[Ke2e-runner-1 exited with code 1
|
||||
Compose Stopping Aborting on container exit...
|
||||
Container gps-denied-onboard-e2e-runner-1 Stopping
|
||||
Container gps-denied-onboard-e2e-runner-1 Stopped
|
||||
Container gps-denied-onboard-db-1 Stopping
|
||||
Container gps-denied-e2e-satellite-provider Stopping
|
||||
gps-denied-e2e-satellite-provider | [08:15:46 INF] Application is shutting down...
|
||||
db-1 | 2026-06-20 08:15:46.891 UTC [1] LOG: received fast shutdown request
|
||||
db-1 | 2026-06-20 08:15:46.892 UTC [1] LOG: aborting any active transactions
|
||||
db-1 | 2026-06-20 08:15:46.897 UTC [1] LOG: background worker "logical replication launcher" (PID 32) exited with exit code 1
|
||||
db-1 | 2026-06-20 08:15:46.897 UTC [27] LOG: shutting down
|
||||
db-1 | 2026-06-20 08:15:46.898 UTC [27] LOG: checkpoint starting: shutdown immediate
|
||||
db-1 | 2026-06-20 08:15:46.904 UTC [27] LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.001 s, total=0.008 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB; lsn=0/1A00478, redo lsn=0/1A00478
|
||||
gps-denied-e2e-satellite-provider | [08:15:46 INF] Region Processing Service stopped
|
||||
db-1 | 2026-06-20 08:15:46.919 UTC [1] LOG: database system is shut down
|
||||
Container gps-denied-e2e-satellite-provider Stopped
|
||||
Container gps-denied-e2e-satellite-provider-postgres Stopping
|
||||
[Kgps-denied-e2e-satellite-provider exited with code 0
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:15:47.287 UTC [1] LOG: received fast shutdown request
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:15:47.288 UTC [1] LOG: aborting any active transactions
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:15:47.298 UTC [1] LOG: background worker "logical replication launcher" (PID 32) exited with exit code 1
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:15:47.298 UTC [27] LOG: shutting down
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:15:47.300 UTC [27] LOG: checkpoint starting: shutdown immediate
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:15:47.306 UTC [27] LOG: checkpoint complete: wrote 2 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.003 s, sync=0.001 s, total=0.008 s; sync files=3, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB; lsn=0/11341D40, redo lsn=0/11341D40
|
||||
gps-denied-e2e-satellite-provider-postgres | 2026-06-20 08:15:47.318 UTC [1] LOG: database system is shut down
|
||||
Container gps-denied-onboard-db-1 Stopped
|
||||
[Kdb-1 exited with code 0
|
||||
Container gps-denied-e2e-satellite-provider-postgres Stopped
|
||||
[Kgps-denied-e2e-satellite-provider-postgres exited with code 0
|
||||
|
||||
@@ -634,3 +634,114 @@ Pre-launch fix in commit `a15a062 [AZ-844] Exclude satellite-provider runtime di
|
||||
|
||||
Auto-chain → Step 12 (Test-Spec Sync) on next `/autodev` invocation.
|
||||
|
||||
---
|
||||
|
||||
## Cycle 4 (2026-06-19)
|
||||
|
||||
Scope of cycle-4 implementation (5 batches, `batch_01`..`batch_05_cycle4_report.md`):
|
||||
|
||||
- Wave-1 housekeeping: AZ-899 architecture compliance baseline
|
||||
- Replay-input redesign: AZ-894 CSV adapter, AZ-896 tlog route, AZ-895 auto-sync deprecation, AZ-842 protocol docs
|
||||
- AZ-963: Derkachi 60s smoke regressions — Option D+E (xfail + XPASS root-cause fix)
|
||||
|
||||
### Local unit suite
|
||||
|
||||
```
|
||||
.venv/bin/python -m pytest tests/unit/ -v --tb=short
|
||||
====== 2307 passed, 84 skipped in 48.68s =======
|
||||
```
|
||||
|
||||
0 failed. 84 skips classified as legitimate on a macOS dev host:
|
||||
|
||||
| Reason | Count | Verdict |
|
||||
|--------|------:|---------|
|
||||
| Requires Docker compose services (postgres / mock-sat) | 57 | legitimate locally — covered on Jetson e2e lane |
|
||||
| Tier-2-only / Jetson hardware (NVML, L4T) | 1 | legitimate |
|
||||
| TensorRT / onnxruntime not installed | 7 | legitimate (Tier-2 Jetson only) |
|
||||
| Derkachi reference tlog gitignored / absent | 2 | legitimate |
|
||||
| AC-1 RSS measurement deferred to e2e | 1 | legitimate |
|
||||
| `actionlint` not on PATH (CI-only) | 1 | legitimate |
|
||||
| Empty parametrize (`runtime`) | 1 | legitimate |
|
||||
| Other env-conditional | 14 | legitimate |
|
||||
|
||||
Note: pytest segfaults inside the Cursor sandbox (numpy import during collection); runs cleanly outside sandbox with project `.venv`.
|
||||
|
||||
### Jetson e2e
|
||||
|
||||
Ran 2026-06-19 via `PATH=".venv/bin:$PATH" JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh`.
|
||||
Log: `_docs/03_implementation/jetson_runs/2026-06-19_cycle4_run.txt` (wall clock ~9 min incl. rsync + build).
|
||||
|
||||
```
|
||||
====== 8 failed, 45 passed, 4 skipped, 1 warning in 17.37s =======
|
||||
```
|
||||
|
||||
#### Failure root causes
|
||||
|
||||
| # | Test(s) | Root cause | Category |
|
||||
|---|---------|------------|----------|
|
||||
| 1 | `test_ac1`..`test_ac6` (6×) | `flight_derkachi.mp4` is a 134-byte Git LFS pointer on disk; rsync excludes LFS blobs → `moov atom not found` / `VideoCapture could not open` | **missing fixture/data** |
|
||||
| 2 | `test_smoke_satellite_provider_*` (2×) | `POST …/api/satellite/tiles/inventory` → HTTP 404 from satellite-provider container | **environment / API drift** |
|
||||
|
||||
#### AZ-963 gap
|
||||
|
||||
`batch_05_cycle4_report.md` documents `@pytest.mark.xfail` on five Derkachi tests, but the working tree has **zero** `xfail` markers in `test_derkachi_1min.py` (grep confirms). Jira AZ-963 is Done; the xfail triage code was never landed in this checkout.
|
||||
|
||||
#### Skip classification (4)
|
||||
|
||||
All legitimate: AZ-839 descriptor_dim gate (2×), AC-8 mock-sat stub (1×), real tlog absent (1×).
|
||||
|
||||
### Step 11 status: **blocked (cycle 4)** — unit gate PASS; Jetson e2e 2 FAIL (stale satprov image); AZ-963 xfail landed
|
||||
|
||||
---
|
||||
|
||||
## Cycle 4 rerun (2026-06-20)
|
||||
|
||||
Resumed Step 11 after AZ-963 xfail markers were missing from the tree
|
||||
(batch_05 report documented them but they were never committed).
|
||||
|
||||
### Fixes applied this session
|
||||
|
||||
| Change | Purpose |
|
||||
|--------|---------|
|
||||
| `@pytest.mark.xfail` on AC-1/3/5/6 (AZ-963) in `test_derkachi_1min.py` | Honest gating for open-loop ESKF divergence without C6 cache |
|
||||
| LFS preflight in `scripts/run-tests-jetson.sh` | Fail fast when `flight_derkachi.mp4` is a 134-byte pointer |
|
||||
| `run-tests-jetson.sh` builds **e2e-runner only** | Parent-suite `protoc` segfaults on arm64 inside dotnet-sdk (AZ-977 gRPC proto); cached `satellite-provider:dev` image used as-is |
|
||||
|
||||
### Local unit suite
|
||||
|
||||
```
|
||||
.venv/bin/python -m pytest tests/unit/ -q --tb=no
|
||||
2307 passed, 84 skipped in 43.72s
|
||||
```
|
||||
|
||||
### Jetson e2e (rerun)
|
||||
|
||||
```
|
||||
PATH=".venv/bin:$PATH" JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh
|
||||
```
|
||||
|
||||
Log: `_docs/03_implementation/jetson_runs/2026-06-20_cycle4_rerun.txt`
|
||||
|
||||
```
|
||||
====== 2 failed, 46 passed, 4 skipped, 5 xfailed, 1 warning in 79.92s =======
|
||||
```
|
||||
|
||||
| Outcome | Count | Notes |
|
||||
|---------|------:|-------|
|
||||
| PASSED | 46 | incl. `test_ac2_jsonl_schema_match` (mp4 smudged; was 6× FAIL on 2026-06-19) |
|
||||
| XFAIL | 5 | AZ-963 open-loop ESKF (expected) |
|
||||
| SKIPPED | 4 | AC-8 mock-sat, AZ-839 backbone gate, real tlog absent |
|
||||
| FAILED | 2 | `test_smoke_satellite_provider_*` — HTTP 404 on `POST /api/satellite/tiles/inventory` |
|
||||
|
||||
#### Remaining failure root cause
|
||||
|
||||
The cached `gps-denied-onboard/satellite-provider:dev` image on the Jetson
|
||||
predates the AZ-505 inventory endpoint (or is otherwise stale). Rebuild is
|
||||
blocked: current parent-suite source adds `tile_provision.proto` (AZ-977) and
|
||||
`protoc` exits 139 on arm64 during `docker compose build satellite-provider`.
|
||||
|
||||
Resolution path: fix arm64 gRPC proto build in `../satellite-provider` (AZ-977),
|
||||
then re-enable `build satellite-provider` in `run-tests-jetson.sh`.
|
||||
|
||||
### Step 11 status: **in_progress (cycle 4)** — unit PASS; Jetson 2 FAIL (satprov image stale / AZ-977 build blocker)
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
+11
-8
@@ -1,11 +1,14 @@
|
||||
Testing strategy without real flight.
|
||||
# Demo replay validation (operator workflow — F11)
|
||||
|
||||
upload tlog file
|
||||
upload video synced with tlog
|
||||
Upload a flight video and ArduPilot tlog from the same sortie. The suite UI shows two timeline bars: video above, tlog IMU activity below. Drag the video bar to align with takeoff on the tlog, refine the match, then run the demo. The system:
|
||||
|
||||
1. Extracts IMU and GPS from the tlog.
|
||||
2. Aligns video to tlog using your coarse placement plus backend refinement.
|
||||
3. Exports a canonical aligned CSV (single time base for replay).
|
||||
4. Seeds satellite corridor tiles from the tlog GPS route.
|
||||
5. Runs the same GPS-denied pipeline as live flight against the video.
|
||||
6. Returns estimated GPS fixes, a map, and a PASS/FAIL accuracy verdict.
|
||||
|
||||
system should:
|
||||
1. extract timestamps, imu and gps from the tlog file.
|
||||
2. usually video and tlog aren't synchronized. So system should synchronize them by itself.
|
||||
Usual test is done on the quadcopters, so usually it starts from the drone on the ground and ends with the drone on the ground. These sessions are clearly visible in the chart IMU data of the tlog file. So, system can check the duration of the video and events in IMU chart in tlog. Then it can analyze by IMU the moment of actual take off and sync them
|
||||
3. then make SITL and provide IMU and frames to the gps denied onboard system
|
||||
Advanced: upload a pre-aligned `(video, CSV)` pair to skip alignment (AZ-959).
|
||||
|
||||
Live flight (F3) is unchanged: IMU and frames from the aircraft in real time.
|
||||
|
||||
@@ -150,6 +150,16 @@ echo "[run-tests-jetson] compose file: ${COMPOSE_FILE}"
|
||||
echo "[run-tests-jetson] ensure-dev-cert (local)"
|
||||
bash "${SCRIPT_DIR}/ensure-dev-cert.sh"
|
||||
|
||||
DERKACHI_MP4="${REPO_ROOT}/_docs/00_problem/input_data/flight_derkachi/flight_derkachi.mp4"
|
||||
if [[ -f "${DERKACHI_MP4}" ]]; then
|
||||
mp4_bytes=$(wc -c < "${DERKACHI_MP4}" | tr -d ' ')
|
||||
if [[ "${mp4_bytes}" -lt 1000000 ]]; then
|
||||
echo "[run-tests-jetson] ERROR: ${DERKACHI_MP4} is ${mp4_bytes} bytes — looks like a Git LFS pointer." >&2
|
||||
echo "[run-tests-jetson] Run 'git lfs pull' (or copy the real mp4) before rsyncing to Jetson." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Step 1: sync source
|
||||
|
||||
@@ -209,12 +219,14 @@ rsync -az --delete --stats \
|
||||
"${SATPROV_DIR}/" "${SSH_ALIAS}:${REMOTE_SATPROV_DIR}/"
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Step 2: build the e2e-runner + satellite-provider images on the Jetson
|
||||
# Step 2: build the e2e-runner image on the Jetson
|
||||
|
||||
# Both images MUST be built on the Jetson — Dockerfile.jetson needs Tegra
|
||||
# libs, and the .NET dotnet-sdk image is multi-arch but only the arm64
|
||||
# variant is on the Orin.
|
||||
echo "[run-tests-jetson] docker compose build (on Jetson)"
|
||||
# Dockerfile.jetson needs Tegra libs, so e2e-runner MUST be built on-device.
|
||||
# satellite-provider is NOT rebuilt here: the parent-suite image now compiles
|
||||
# gRPC protos (AZ-977) and protoc segfaults on arm64 inside dotnet-sdk
|
||||
# (exit 139). The cached gps-denied-onboard/satellite-provider:dev image is
|
||||
# used as-is until AZ-977 ships an arm64-safe build path.
|
||||
echo "[run-tests-jetson] docker compose build e2e-runner (on Jetson)"
|
||||
# The compose `include:` resolves the upstream env vars from the shell, so
|
||||
# pass JWT_SECRET / JWT_ISSUER / JWT_AUDIENCE / GOOGLE_MAPS_API_KEY through
|
||||
# the heredoc as explicit exports. (We can't rely on `ssh -o SendEnv` —
|
||||
@@ -228,7 +240,7 @@ export JWT_AUDIENCE=${JWT_AUDIENCE_Q}
|
||||
export GOOGLE_MAPS_API_KEY=${GOOGLE_MAPS_API_KEY_Q}
|
||||
export SATELLITE_PROVIDER_API_KEY=${SATELLITE_PROVIDER_API_KEY_Q}
|
||||
cd "${REMOTE_DIR}"
|
||||
docker compose -f "${COMPOSE_FILE}" build e2e-runner satellite-provider
|
||||
docker compose -f "${COMPOSE_FILE}" build e2e-runner
|
||||
EOF
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
@@ -1,40 +1,3 @@
|
||||
"""C11 ``SatelliteProviderRouteClient`` (AZ-838 / Epic AZ-835 C2).
|
||||
|
||||
Operator-side HTTP client for the parent-suite Route API. Takes a
|
||||
:class:`gps_denied_onboard._types.route.RouteSpec` (produced
|
||||
by AZ-836 / C1) and onboards it with ``satellite-provider``:
|
||||
|
||||
1. **Pre-emptive validation** mirrors the AZ-809
|
||||
``CreateRouteRequestValidator`` rules so obviously-bad input fails
|
||||
before the HTTP POST.
|
||||
2. **POST** ``/api/satellite/route`` with ``requestMaps=true`` and
|
||||
``createTilesZip=false``. Wire shape derived from the live DTOs in
|
||||
``../satellite-provider/SatelliteProvider.Common/DTO/{CreateRouteRequest,RoutePoint,GeoPoint}.cs``.
|
||||
3. **Poll** ``GET /api/satellite/route/{id}`` until ``mapsReady=true``
|
||||
OR a terminal failure status; respects
|
||||
:attr:`SatelliteProviderRouteClient.poll_max_attempts` and
|
||||
:attr:`SatelliteProviderRouteClient.poll_interval_s`.
|
||||
4. **Inventory verify** via ``POST /api/satellite/tiles/inventory`` —
|
||||
enumerates the route's tile coverage locally from the
|
||||
``RouteSpec`` waypoints + ``regionSizeMeters`` and counts the
|
||||
``present=true`` entries returned by the server (lower bound on
|
||||
the actual coverage, since the server interpolates intermediate
|
||||
waypoints — documented in the contract).
|
||||
5. **Return** :class:`RouteSeedResult` with provenance fields
|
||||
(route id, terminal status, maps_ready flag, tile count, elapsed
|
||||
time, sha256 of the submitted payload).
|
||||
|
||||
The error hierarchy is rooted at :class:`SatelliteProviderRouteError`
|
||||
(in :mod:`.errors`), independent of :class:`TileManagerError` because
|
||||
the Route API is a corridor-onboarding flow, not a per-tile transfer.
|
||||
|
||||
Lives under ``c11_tile_manager`` because the existing C11 plumbing
|
||||
(JWT auth, TLS-insecure flag for self-signed dev certs) is shared and
|
||||
because C11 is already gated ``BUILD_C11_TILE_MANAGER=ON`` for the
|
||||
operator-orchestrator binary (and OFF for airborne) — same audience
|
||||
as the Route API.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
@@ -60,20 +23,11 @@ __all__ = [
|
||||
"SatelliteProviderRouteClient",
|
||||
]
|
||||
|
||||
|
||||
# AZ-838 wire constants — paths confirmed against
|
||||
# `../satellite-provider/SatelliteProvider.Api/Program.cs:266`+ on
|
||||
# 2026-05-22 (route create + route status) and against
|
||||
# `tile_downloader.py::_INVENTORY_PATH` for the inventory verify step.
|
||||
_ROUTE_CREATE_PATH = "/api/satellite/route"
|
||||
_ROUTE_STATUS_PATH_TPL = "/api/satellite/route/{id}"
|
||||
_INVENTORY_PATH = "/api/satellite/tiles/inventory"
|
||||
_INVENTORY_MAX_ENTRIES_PER_REQUEST = 5000
|
||||
|
||||
# AZ-809 validator bounds (mirrored from
|
||||
# `SatelliteProvider.Api/Validators/CreateRouteRequestValidator.cs`).
|
||||
# Keep these in sync with that file — the client pre-emptively
|
||||
# enforces them so obviously-bad input fails before the HTTP POST.
|
||||
_VALIDATOR_NAME_MAX_LEN: int = 200
|
||||
_VALIDATOR_DESCRIPTION_MAX_LEN: int = 1000
|
||||
_VALIDATOR_REGION_SIZE_MIN_M: float = 100.0
|
||||
@@ -83,16 +37,8 @@ _VALIDATOR_ZOOM_MAX: int = 22
|
||||
_VALIDATOR_POINTS_MIN: int = 2
|
||||
_VALIDATOR_POINTS_MAX: int = 500
|
||||
|
||||
# Mirror of the parent-suite tile-size math used by C11
|
||||
# (`tile_downloader._EARTH_EQUATORIAL_CIRCUMFERENCE_M` /
|
||||
# `_TILE_SIZE_PIXELS`). Re-stated here so the inventory-coverage
|
||||
# enumeration does not depend on a private constant from the
|
||||
# downloader module.
|
||||
_EARTH_EQUATORIAL_CIRCUMFERENCE_M: float = 40_075_016.686
|
||||
|
||||
# Terminal status strings the parent suite reports via
|
||||
# `GET /api/satellite/route/{id}`. Mirrors `seed_region.py`'s set so
|
||||
# both Region and Route flows agree on terminal semantics.
|
||||
_TERMINAL_STATUSES: frozenset[str] = frozenset(
|
||||
{"completed", "failed", "error", "done", "succeeded", "rejected"}
|
||||
)
|
||||
@@ -116,33 +62,6 @@ _LOG_KIND_VALIDATION_FAIL = "c11.route.validation_failed"
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RouteSeedResult:
|
||||
"""Outcome of one :meth:`SatelliteProviderRouteClient.seed_route` call.
|
||||
|
||||
Attributes:
|
||||
route_id: The ``id`` field POSTed in the request — kept here
|
||||
so the caller can re-query ``GET /api/satellite/route/{id}``
|
||||
without re-deriving it.
|
||||
terminal_status: The server's last observed status string
|
||||
(one of the values in :data:`_TERMINAL_STATUSES`, lower-
|
||||
cased). On a healthy run this is typically ``completed``.
|
||||
maps_ready: ``True`` if the server reported ``mapsReady=true``
|
||||
within the poll budget. ``False`` only on terminal
|
||||
failure paths that do NOT raise (currently impossible —
|
||||
terminal failures always raise; the field is here for
|
||||
forward compatibility if the server adds a "ready
|
||||
without maps" state).
|
||||
tile_count: Number of (z, x, y) entries the inventory call
|
||||
reported as ``present=true``. Lower bound on the actual
|
||||
tile coverage produced by the server, since the local
|
||||
enumeration does NOT account for the server-side
|
||||
~200 m intermediate-point interpolation documented in
|
||||
``../satellite-provider/_docs/02_document/contracts/api/route-creation.md``.
|
||||
elapsed_ms: Wall-clock milliseconds from the start of the
|
||||
POST submission to the completion of the inventory verify.
|
||||
submitted_payload_sha256: SHA-256 hex digest of the JSON body
|
||||
POSTed to ``/api/satellite/route`` (provenance / audit).
|
||||
"""
|
||||
|
||||
route_id: uuid.UUID
|
||||
terminal_status: str
|
||||
maps_ready: bool
|
||||
@@ -152,21 +71,6 @@ class RouteSeedResult:
|
||||
|
||||
|
||||
class SatelliteProviderRouteClient:
|
||||
"""HTTP client for the parent-suite Route API (AZ-838).
|
||||
|
||||
Constructor parameters mirror the operator-side ergonomics
|
||||
(``base_url`` + ``jwt`` + ``tls_insecure`` for self-signed dev
|
||||
certs), matching the existing ``seed_region.py`` flag surface so
|
||||
operators can use a single ``.env.test`` file.
|
||||
|
||||
For tests, an optional ``http_client`` may be injected — the
|
||||
standard ``httpx.MockTransport`` pattern from
|
||||
``test_tile_downloader.py`` works directly. When ``http_client``
|
||||
is ``None`` (production / CLI use), the client owns its own
|
||||
short-lived :class:`httpx.Client` per ``seed_route`` call so the
|
||||
caller does not need to manage connection lifetime.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str,
|
||||
@@ -211,10 +115,6 @@ class SatelliteProviderRouteClient:
|
||||
"gps_denied_onboard.components.c11_tile_manager.route_client"
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Public API
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def seed_route(
|
||||
self,
|
||||
spec: RouteSpec,
|
||||
@@ -224,38 +124,6 @@ class SatelliteProviderRouteClient:
|
||||
zoom_level: int = 18,
|
||||
description: str | None = None,
|
||||
) -> RouteSeedResult:
|
||||
"""Onboard ``spec`` with the parent-suite Route API.
|
||||
|
||||
Args:
|
||||
spec: The :class:`RouteSpec` produced by AZ-836's
|
||||
``extract_route_from_tlog``.
|
||||
name: Optional human-readable name. When ``None``, derived
|
||||
from the spec's ``source_tlog`` stem + a short hash of
|
||||
the waypoints (deterministic for the same RouteSpec).
|
||||
region_size_meters: Per-waypoint coverage radius in
|
||||
metres. When ``None``, falls back to
|
||||
:attr:`RouteSpec.suggested_region_size_meters`. The
|
||||
combined value MUST be in the AZ-809 validator range
|
||||
``[100, 10000]``.
|
||||
zoom_level: Web-Mercator zoom for the route. Defaults to
|
||||
18 — matches ``seed_region.py``'s ``zoom_levels``
|
||||
default. AZ-809 validator accepts ``[0, 22]``.
|
||||
description: Optional free-text description (max 1000
|
||||
chars per AZ-809).
|
||||
|
||||
Returns:
|
||||
:class:`RouteSeedResult` on success.
|
||||
|
||||
Raises:
|
||||
RouteValidationError: Pre-emptive validation rejected the
|
||||
inputs OR the server returned 4xx + RFC 7807.
|
||||
RouteTransientError: 5xx / network / timeout. The
|
||||
underlying ``httpx`` exception is on ``__cause__``.
|
||||
RouteTerminalFailureError: ``mapsReady=true`` was never
|
||||
reached within the poll budget OR the server reported
|
||||
a terminal failure status.
|
||||
"""
|
||||
|
||||
effective_region_size = float(
|
||||
region_size_meters
|
||||
if region_size_meters is not None
|
||||
@@ -273,8 +141,6 @@ class SatelliteProviderRouteClient:
|
||||
description=description,
|
||||
)
|
||||
|
||||
# Pre-emptive validation runs against the assembled body so
|
||||
# the rules apply to whatever the server is about to see.
|
||||
self._preemptive_validate(request_body)
|
||||
|
||||
payload_bytes = _canonical_json_bytes(request_body)
|
||||
@@ -311,12 +177,6 @@ class SatelliteProviderRouteClient:
|
||||
zoom_level: int = 18,
|
||||
description: str | None = None,
|
||||
) -> tuple[dict[str, Any], str]:
|
||||
"""Return the planned request body + its sha256 without HTTP.
|
||||
|
||||
Powers ``seed_route.py --dry-run`` (AC-7). Runs the same
|
||||
pre-emptive validation as :meth:`seed_route`, so a dry-run
|
||||
surfaces validation errors the same way a live run would.
|
||||
"""
|
||||
|
||||
effective_region_size = float(
|
||||
region_size_meters
|
||||
@@ -337,10 +197,6 @@ class SatelliteProviderRouteClient:
|
||||
sha256 = hashlib.sha256(_canonical_json_bytes(body)).hexdigest()
|
||||
return body, sha256
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Internal pipeline
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _run(
|
||||
self,
|
||||
*,
|
||||
@@ -613,10 +469,6 @@ class SatelliteProviderRouteClient:
|
||||
)
|
||||
return present_count
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Validation + payload assembly
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _build_request_body(
|
||||
self,
|
||||
*,
|
||||
@@ -627,13 +479,6 @@ class SatelliteProviderRouteClient:
|
||||
zoom_level: int,
|
||||
description: str | None,
|
||||
) -> dict[str, Any]:
|
||||
"""Assemble the wire body matching CreateRouteRequest.cs / RoutePoint.cs.
|
||||
|
||||
Per the AZ-809 batch-03 review F3, ``RoutePoint`` uses
|
||||
``[JsonPropertyName("lat"|"lon")]`` so we serialize ``lat`` /
|
||||
``lon`` (NOT ``latitude`` / ``longitude``).
|
||||
"""
|
||||
|
||||
body: dict[str, Any] = {
|
||||
"id": str(route_id),
|
||||
"name": name,
|
||||
|
||||
+18
-14
@@ -116,7 +116,7 @@ the attribution into the test fixture's metadata; do not remove it.
|
||||
| `flight_derkachi.mp4` | available | `_docs/00_problem/input_data/flight_derkachi/` |
|
||||
| `data_imu.csv` | available | same dir; 4900 rows at 10 Hz over 489.9 s |
|
||||
| Synthetic tlog | generated at fixture time | `_tlog_synth.py` reproduces a `pymavlink` `.tlog` from the CSV (the original tlog is not in-repo; the CSV was its export) |
|
||||
| Camera calibration | placeholder (`tests/fixtures/calibration/adti26.json`) | The real Topotek KHP20S30 intrinsics are unknown per `camera_info.md`. AC-3 is `xfail`ed until a real calibration ships. |
|
||||
| Camera calibration | placeholder (`tests/fixtures/calibration/adti26.json`) | The real Topotek KHP20S30 intrinsics are unknown per `camera_info.md`. AC-3 accuracy depends on this. |
|
||||
| Operator pre-flight rehearsal | blocked | `tests/fixtures/mock-suite-sat-service/` is a bootstrap stub (only `GET /healthz`); AC-8 skips until the full D-PROJ-2 contract lands. |
|
||||
|
||||
## Clip range
|
||||
@@ -131,12 +131,12 @@ drift-correction path. To change the trim, edit `_CLIP_START_S` and
|
||||
|
||||
| Test | Expected wall clock |
|
||||
|------|---------------------|
|
||||
| AC-1 (`--pace asap`) | ≤ 30 s |
|
||||
| AC-2 schema match | piggybacks on AC-1 |
|
||||
| AC-5 determinism | 2 × asap runs (≤ 60 s total) |
|
||||
| AC-6 realtime | 60 s ± 3 s |
|
||||
| AC-6 asap | ≤ 30 s |
|
||||
| Total suite | ≤ 6 min on Jetson AGX Orin |
|
||||
| AC-1 (`--pace asap`) | ≤ 30 s (Tier-2 only) |
|
||||
| AC-2 schema match | piggybacks on AC-1 (Tier-2 only) |
|
||||
| AC-5 determinism | 2 × asap runs (≤ 60 s total; Tier-2 only) |
|
||||
| AC-6 realtime | 60 s ± 3 s (Tier-2 only) |
|
||||
| AC-6 asap | ≤ 30 s (Tier-2 only) |
|
||||
| Total suite | ~6 min wall clock on Tier-2; skips on Mac |
|
||||
|
||||
The AC-1 / AC-2 / AC-5 tests share `--pace asap` runs but each
|
||||
fixture invocation produces a fresh output file, so they do not
|
||||
@@ -146,14 +146,14 @@ short-circuit each other (preserves AC-5's two-runs-diff guarantee).
|
||||
|
||||
| AC | Test | State |
|
||||
|----|------|-------|
|
||||
| AC-1: exit 0 + JSONL count match | `test_ac1_exits_0_jsonl_count_match` | runs on Tier-1 |
|
||||
| AC-2: JSONL schema match | `test_ac2_jsonl_schema_match` | runs on Tier-1 |
|
||||
| AC-3: ≤ 100 m for 80 % of ticks | `test_ac3_within_100m_80pct_of_ticks` | `xfail` (waiting on real calibration) |
|
||||
| AC-1: exit 0 + JSONL count match | `test_ac1_exits_0_jsonl_count_match` | `xfail` (AZ-963 — open-loop ESKF) |
|
||||
| AC-2: JSONL schema match | `test_ac2_jsonl_schema_match` | Tier-2 (Jetson only) |
|
||||
| AC-3: ≤ 100 m for 80 % of ticks | `test_ac3_within_100m_80pct_of_ticks` | `xfail` (AZ-963 — open-loop ESKF) |
|
||||
| AC-4a: mode-agnosticism AST scan | `test_ac4_mode_agnosticism_ast_scan` | unconditional |
|
||||
| AC-4b: encoder byte-equality | `test_ac4_encoder_byte_equality` | `skip` (waiting on AZ-558) |
|
||||
| AC-5: determinism | `test_ac5_determinism_two_runs_diff` | runs on Tier-1 |
|
||||
| AC-6a: realtime 60 s ± 5 % | `test_ac6_pace_realtime_60s_within_5pct` | runs on Tier-1 |
|
||||
| AC-6b: asap ≤ 30 s | `test_ac6_pace_asap_under_30s` | runs on Tier-1 |
|
||||
| AC-5: determinism | `test_ac5_determinism_two_runs_diff` | `xfail` (AZ-963 — open-loop ESKF) |
|
||||
| AC-6a: realtime 60 s ± 5 % | `test_ac6_pace_realtime_60s_within_5pct` | `xfail` (AZ-963 — open-loop ESKF) |
|
||||
| AC-6b: asap ≤ 30 s | `test_ac6_pace_asap_under_30s` | `xfail` (AZ-963 — open-loop ESKF) |
|
||||
| AC-7: skip-gate self-check | `test_ac7_skip_gate_consistent_with_env_var` | unconditional |
|
||||
| AC-8: operator workflow rehearsal | `test_ac8_operator_workflow` | `skip` (waiting on D-PROJ-2 mock) |
|
||||
| AC-9: helper L2 correctness | `test_helpers.py::test_ac9_l2_*` | unconditional |
|
||||
@@ -187,7 +187,11 @@ tests/e2e/replay/
|
||||
|
||||
## Follow-up work
|
||||
|
||||
* **Real Topotek KHP20S30 calibration** — unblocks AC-3.
|
||||
* **AZ-963** — five Derkachi ACs (`AC-1`, `AC-3`, `AC-5`, `AC-6a`, `AC-6b`)
|
||||
are `xfail` until a reference C6 tile cache exists (resolution path:
|
||||
AZ-777 / AZ-974).
|
||||
* **Real Topotek KHP20S30 calibration** — needed for AC-3 accuracy even
|
||||
after AZ-777 lands (the threshold is ≤100 m for 80 % of ticks).
|
||||
* **AZ-558** — closes AC-4b (route C8 encoders through `MavlinkTransport`).
|
||||
* **D-PROJ-2 mock-suite-sat-service** — unblocks AC-8 (operator
|
||||
workflow rehearsal).
|
||||
|
||||
@@ -7,12 +7,15 @@ E2E pattern the heavy tests are gated by ``RUN_REPLAY_E2E=1``; the
|
||||
lightweight AC-4a (mode-agnosticism AST scan) and AC-7 (skip-gate
|
||||
self-check) run unconditionally.
|
||||
|
||||
Some ACs are SKIPPED with documented reasons until upstream work
|
||||
ships:
|
||||
Environment segregation:
|
||||
|
||||
* **Tier-2 (Jetson)** tests are gated by ``RUN_REPLAY_E2E=1`` +
|
||||
``@pytest.mark.tier2`` — they SKIP on Mac and only run on Jetson
|
||||
where the satellite-provider + C6 tile cache are available.
|
||||
* **Unconditional** tests (AC-4a, AC-4b, AC-7) run everywhere.
|
||||
|
||||
Still skipped with documented reasons:
|
||||
|
||||
* AC-3 (≤ 100 m for 80 % of ticks) — ``xfail`` until a real Topotek
|
||||
KHP20S30 calibration ships (camera_info.md notes the intrinsics
|
||||
are unknown).
|
||||
* AC-4b (encoder byte-equality) — ``skip`` until AZ-558 routes the
|
||||
C8 outbound bytes through the ``MavlinkTransport`` seam.
|
||||
* AC-8 / AC-9 in spec (operator workflow rehearsal) — ``skip`` until
|
||||
@@ -51,6 +54,14 @@ _HEAVY_SKIP = pytest.mark.skipif(
|
||||
_heavy_skip_reason() is not None, reason=_heavy_skip_reason() or "ok"
|
||||
)
|
||||
|
||||
_XFAIL_AZ963_OPEN_LOOP_ESKF = pytest.mark.xfail(
|
||||
strict=False,
|
||||
reason=(
|
||||
"AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF "
|
||||
"diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# AC-1: CLI exits 0; JSONL line count matches per-frame emission count
|
||||
@@ -58,6 +69,7 @@ _HEAVY_SKIP = pytest.mark.skipif(
|
||||
|
||||
@pytest.mark.tier2
|
||||
@_HEAVY_SKIP
|
||||
@_XFAIL_AZ963_OPEN_LOOP_ESKF
|
||||
def test_ac1_exits_0_jsonl_count_match(replay_runner, derkachi_replay_inputs) -> None:
|
||||
"""Real loop emits one EstimatorOutput per video frame, not per GPS fix.
|
||||
|
||||
@@ -144,22 +156,18 @@ def test_ac2_jsonl_schema_match(replay_runner) -> None:
|
||||
|
||||
@pytest.mark.tier2
|
||||
@_HEAVY_SKIP
|
||||
@pytest.mark.xfail(
|
||||
reason=(
|
||||
"AC-3 requires the C1+C2+C3+C4+C5 satellite-re-anchoring "
|
||||
"pipeline. Blocked by AZ-777: with AZ-776 landed, the "
|
||||
"open-loop C1+C5(ESKF) composition now runs end-to-end but "
|
||||
"with NO satellite anchoring (no C2/C3/C4) because the "
|
||||
"Derkachi fixture has no reference C6 tile cache. ESKF "
|
||||
"integrates open-loop, so position drifts unbounded over "
|
||||
"the 8-min flight and the ≤100 m threshold cannot be met "
|
||||
"by physics until the reference tile cache (AZ-777) lands."
|
||||
),
|
||||
strict=False,
|
||||
)
|
||||
@_XFAIL_AZ963_OPEN_LOOP_ESKF
|
||||
def test_ac3_within_100m_80pct_of_ticks(replay_runner, derkachi_replay_inputs) -> None:
|
||||
# Act
|
||||
result = replay_runner(pace="asap")
|
||||
|
||||
# Assert — pipeline must complete cleanly (AZ-963: prevents XPASS
|
||||
# on partial pre-divergence output).
|
||||
assert result.returncode == 0, (
|
||||
f"gps-denied-replay exited {result.returncode}\n"
|
||||
f"stdout:\n{result.stdout}\nstderr:\n{result.stderr}"
|
||||
)
|
||||
|
||||
rows = parse_jsonl(result.output_path)
|
||||
|
||||
# Assert
|
||||
@@ -378,6 +386,7 @@ def test_ac4_encoder_byte_equality_via_transport_seam() -> None:
|
||||
|
||||
@pytest.mark.tier2
|
||||
@_HEAVY_SKIP
|
||||
@_XFAIL_AZ963_OPEN_LOOP_ESKF
|
||||
def test_ac5_determinism_two_runs_diff(replay_runner) -> None:
|
||||
# Act
|
||||
r1 = replay_runner(pace="asap")
|
||||
@@ -407,6 +416,7 @@ def test_ac5_determinism_two_runs_diff(replay_runner) -> None:
|
||||
|
||||
@pytest.mark.tier2
|
||||
@_HEAVY_SKIP
|
||||
@_XFAIL_AZ963_OPEN_LOOP_ESKF
|
||||
def test_ac6_pace_realtime_60s_within_5pct(replay_runner) -> None:
|
||||
# Act — cap to 60 s so a full 490-second flight doesn't pin the test
|
||||
# to an 8-minute realtime run; the pacing correctness is validated
|
||||
@@ -425,6 +435,7 @@ def test_ac6_pace_realtime_60s_within_5pct(replay_runner) -> None:
|
||||
|
||||
@pytest.mark.tier2
|
||||
@_HEAVY_SKIP
|
||||
@_XFAIL_AZ963_OPEN_LOOP_ESKF
|
||||
def test_ac6_pace_asap_under_30s(replay_runner) -> None:
|
||||
# Act
|
||||
result = replay_runner(pace="asap")
|
||||
|
||||
@@ -18,6 +18,8 @@ from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("torch")
|
||||
import torch
|
||||
from torch import nn
|
||||
|
||||
|
||||
Reference in New Issue
Block a user