[run-tests-jetson] minting fresh dev JWT via scripts/mint_dev_jwt.py
[run-tests-jetson] using ssh alias: jetson
[run-tests-jetson] remote dir:      /home/jetson/gps-denied-onboard
[run-tests-jetson] remote satprov:  /home/jetson/satellite-provider
[run-tests-jetson] compose file:    docker-compose.test.jetson.yml
[run-tests-jetson] ensure-dev-cert (local)
[ensure-dev-cert] cert present at /Users/zxsanny/dev/azaion/gps-denied-onboard/satellite-provider/certs/api.pfx
[run-tests-jetson] rsync gps-denied-onboard → jetson:/home/jetson/gps-denied-onboard/
Number of files: 1927
Number of files transferred: 2
Total file size: 384584252 B
Total transferred file size: 12082 B
Unmatched data: 2815 B
Matched data: 9267 B
File list size: 136728 B
File list generation time: 0.020 seconds
File list transfer time: 0.041 seconds
Total sent: 137905 B
Total received: 172 B

sent 137905 bytes  received 172 bytes  811740 bytes/sec
total size is 384584252  speedup is 2785.29
[run-tests-jetson] rsync satellite-provider → jetson:/home/jetson/satellite-provider/
Number of files: 805
Number of files transferred: 2
Total file size: 4448030 B
Total transferred file size: 19521 B
Unmatched data: 3698 B
Matched data: 15823 B
File list size: 58214 B
File list generation time: 0.012 seconds
File list transfer time: 0.022 seconds
Total sent: 59226 B
Total received: 232 B

sent 59226 bytes  received 232 bytes  475283 bytes/sec
total size is 4448030  speedup is 74.81
[run-tests-jetson] docker compose build e2e-runner (on Jetson)
 Image gps-denied-onboard/e2e-runner:jetson Building 
 Image gps-denied-onboard/satellite-provider:dev Building 
#1 [internal] load local bake definitions
#1 reading from stdin 1.07kB done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile.jetson
#2 transferring dockerfile: 37B
#2 transferring dockerfile: 5.82kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/dustynv/l4t-pytorch:r36.4.0
#3 DONE 0.5s

#4 [internal] load .dockerignore
#4 transferring context: 383B done
#4 DONE 0.0s

#5 [1/8] FROM docker.io/dustynv/l4t-pytorch:r36.4.0@sha256:a05c85def9139c21014546451d3baab44052d7cabe854d937f163390bfd5201b
#5 resolve docker.io/dustynv/l4t-pytorch:r36.4.0@sha256:a05c85def9139c21014546451d3baab44052d7cabe854d937f163390bfd5201b 0.0s done
#5 DONE 0.0s

#6 [internal] load build context
#6 transferring context: 24.56kB 0.0s done
#6 DONE 0.0s

#7 [4/8] COPY pyproject.toml README.md ./
#7 CACHED

#8 [6/8] RUN rm -f /etc/pip.conf /root/.pip/pip.conf /root/.config/pip/pip.conf
#8 CACHED

#9 [2/8] RUN apt-get update && apt-get install -y --no-install-recommends         ca-certificates         build-essential         libpq-dev         libspatialindex-dev         libpq5         libspatialindex-c6         libgl1         libglib2.0-0         python3-pip         python3-venv     && rm -rf /var/lib/apt/lists/*
#9 CACHED

#10 [3/8] WORKDIR /opt
#10 CACHED

#11 [5/8] COPY src ./src
#11 CACHED

#12 [7/8] RUN pip3 install --no-cache-dir --break-system-packages         --index-url https://pypi.org/simple         --upgrade pip
#12 CACHED

#13 [8/8] RUN pip3 install --no-cache-dir --break-system-packages         --index-url https://pypi.org/simple         -e ".[dev]"
#13 CACHED

#14 exporting to image
#14 exporting layers 0.0s done
#14 exporting manifest sha256:576a6cf55b8c565abc6f2c26b45b8119ef3924d343bfc7f6e2ee32c079230825 done
#14 exporting config sha256:155e7d5a011ea9ab1493a930c71a9d0ed2874479d02f58ece9951c97207454cb done
#14 exporting attestation manifest sha256:bdd66832b7a8d16539d3398081539fcbd31d568f6195ff15d5275bbc414d6db4 0.0s done
#14 exporting manifest list sha256:6253d1aea7392182b2021241c4a4265ea5943e021f3b504de7a721e7e9271884 done
#14 naming to docker.io/gps-denied-onboard/e2e-runner:jetson done
#14 unpacking to docker.io/gps-denied-onboard/e2e-runner:jetson 0.0s done
#14 DONE 0.2s

#15 resolving provenance for metadata file
#15 DONE 0.0s
 Image gps-denied-onboard/e2e-runner:jetson Built 
[run-tests-jetson] docker compose up e2e-runner (on Jetson)
 Network gps-denied-onboard_default Creating 
 Network gps-denied-onboard_default Created 
 Container gps-denied-onboard-db-1 Creating 
 Container gps-denied-e2e-satellite-provider-postgres Creating 
 Container gps-denied-e2e-satellite-provider-postgres Created 
 Container gps-denied-e2e-satellite-provider Creating 
 Container gps-denied-onboard-db-1 Created 
 Container gps-denied-e2e-satellite-provider Created 
 Container gps-denied-onboard-e2e-runner-1 Creating 
 Container gps-denied-onboard-e2e-runner-1 Created 
Attaching to gps-denied-e2e-satellite-provider, gps-denied-e2e-satellite-provider-postgres, db-1, e2e-runner-1
 Container gps-denied-e2e-satellite-provider-postgres Starting 
 Container gps-denied-onboard-db-1 Starting 
 Container gps-denied-onboard-db-1 Started 
 Container gps-denied-e2e-satellite-provider-postgres Started 
 Container gps-denied-e2e-satellite-provider-postgres Waiting 
db-1  | 
db-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
db-1  | 
gps-denied-e2e-satellite-provider-postgres  | 
gps-denied-e2e-satellite-provider-postgres  | PostgreSQL Database directory appears to contain a database; Skipping initialization
gps-denied-e2e-satellite-provider-postgres  | 
db-1                                        | 2026-06-20 08:14:12.259 UTC [1] LOG:  starting PostgreSQL 16.14 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit
db-1                                        | 2026-06-20 08:14:12.259 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db-1                                        | 2026-06-20 08:14:12.259 UTC [1] LOG:  listening on IPv6 address "::", port 5432
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:14:12.261 UTC [1] LOG:  starting PostgreSQL 16.14 (Debian 16.14-1.pgdg13+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
db-1                                        | 2026-06-20 08:14:12.261 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:14:12.261 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:14:12.261 UTC [1] LOG:  listening on IPv6 address "::", port 5432
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:14:12.263 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db-1                                        | 2026-06-20 08:14:12.268 UTC [29] LOG:  database system was shut down at 2026-06-19 12:22:55 UTC
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:14:12.269 UTC [29] LOG:  database system was shut down at 2026-06-19 12:22:56 UTC
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:14:12.278 UTC [1] LOG:  database system is ready to accept connections
db-1                                        | 2026-06-20 08:14:12.278 UTC [1] LOG:  database system is ready to accept connections
 Container gps-denied-e2e-satellite-provider-postgres Healthy 
 Container gps-denied-e2e-satellite-provider Starting 
 Container gps-denied-e2e-satellite-provider Started 
 Container gps-denied-onboard-db-1 Waiting 
 Container gps-denied-e2e-satellite-provider Waiting 
 Container gps-denied-onboard-db-1 Healthy 
gps-denied-e2e-satellite-provider           | 2026-06-20 08:14:18 +00:00 [DBG] Master ConnectionString => Host=satellite-provider-postgres;Port=5432;Database=postgres;Username=postgres;Password=******
gps-denied-e2e-satellite-provider           | 2026-06-20 08:14:19 +00:00 [INF] Beginning database upgrade
gps-denied-e2e-satellite-provider           | 2026-06-20 08:14:19 +00:00 [INF] Checking whether journal table exists
gps-denied-e2e-satellite-provider           | 2026-06-20 08:14:19 +00:00 [INF] Fetching list of already executed scripts.
gps-denied-e2e-satellite-provider           | 2026-06-20 08:14:19 +00:00 [INF] No new scripts need to be executed - completing.
gps-denied-e2e-satellite-provider           | [08:14:19 INF] RegionRequestQueue created with capacity 1000
gps-denied-e2e-satellite-provider           | [08:14:19 INF] Region Processing Service started with 20 parallel workers
gps-denied-e2e-satellite-provider           | [08:14:19 INF] Route Processing Service started
gps-denied-e2e-satellite-provider           | [08:14:19 WRN] Overriding HTTP_PORTS '8080' and HTTPS_PORTS ''. Binding to values defined by URLS instead 'https://+:8080'.
gps-denied-e2e-satellite-provider           | [08:14:19 INF] Now listening on: https://[::]:8080
gps-denied-e2e-satellite-provider           | [08:14:19 INF] Application started. Press Ctrl+C to shut down.
gps-denied-e2e-satellite-provider           | [08:14:19 INF] Hosting environment: Development
gps-denied-e2e-satellite-provider           | [08:14:19 INF] Content root path: /app
 Container gps-denied-e2e-satellite-provider Healthy 
 Container gps-denied-onboard-e2e-runner-1 Starting 
 Container gps-denied-onboard-e2e-runner-1 Started 
e2e-runner-1                                | ============================= test session starts ==============================
e2e-runner-1                                | platform linux -- Python 3.10.12, pytest-9.1.1, pluggy-1.6.0 -- /usr/bin/python3.10
e2e-runner-1                                | cachedir: .pytest_cache
e2e-runner-1                                | rootdir: /opt
e2e-runner-1                                | configfile: pyproject.toml
e2e-runner-1                                | plugins: cov-7.1.0, anyio-4.14.0, asyncio-1.4.0
e2e-runner-1                                | asyncio: mode=strict, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
e2e-runner-1                                | collecting ... collected 57 items
e2e-runner-1                                | 
e2e-runner-1                                | tests/e2e/replay/test_az835_e2e_real_flight.py::test_az840_e2e_real_flight_orchestration SKIPPED [  1%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_1min.py::test_ac1_exits_0_jsonl_count_match XFAIL [  3%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_1min.py::test_ac2_jsonl_schema_match PASSED [  5%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_1min.py::test_ac3_within_100m_80pct_of_ticks XFAIL [  7%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_1min.py::test_ac4_mode_agnosticism_ast_scan PASSED [  8%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_1min.py::test_ac4_encoder_byte_equality_via_transport_seam PASSED [ 10%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_1min.py::test_ac5_determinism_two_runs_diff XFAIL [ 12%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_1min.py::test_ac6_pace_realtime_60s_within_5pct XFAIL [ 14%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_1min.py::test_ac6_pace_asap_under_30s XFAIL [ 15%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_1min.py::test_ac7_skip_gate_consistent_with_env_var PASSED [ 17%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_1min.py::test_ac8_operator_workflow SKIPPED [ 19%]
e2e-runner-1                                | tests/e2e/replay/test_derkachi_real_tlog.py::test_az699_real_flight_validation_emits_verdict_and_report SKIPPED [ 21%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_write_effective_replay_config_overlays_root_dir PASSED [ 22%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_write_effective_replay_config_creates_block_when_absent PASSED [ 24%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_write_effective_replay_config_malformed_yaml_fails PASSED [ 26%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_write_effective_replay_config_non_mapping_top_level_fails PASSED [ 28%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_read_calibration_acquisition_method_returns_field_when_present PASSED [ 29%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_read_calibration_acquisition_method_returns_unknown_on_missing PASSED [ 31%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_read_calibration_acquisition_method_returns_unknown_on_malformed PASSED [ 33%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_missing_tlog_fails_loud PASSED [ 35%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_missing_binary_fails_loud PASSED [ 36%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_replay_nonzero_exit_fails_loud PASSED [ 38%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_replay_timeout_fails_loud PASSED [ 40%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_replay_oserror_fails_loud PASSED [ 42%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_empty_jsonl_fails_loud PASSED [ 43%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_malformed_jsonl_fails_loud PASSED [ 45%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_ground_truth_loader_failure_fails_loud PASSED [ 47%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_happy_path_writes_report PASSED [ 49%]
e2e-runner-1                                | tests/e2e/replay/test_e2e_orchestrator_unit.py::test_run_e2e_orchestration_writes_report_even_on_fail_verdict PASSED [ 50%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_ac9_l2_zero_at_same_point PASSED  [ 52%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_ac9_l2_north_one_degree_111km PASSED [ 54%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_ac9_l2_known_pair_kharkiv_kyiv PASSED [ 56%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_ac9_l2_symmetric PASSED           [ 57%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_match_percentage_all_within_threshold PASSED [ 59%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_match_percentage_none_within_threshold PASSED [ 61%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_match_percentage_empty_emissions_zero PASSED [ 63%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_match_percentage_empty_ground_truth_raises PASSED [ 64%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_parse_jsonl_round_trip PASSED     [ 66%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_parse_jsonl_skips_trailing_blank PASSED [ 68%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_parse_jsonl_invalid_line_raises PASSED [ 70%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_capturing_transport_records_writes PASSED [ 71%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_capturing_transport_close_then_write_raises PASSED [ 73%]
e2e-runner-1                                | tests/e2e/replay/test_helpers.py::test_capturing_transport_implements_protocol PASSED [ 75%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_populate_c6_from_route_returns_populated_cache PASSED [ 77%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_populate_c6_from_route_passes_sector_class_to_downloader PASSED [ 78%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_route_validation_error_propagates_unchanged PASSED [ 80%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_route_terminal_failure_propagates_unchanged PASSED [ 82%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_route_transient_error_retries_then_succeeds PASSED [ 84%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_route_transient_error_exhausted_propagates_last_attempt PASSED [ 85%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_descriptor_index_factory_index_unavailable_propagates PASSED [ 87%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_cleanup_removes_partial_sidecar_files_on_failure PASSED [ 89%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_cleanup_preserves_pre_existing_warm_cache PASSED [ 91%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_batcher_failure_propagates_and_cleans_up PASSED [ 92%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_driver.py::test_downloader_failure_propagates_and_cleans_up PASSED [ 94%]
e2e-runner-1                                | tests/e2e/replay/test_operator_pre_flight_integration.py::test_operator_pre_flight_setup_produces_populated_cache SKIPPED [ 96%]
e2e-runner-1                                | tests/e2e/satellite_provider/test_smoke.py::test_smoke_satellite_provider_inventory_contract FAILED [ 98%]
e2e-runner-1                                | tests/e2e/satellite_provider/test_smoke.py::test_smoke_c11_download_via_http_pipeline FAILED [100%]
e2e-runner-1                                | 
e2e-runner-1                                | =================================== FAILURES ===================================
e2e-runner-1                                | _______________ test_smoke_satellite_provider_inventory_contract _______________
e2e-runner-1                                | tests/e2e/satellite_provider/test_smoke.py:189: in test_smoke_satellite_provider_inventory_contract
e2e-runner-1                                |     assert response.status_code == 200, (
e2e-runner-1                                | E   AssertionError: satellite-provider inventory POST returned 404: ''
e2e-runner-1                                | E   assert 404 == 200
e2e-runner-1                                | E    +  where 404 = <Response [404 Not Found]>.status_code
e2e-runner-1                                | ----------------------------- Captured stdout call -----------------------------
e2e-runner-1                                | {"ts":"2026-06-20T08:15:44.848668Z","level":"INFO","component":"httpx","frame_id":null,"kind":"log.diag","msg":"HTTP Request: POST https://satellite-provider:8080/api/satellite/tiles/inventory \"HTTP/1.1 404 Not Found\"","kv":{},"exc":null}
e2e-runner-1                                | ------------------------------ Captured log call -------------------------------
e2e-runner-1                                | INFO     httpx:_client.py:1025 HTTP Request: POST https://satellite-provider:8080/api/satellite/tiles/inventory "HTTP/1.1 404 Not Found"
e2e-runner-1                                | __________________ test_smoke_c11_download_via_http_pipeline ___________________
e2e-runner-1                                | tests/e2e/satellite_provider/test_smoke.py:301: in test_smoke_c11_download_via_http_pipeline
e2e-runner-1                                |     report = downloader.download_tiles_for_area(request)
e2e-runner-1                                | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:543: in download_tiles_for_area
e2e-runner-1                                |     summaries = self._enumerate_remote(request)
e2e-runner-1                                | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:636: in _enumerate_remote
e2e-runner-1                                |     self._do_enumerate(
e2e-runner-1                                | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:678: in _do_enumerate
e2e-runner-1                                |     summaries.extend(self._fetch_inventory_chunk(chunk))
e2e-runner-1                                | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:683: in _fetch_inventory_chunk
e2e-runner-1                                |     response = self._send_post(
e2e-runner-1                                | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:878: in _send_post
e2e-runner-1                                |     return self._send_request("POST", url, params=None, json_body=json_body, session=session)
e2e-runner-1                                | src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py:963: in _send_request
e2e-runner-1                                |     raise SatelliteProviderError(
e2e-runner-1                                | E   gps_denied_onboard.components.c11_tile_manager.errors.SatelliteProviderError: satellite-provider returned unexpected status 404 (expected 200)
e2e-runner-1                                | ----------------------------- Captured stdout call -----------------------------
e2e-runner-1                                | {"ts":"2026-06-20T08:15:44.866897Z","level":"INFO","component":"c11_tile_manager.tile_downloader","frame_id":null,"kind":"c11.download.session.start","msg":"Pre-flight tile download session started","kv":{"flight_id":"9346cdb7-a5b4-4d87-a47c-370415c297dd","request_hash":"46a59716a231eeab","bbox":[50.099,36.099,50.101,36.101],"zoom_levels":[15],"sector_class":"stable_rear","resume_from_journal":false,"tiles_already_completed":0},"exc":null}
e2e-runner-1                                | {"ts":"2026-06-20T08:15:44.883304Z","level":"INFO","component":"httpx","frame_id":null,"kind":"log.diag","msg":"HTTP Request: POST https://satellite-provider:8080/api/satellite/tiles/inventory \"HTTP/1.1 404 Not Found\"","kv":{},"exc":null}
e2e-runner-1                                | {"ts":"2026-06-20T08:15:44.884249Z","level":"ERROR","component":"c11_tile_manager.tile_downloader","frame_id":null,"kind":"c11.download.provider.failed","msg":"Download provider failed","kv":{"reason":"unexpected_status","http_status":404,"detail":"non-200","auth_header":"Bearer ***"},"exc":null}
e2e-runner-1                                | {"ts":"2026-06-20T08:15:44.888017Z","level":"INFO","component":"c11_tile_manager.tile_downloader","frame_id":null,"kind":"c11.download.session.end","msg":"Pre-flight tile download session ended","kv":{"flight_id":"9346cdb7-a5b4-4d87-a47c-370415c297dd","request_hash":"46a59716a231eeab","outcome":"failure","tiles_requested":0,"tiles_downloaded":0,"tiles_rejected_resolution":0,"tiles_rejected_freshness":0,"tiles_downgraded":0,"retry_count":0},"exc":null}
e2e-runner-1                                | ------------------------------ Captured log call -------------------------------
e2e-runner-1                                | INFO     test_az777_smoke:tile_downloader.py:519 Pre-flight tile download session started
e2e-runner-1                                | INFO     httpx:_client.py:1025 HTTP Request: POST https://satellite-provider:8080/api/satellite/tiles/inventory "HTTP/1.1 404 Not Found"
e2e-runner-1                                | ERROR    test_az777_smoke:tile_downloader.py:994 Download provider failed
e2e-runner-1                                | INFO     test_az777_smoke:tile_downloader.py:578 Pre-flight tile download session ended
e2e-runner-1                                | =============================== warnings summary ===============================
e2e-runner-1                                | ../usr/local/lib/python3.10/dist-packages/faiss/loader.py:44
e2e-runner-1                                |   /usr/local/lib/python3.10/dist-packages/faiss/loader.py:44: DeprecationWarning: 
e2e-runner-1                                |   
e2e-runner-1                                |     `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
e2e-runner-1                                |     of the deprecation of `distutils` itself. It will be removed for
e2e-runner-1                                |     Python >= 3.12. For older Python versions it will remain present.
e2e-runner-1                                |     It is recommended to use `setuptools < 60.0` for those Python versions.
e2e-runner-1                                |     For more details, see:
e2e-runner-1                                |       https://numpy.org/devdocs/reference/distutils_status_migration.html 
e2e-runner-1                                |   
e2e-runner-1                                |   
e2e-runner-1                                |     import numpy.distutils.cpuinfo
e2e-runner-1                                | 
e2e-runner-1                                | -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
e2e-runner-1                                | =========================== short test summary info ============================
e2e-runner-1                                | SKIPPED [1] tests/e2e/replay/test_az835_e2e_real_flight.py:127: AZ-839 operator_pre_flight_setup: descriptor_dim resolver only supports c2_vpr.strategy='net_vlad'; got '<missing>' on backbone 'net_vlad'. See AZ-839 spec § Out of scope.
e2e-runner-1                                | SKIPPED [1] tests/e2e/replay/test_derkachi_1min.py:479: AC-8 (operator workflow rehearsal) blocked on the full D-PROJ-2 mock-suite-sat-service implementation — current tests/fixtures/mock-suite-sat-service/ is a bootstrap stub with only GET /healthz. Unskips when the mock implements tile-fetch + index-build endpoints.
e2e-runner-1                                | SKIPPED [1] tests/e2e/replay/test_derkachi_real_tlog.py:202: real tlog missing: /opt/_docs/00_problem/input_data/flight_derkachi/derkachi.tlog
e2e-runner-1                                | SKIPPED [1] tests/e2e/replay/test_operator_pre_flight_integration.py:22: AZ-839 operator_pre_flight_setup: descriptor_dim resolver only supports c2_vpr.strategy='net_vlad'; got '<missing>' on backbone 'net_vlad'. See AZ-839 spec § Out of scope.
e2e-runner-1                                | XFAIL tests/e2e/replay/test_derkachi_1min.py::test_ac1_exits_0_jsonl_count_match - AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands.
e2e-runner-1                                | XFAIL tests/e2e/replay/test_derkachi_1min.py::test_ac3_within_100m_80pct_of_ticks - AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands.
e2e-runner-1                                | XFAIL tests/e2e/replay/test_derkachi_1min.py::test_ac5_determinism_two_runs_diff - AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands.
e2e-runner-1                                | XFAIL tests/e2e/replay/test_derkachi_1min.py::test_ac6_pace_realtime_60s_within_5pct - AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands.
e2e-runner-1                                | XFAIL tests/e2e/replay/test_derkachi_1min.py::test_ac6_pace_asap_under_30s - AZ-963: Derkachi fixture has no reference C6 tile cache; open-loop ESKF diverges at ~frame 233 (Mahalanobis² > 100). Un-xfail when AZ-777 lands.
e2e-runner-1                                | FAILED tests/e2e/satellite_provider/test_smoke.py::test_smoke_satellite_provider_inventory_contract
e2e-runner-1                                | FAILED tests/e2e/satellite_provider/test_smoke.py::test_smoke_c11_download_via_http_pipeline
e2e-runner-1                                | === 2 failed, 46 passed, 4 skipped, 5 xfailed, 1 warning in 79.92s (0:01:19) ===

[Ke2e-runner-1 exited with code 1
 Compose Stopping Aborting on container exit...
 Container gps-denied-onboard-e2e-runner-1 Stopping 
 Container gps-denied-onboard-e2e-runner-1 Stopped 
 Container gps-denied-onboard-db-1 Stopping 
 Container gps-denied-e2e-satellite-provider Stopping 
gps-denied-e2e-satellite-provider           | [08:15:46 INF] Application is shutting down...
db-1                                        | 2026-06-20 08:15:46.891 UTC [1] LOG:  received fast shutdown request
db-1                                        | 2026-06-20 08:15:46.892 UTC [1] LOG:  aborting any active transactions
db-1                                        | 2026-06-20 08:15:46.897 UTC [1] LOG:  background worker "logical replication launcher" (PID 32) exited with exit code 1
db-1                                        | 2026-06-20 08:15:46.897 UTC [27] LOG:  shutting down
db-1                                        | 2026-06-20 08:15:46.898 UTC [27] LOG:  checkpoint starting: shutdown immediate
db-1                                        | 2026-06-20 08:15:46.904 UTC [27] LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.001 s, total=0.008 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB; lsn=0/1A00478, redo lsn=0/1A00478
gps-denied-e2e-satellite-provider           | [08:15:46 INF] Region Processing Service stopped
db-1                                        | 2026-06-20 08:15:46.919 UTC [1] LOG:  database system is shut down
 Container gps-denied-e2e-satellite-provider Stopped 
 Container gps-denied-e2e-satellite-provider-postgres Stopping 

[Kgps-denied-e2e-satellite-provider exited with code 0
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:15:47.287 UTC [1] LOG:  received fast shutdown request
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:15:47.288 UTC [1] LOG:  aborting any active transactions
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:15:47.298 UTC [1] LOG:  background worker "logical replication launcher" (PID 32) exited with exit code 1
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:15:47.298 UTC [27] LOG:  shutting down
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:15:47.300 UTC [27] LOG:  checkpoint starting: shutdown immediate
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:15:47.306 UTC [27] LOG:  checkpoint complete: wrote 2 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.003 s, sync=0.001 s, total=0.008 s; sync files=3, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB; lsn=0/11341D40, redo lsn=0/11341D40
gps-denied-e2e-satellite-provider-postgres  | 2026-06-20 08:15:47.318 UTC [1] LOG:  database system is shut down
 Container gps-denied-onboard-db-1 Stopped 

[Kdb-1 exited with code 0
 Container gps-denied-e2e-satellite-provider-postgres Stopped 

[Kgps-denied-e2e-satellite-provider-postgres exited with code 0

