Decision record — lifecycle edge-case & failure-mode audit
Status: AUDIT COMPLETE (Spike PBI #2005, 2026-07-10). A systematic sweep of every failure/edge a book can hit, confirming each maps to a defined state with a clean, resumable recovery — or spawning a fix ticket where it does not. This underwrites the "secure, stable, reliable, don't-want-to-maintain" bar for a product we sell and don't babysit. It changes docs, not product code; the spawned tickets (§5) carry the fixes. Reads its state vocabulary from the state-machine record
book-status-state-machine.md(#1999), the transition authority it consolidated (#2001), and the described status DTO (#2000).
1. Method
The happy path is solid; a product judged on its edges needs a systematic sweep, not ad-hoc discovery. For each scenario the audit records current handling (with a file:line anchor), a verdict — ✅ handled (cite the covering test), ⚠️ partial (works but a sharp edge / test-only gap), or ❌ gap (a real reliability hole) — and a fix (a spawned, linked ticket for every ⚠️/❌; the covering test for every ✅). Everything traces to code in this repo (verified, not assumed).
2. TL;DR — the scorecard
| # | Scenario cluster | Headline verdict | Spawned |
|---|---|---|---|
| 3.1 | Crash / restart; orphaned jobs | ❌ No job-level crash recovery — a crash mid-batch permanently wedges the book | Bug #2012 |
| 3.2 | Concurrency, idempotency, xmin | ❌ Manual /process//retry lack in-flight dedupe → double-charge; ✅ idempotency-key replay + /cancel xmin loop | Bug #2013 |
| 3.3 | Cancel races; resume; retry races | ✅ cancel→terminal, top-up resume, capacity resume; ❌ cancel-vs-failed-final-batch strand; ⚠️ retry job not in the dedupe scope | Bug #2014 (+ #2013) |
| 3.4 | Batch failure; RunPod modes; cap | ✅ partial failure, throttle/timeout/FAILED disambiguation; ✅ all-pages-past-cap now lands terminal Failed (PBI #2015, shipped) | PBI #2015 |
| 3.5 | Wallet/pages/retention/clock | ✅ completion, cancel-before-delete ordering, UTC discipline clean; ⚠️ several test-only gaps | PBI #2018 |
| 3.6 | Notifications; webhooks | ✅ dedupe (notif + webhook); ❌ lost completion notif on crash; ❌ no reconciliation for undelivered webhooks | PBI #2016, #2017 |
Bottom line: the system is durable for the happy path and clean restarts (state on disk, ordered QueuePosition, a project-level Worker backstop, idempotent settle/notify/webhook, disciplined UTC). Its reliability holes are all at the abnormal-termination seams — a crash, a concurrent enqueue, an exhausted-retry book, an undelivered webhook — none of which self-heal today. Seven tickets close them.
3. The scenario matrix
3.1 Crash / restart with an in-flight job; orphaned or duplicated jobs
| Scenario | Current handling | Verdict |
|---|---|---|
| Clean restart between batches | State lives on disk; DbJobQueue polls NextQueuedAsync every 2 s and re-picks Queued jobs across a restart (DbJobQueue.cs:42-57). Resumability proven by ResumabilityTests.A_fresh_pipeline_over_the_same_data_continues_the_workflow. | ✅ handled |
Crash while a job is Running | The job is flipped Running before the batch runs (JobProcessor.cs:66-68); NextQueuedAsync selects only Queued (EfAccountStores.cs:569). No lease/heartbeat/timeout/startup reaper resets a stale Running job. | ❌ gap |
| Auto-advance blocked by the zombie | BatchAutoAdvancer refuses to advance while any job is Queued or Running (BatchAutoAdvancer.cs:102-106), so the orphaned Running job wedges the book permanently — the core deadlock. | ❌ gap |
| Page in-flight at crash | Left PageStatus.Processing forever; the batch selector matches only Pending (TranslationPipeline.cs:231-234) and moreWork only Pending/Failed (:372); nothing resets Processing → Pending, so the page is silently stranded. | ❌ gap |
| Duplicate batch prevention (auto path) | App-level read-check: existing.Any(Type==ProcessNextBatch && Status is Queued or Running) (BatchAutoAdvancer.cs:102-106), tested by BatchAutoAdvancerTests.An_in_flight_batch_job_blocks_a_second_enqueue. But non-atomic, no DB unique index/concurrency token (ProcessingJobConfiguration.cs:14-18), and the test seeds only a Queued blocker. | ⚠️ partial (race) |
→ Spawned: Bug #2012 (stale-Running reaper + stranded-Processing reset) and Bug #2013 (enforce single in-flight job; §3.2). Test-only gaps → PBI #2018.
3.2 Concurrency: concurrent /process, idempotency-key replay, the xmin write race
| Scenario | Current handling | Verdict |
|---|---|---|
Two concurrent keyless /process | Only a status guard (Program.cs:1966) then unconditional enqueue (:1974-1976); no in-flight dedupe on the manual path. Both pass → two jobs → two settles → double-charge. | ❌ gap |
Manual /process racing auto-advance | The dedupe (BatchAutoAdvancer.cs:102-106) guards only the auto path; the manual endpoint has no equivalent, and the pipeline re-check is status-only (TranslationPipeline.cs:223), not duplicate-aware. | ❌ gap |
| Idempotency replay (server-derived, PBI #2120 — no client header) | WithIdempotency reserve→complete/replay, Redis SET NX (RedisIdempotencyStore.cs:52), fail-open when Redis down, wrapped after the gate. The key is derived server-side from request content + current state (no Idempotency-Key header). Proven by IdempotencyApiTests (Double_submit_of_a_top_up_credits_the_wallet_once_with_no_header, Sequential_sync_batches_never_bill_a_page_twice_with_no_header) + RedisIdempotencyStoreTests. | ✅ handled |
xmin write race on /cancel | The /cancel handler re-reads and re-decides on DbUpdateConcurrencyException (5-attempt loop, Program.cs:2032-2064); SaveProjectAsync deliberately does not retry-and-reapply (Bug #1726, EfProjectStore.cs:37-50). Checkpoint side proven by ProjectStoreStatusProjectionTests.Never_reverts_an_already_persisted_cancellation…. | ✅ handled (but the endpoint loop itself is untested — see #2018) |
→ Spawned: Bug #2013 (in-flight dedupe on manual /process//retry + a partial unique index).
3.3 Cancel races; top-up mid-batch; resume while throttled; retry races
| Scenario | Current handling | Verdict |
|---|---|---|
| Cancel mid-batch → terminal | Three cooperating landers: RunBatchAsync end-of-batch re-read (TranslationPipeline.cs:377-382), TryFinalizeCancellationAsync (:437-450), and the advancer backstop (BatchAutoAdvancer.cs:66-72). Proven by CancellationTests.Cancel_during_a_running_batch_… + BatchAutoAdvancerTests.A_cancelling_project_is_finalized_to_cancelled_not_advanced; no zombie resurrects (A_cancelled_project_is_never_advanced). | ✅ handled |
| Cancel racing a whole-batch failure | If the final winding-down batch fails entirely, the job is Failed → the advancer (gated on Succeeded, JobProcessor.cs:143) is skipped, and the Worker skips Cancelling (not IsResumable). Book stranded in Cancelling. | ❌ gap |
| Top-up mid-batch resume | Advancer re-checks affordability and flips PausedNeedsCredit → Processing only when affordable (BatchAutoAdvancer.cs:112-132). Proven by BatchAutoAdvancerTests.A_topped_up_wallet_auto_resumes_a_paused_needs_credit_project. | ✅ handled |
| Resume while still throttled | PausedNeedsCapacity enqueues while staying PausedNeedsCapacity; the status only moves when RunBatchAsync settles it (or re-pauses). Proven via the pipeline by CapacityPauseTests.The_run_resumes_and_processes_the_same_batch_once_capacity_frees. | ✅ handled (advancer-path enqueue-while-paused untested — #2018) |
| Retry racing cancel / cap | Retry shares TryFinalizeCancellationAsync + the RunBatchAsync re-read; retry-vs-cancel proven by CancellationTests.Retrying_a_cancelled_project_is_a_noop_too; retry past the cap by RetryFailedPagesTests.A_page_that_keeps_failing…. | ✅ handled |
| Retry racing a batch boundary | The advancer's in-flight guard is type-scoped to ProcessNextBatch (BatchAutoAdvancer.cs:103), so a RetryFailedPages job is invisible — a retry and an auto-advanced batch can run concurrently against the same project (lost-update / double-settle). | ❌ gap |
→ Spawned: Bug #2014 (finalize Cancelling after a failed batch); the retry-vs-batch race folds into Bug #2013 (make the dedupe type-agnostic).
3.4 Partial batch failure; storage hiccup; RunPod timeout vs throttle vs FAILED; the attempt cap
| Scenario | Current handling | Verdict |
|---|---|---|
| Partial batch failure | Per-page failure is caught (TranslationPipeline.cs:343-349, :623-629), page left Failed (retryable), batch continues; book returns to Processing. Whole-batch failure rethrows → JobStatus.Failed, book unchanged. Proven by RetryFailedPagesTests.A_failed_page_is_durable_and_does_not_sink_the_rest_of_the_batch. | ✅ handled |
| RunPod throttle vs FAILED vs timeout | Typed mapping in RunPodClient.cs: terminal FAILED/CANCELLED/TIMED_OUT → RunPodException (:128-131); sustained throttle → RunPodCapacityException (:142-152); cold-start rides the full timeout (:154-165). Proven by RunPodResilienceTests (classify / fast-fail / cold-start-keeps-waiting). | ✅ handled (FAILED/TIMED_OUT branch untested — #2018) |
| Storage hiccup mid-page | Best-effort publish/figure/checkpoint swallow storage throws (non-fatal, :100-109,:877-928,:535-542); a main-path storage throw becomes a retryable page failure. Correct by construction, no dedicated test. | ⚠️ partial (test) |
| Per-page attempt cap | ProcessingAttempts capped at MaxProcessingAttempts=3; retry past it throws a terminal "won't be retried" message (TranslationPipeline.cs:276-285). Proven by RetryFailedPagesTests.A_page_that_keeps_failing…. | ✅ handled |
| All pages past the cap | Fixed by PBI #2015. Once no included page has reachable work left (no Pending page anywhere, no Failed page still under the attempt cap) but at least one page permanently exhausted its attempts, TranslationPipeline lands the book on the terminal ProjectStatus.Failed instead of leaving it parked in Processing forever — from both RunBatchAsync's settle decision and RetryFailedPagesAsync's exhausted-gate. The #2000 status DTO's blockedOn=retry/nextAction=retry mapping for Failed (already defined in ProjectStatusDescriber) is now reachable and truthful. Proven by RetryFailedPagesTests.A_page_that_keeps_failing_surfaces_a_terminal_message_instead_of_looping_forever (no other pending work → Failed), A_permanently_failed_page_does_not_block_completion_forever_while_other_pages_are_still_pending (pending work elsewhere keeps it Processing until that work finishes, then Failed), and A_batch_that_both_exhausts_one_page_and_succeeds_on_another_settles_the_book_as_Failed (the RunBatchAsync moreWork path, not just the retry gate). | ✅ handled |
→ Spawned: PBI #2015 (terminal state for an all-pages-past-cap book) — shipped; test gaps → PBI #2018.
3.5 Wallet at exact balance; zero includable pages; retention deletion mid-run; clock skew
| Scenario | Current handling | Verdict |
|---|---|---|
| Wallet estimate == balance | IsAffordable uses estimatedUsd <= wallet.BalanceUsd (BillingService.cs:1098) → affordable (spends to $0); gated on the ceiling. Correct, but no exact-equality boundary test. | ⚠️ partial (test) |
| Zero includable pages / all-Approved | All-Approved & subset reach Complete (TranslationPipeline.cs:372-381); empty selection refused (SelectPagesAsync:176-179); excluded pages never block completion. Proven by PageSelectionTests.Selecting_a_subset_excludes_the_rest…. | ✅ handled |
| Retention deletion mid-run | ProjectDeletionService.DeleteAsync calls _canceller.CancelForProject first, before blob/job/record removal (:29-45) — no zombie writes to deleted data. Ordering correct, not asserted by any test. | ⚠️ partial (test) |
| Clock skew / UTC discipline | Repo-wide grep for DateTime.Now/DateTimeKind.Local/ToLocalTime/TimeZoneInfo.Local over src → zero matches. RetentionPolicy.ExpiresUtc is UTC-kind-stamped + creation-anchored (RetentionPolicy.cs:67-68), test-pinned by RetentionPolicyTests. | ✅ handled |
→ Spawned: PBI #2018 (the estimate==balance, cancel-before-delete, and other test gaps).
3.6 Notifications lost/duplicated; a payment webhook that never delivers
| Scenario | Current handling | Verdict |
|---|---|---|
| Completion notification duplicate | Emission deduped on (OwnerId, JobId, Kind) via AddIfNewAsync + a DB unique index (NotificationService.cs:167-171, EfNotificationStore.cs:18-45). Proven by NotificationEmitTests.Notification_is_deduplicated_per_batch_even_if_emitted_twice. | ✅ handled |
| Completion notification lost | Emitted only inside the live job run right after settle (JobProcessor.cs:98-101); a crash (or swallowed emit failure) between settle and emit loses it permanently — not re-derivable (project already Complete). No outbox/at-least-once (#2002 not built). | ❌ gap |
| Webhook duplicate delivery | GrantCreditsAsync dedupes on (Method, ProviderReference) against a Confirmed ledger row (BillingService.cs:356-363). Proven by BillingApiTests.A_signed_card_webhook_grants_credit_exactly_once… + the CoinPayments regression test. | ✅ handled |
| Webhook never delivers | Hosted Stripe/CoinPayments checkouts grant credit only on the inbound webhook (Program.cs:1491-1550); the checkout stays Pending until it arrives. No reconciliation/polling → user paid, no credit. (Synchronous saved-card path grants inline, unaffected.) | ❌ gap |
→ Spawned: PBI #2016 (at-least-once completion notification) and PBI #2017 (webhook reconciliation).
4. Seeded questions from the state-machine spike (#1999 §7)
- Should
ProjectStatus.Failedever be set? — Yes, and it now is (PBI #2015, shipped). An all-pages-past-cap book no longer sticksProcessingforever with no terminal (§3.4): once no included page has reachable work left, the book settles on the terminalFailed. - Same-state settles / whole-batch-fail-in-place — confirmed intentional:
RunBatchAsyncre-writes the same state on a full-batch failure and a capacity re-pause. No client misreads it today (the status DTO #2000 re-derives from live state each read, not from a change event). The future exactly-once event log (#2002) must treat a same-state settle as a real event, not "no change" — flagged there. ✅ no gap now. - Capacity auto-advance keeps
PausedNeedsCapacityuntil the batch runs — confirmed correct (§3.3); the #2000 DTO already maps that window (state:PausedNeedsCapacity+activeJob:Queued→waitingOn=gpu_capacity). ✅ no gap (advancer-path test → #2018). retryis status-agnostic — confirmed; retry-vs-cancel and retry-past-cap are handled + tested. The one hole is retry racing a batch boundary because the in-flight guard ignoresRetryFailedPagesjobs (§3.3) → folded into Bug #2013.
5. Spawned tickets
| # | Type | Sev/Effort | Closes |
|---|---|---|---|
| #2012 | Bug | High | Crash mid-batch permanently wedges a book — stale-Running reaper + stranded-Processing reset |
| #2013 | Bug | High | Manual /process//retry lack in-flight dedupe → concurrent enqueues double-charge (make the guard enforced + type-agnostic) |
| #2014 | Bug | Medium | Cancel racing a fully-failing final batch strands the book in Cancelling |
| #2015 | PBI | 3 | Terminal state for a book whose pages are all past the retry cap (decide + set ProjectStatus.Failed) — shipped |
| #2016 | PBI | 3 | Make the translation-complete notification at-least-once (survive crash-after-settle-before-emit) |
| #2017 | PBI | 5 | Reconcile undelivered payment webhooks — poll pending checkouts so a paid top-up always credits |
| #2018 | PBI | 3 | Harden lifecycle edge-case test coverage (7 behaviours correct today but untested) |
Each is linked to this spike (#2005) as a predecessor. Related: the multi-user serverless concurrency / cold-start-cost spike (#2009), and the exactly-once event log (#2002).
6. Verification — verified, not assumed
| Claim | Proof |
|---|---|
Running job never reaped; NextQueuedAsync selects only Queued | EfAccountStores.cs:569; no reaper hosted service exists (ProductionInfrastructure.cs) |
Advancer refuses while a job is Queued/Running | BatchAutoAdvancer.cs:102-106 |
Manual /process enqueues with only a status guard | Program.cs:1966-1976; keyless path skips idempotency (Program.cs:2523) |
| Idempotency replay returns one job | IdempotencyApiTests.Replayed_key_on_async_process_enqueues_one_job_not_two; RedisIdempotencyStore.cs:52 |
/cancel re-reads/re-decides on conflict; SaveProjectAsync no-retry by design | Program.cs:2032-2064; EfProjectStore.cs:37-50 (Bug #1726) |
| Cancel mid-batch lands terminal; no zombie resurrects | CancellationTests.Cancel_during_a_running_batch_…; BatchAutoAdvancerTests.A_cancelled_project_is_never_advanced |
Cancel + whole-batch-failure strands Cancelling | advancer gated on Succeeded (JobProcessor.cs:143); Worker skips non-IsResumable (Worker.cs:104) |
| Partial failure keeps the batch alive, page retryable | RetryFailedPagesTests.A_failed_page_is_durable_and_does_not_sink_the_rest_of_the_batch; TranslationPipeline.cs:343-349 |
| RunPod throttle/FAILED/timeout typed + mapped | RunPodClient.cs:128-165; RunPodResilienceTests |
All-pages-past-cap now lands terminal Failed (PBI #2015) | TranslationPipeline.cs RunBatchAsync moreWork/hasExhaustedFailure decision + RetryFailedPagesAsync exhausted-gate; RetryFailedPagesTests (3 tests) |
Wallet estimate==balance is affordable (<=) | BillingService.cs:1098 |
Subset/all-Approved reach Complete | PageSelectionTests.Selecting_a_subset_excludes_the_rest_from_processing_and_completion; TranslationPipeline.cs:372-381 |
| Delete cancels the in-flight job first | ProjectDeletionService.cs:29-45 |
UTC discipline clean; ExpiresUtc UTC-kind-stamped | grep src → 0 DateTime.Now/Local; RetentionPolicy.cs:67-68; RetentionPolicyTests |
| Completion + webhook emission idempotent | NotificationService.cs:167-171 / EfNotificationStore.cs:18-45; BillingService.cs:356-363 |
| Completion notification emitted only in the live run | JobProcessor.cs:98-101 (no re-derive path) |
| Webhook-only credit grant; no reconciliation | Program.cs:1491-1550; BillingService.cs:130-173 |
Related decision records: book-status-state-machine.md (#1999, the state vocabulary), the transition-table consolidation (#2001), the status DTO (#2000), and the exactly-once event log (#2002, not yet built).