# Decision record — lifecycle edge-case & failure-mode audit

> **Status: AUDIT COMPLETE (Spike PBI #2005, 2026-07-10).** A systematic sweep of every failure/edge a
> book can hit, confirming each maps to a defined state with a clean, resumable recovery — or spawning a
> fix ticket where it does not. This underwrites the "secure, stable, reliable, don't-want-to-maintain"
> bar for a product we *sell and don't babysit*. It changes **docs, not product code**; the spawned
> tickets (§5) carry the fixes. Reads its state vocabulary from the state-machine record
> [`book-status-state-machine.md`](https://docs.getkenly.com/design/book-status-state-machine) (#1999), the transition authority it
> consolidated (#2001), and the described status DTO (#2000).

---

## 1. Method

The happy path is solid; a product judged on its edges needs a *systematic* sweep, not ad-hoc discovery.
For each scenario the audit records **current handling** (with a `file:line` anchor), a **verdict** — ✅
*handled* (cite the covering test), ⚠️ *partial* (works but a sharp edge / test-only gap), or ❌ *gap* (a
real reliability hole) — and a **fix** (a spawned, linked ticket for every ⚠️/❌; the covering test for
every ✅). Everything traces to code in this repo (verified, not assumed).

---

## 2. TL;DR — the scorecard

| # | Scenario cluster | Headline verdict | Spawned |
|---|---|---|---|
| 3.1 | Crash / restart; orphaned jobs | ❌ **No job-level crash recovery** — a crash mid-batch permanently wedges the book | Bug #2012 |
| 3.2 | Concurrency, idempotency, xmin | ❌ **Manual `/process`/`/retry` lack in-flight dedupe** → double-charge; ✅ idempotency-key replay + `/cancel` xmin loop | Bug #2013 |
| 3.3 | Cancel races; resume; retry races | ✅ cancel→terminal, top-up resume, capacity resume; ❌ cancel-vs-failed-final-batch strand; ⚠️ retry job not in the dedupe scope | Bug #2014 (+ #2013) |
| 3.4 | Batch failure; RunPod modes; cap | ✅ partial failure, throttle/timeout/FAILED disambiguation; ✅ all-pages-past-cap now lands terminal `Failed` (PBI #2015, shipped) | PBI #2015 |
| 3.5 | Wallet/pages/retention/clock | ✅ completion, cancel-before-delete ordering, **UTC discipline clean**; ⚠️ several test-only gaps | PBI #2018 |
| 3.6 | Notifications; webhooks | ✅ dedupe (notif + webhook); ❌ lost completion notif on crash; ❌ no reconciliation for undelivered webhooks | PBI #2016, #2017 |

**Bottom line:** the system is durable for the happy path and *clean* restarts (state on disk, ordered
`QueuePosition`, a project-level Worker backstop, idempotent settle/notify/webhook, disciplined UTC). Its
reliability holes are all at the **abnormal-termination** seams — a crash, a concurrent enqueue, an
exhausted-retry book, an undelivered webhook — none of which self-heal today. Seven tickets close them.

---

## 3. The scenario matrix

### 3.1 Crash / restart with an in-flight job; orphaned or duplicated jobs

| Scenario | Current handling | Verdict |
|---|---|---|
| Clean restart between batches | State lives on disk; `DbJobQueue` polls `NextQueuedAsync` every 2 s and re-picks `Queued` jobs across a restart (`DbJobQueue.cs:42-57`). Resumability proven by `ResumabilityTests.A_fresh_pipeline_over_the_same_data_continues_the_workflow`. | ✅ handled |
| **Crash while a job is `Running`** | The job is flipped `Running` before the batch runs (`JobProcessor.cs:66-68`); `NextQueuedAsync` selects only `Queued` (`EfAccountStores.cs:569`). **No lease/heartbeat/timeout/startup reaper** resets a stale `Running` job. | ❌ **gap** |
| **Auto-advance blocked by the zombie** | `BatchAutoAdvancer` refuses to advance while any job is `Queued or Running` (`BatchAutoAdvancer.cs:102-106`), so the orphaned `Running` job **wedges the book permanently** — the core deadlock. | ❌ **gap** |
| **Page in-flight at crash** | Left `PageStatus.Processing` forever; the batch selector matches only `Pending` (`TranslationPipeline.cs:231-234`) and `moreWork` only `Pending`/`Failed` (`:372`); nothing resets `Processing → Pending`, so the page is silently stranded. | ❌ **gap** |
| Duplicate batch prevention (auto path) | App-level read-check: `existing.Any(Type==ProcessNextBatch && Status is Queued or Running)` (`BatchAutoAdvancer.cs:102-106`), tested by `BatchAutoAdvancerTests.An_in_flight_batch_job_blocks_a_second_enqueue`. **But** non-atomic, no DB unique index/concurrency token (`ProcessingJobConfiguration.cs:14-18`), and the test seeds only a `Queued` blocker. | ⚠️ partial (race) |

→ **Spawned: Bug #2012** (stale-`Running` reaper + stranded-`Processing` reset) and **Bug #2013** (enforce single in-flight job; §3.2). Test-only gaps → **PBI #2018**.

### 3.2 Concurrency: concurrent `/process`, idempotency-key replay, the xmin write race

| Scenario | Current handling | Verdict |
|---|---|---|
| **Two concurrent keyless `/process`** | Only a status guard (`Program.cs:1966`) then unconditional enqueue (`:1974-1976`); no in-flight dedupe on the manual path. Both pass → two jobs → two settles → **double-charge**. | ❌ **gap** |
| **Manual `/process` racing auto-advance** | The dedupe (`BatchAutoAdvancer.cs:102-106`) guards only the *auto* path; the manual endpoint has no equivalent, and the pipeline re-check is status-only (`TranslationPipeline.cs:223`), not duplicate-aware. | ❌ **gap** |
| Idempotency replay (server-derived, PBI #2120 — no client header) | `WithIdempotency` reserve→complete/replay, Redis `SET NX` (`RedisIdempotencyStore.cs:52`), fail-open when Redis down, wrapped *after* the gate. The key is derived server-side from request content + current state (no `Idempotency-Key` header). Proven by `IdempotencyApiTests` (`Double_submit_of_a_top_up_credits_the_wallet_once_with_no_header`, `Sequential_sync_batches_never_bill_a_page_twice_with_no_header`) + `RedisIdempotencyStoreTests`. | ✅ handled |
| xmin write race on `/cancel` | The `/cancel` handler re-reads and re-decides on `DbUpdateConcurrencyException` (5-attempt loop, `Program.cs:2032-2064`); `SaveProjectAsync` deliberately does **not** retry-and-reapply (Bug #1726, `EfProjectStore.cs:37-50`). Checkpoint side proven by `ProjectStoreStatusProjectionTests.Never_reverts_an_already_persisted_cancellation…`. | ✅ handled (but the endpoint loop itself is untested — see #2018) |

→ **Spawned: Bug #2013** (in-flight dedupe on manual `/process`/`/retry` + a partial unique index).

### 3.3 Cancel races; top-up mid-batch; resume while throttled; retry races

| Scenario | Current handling | Verdict |
|---|---|---|
| Cancel mid-batch → terminal | Three cooperating landers: `RunBatchAsync` end-of-batch re-read (`TranslationPipeline.cs:377-382`), `TryFinalizeCancellationAsync` (`:437-450`), and the advancer backstop (`BatchAutoAdvancer.cs:66-72`). Proven by `CancellationTests.Cancel_during_a_running_batch_…` + `BatchAutoAdvancerTests.A_cancelling_project_is_finalized_to_cancelled_not_advanced`; no zombie resurrects (`A_cancelled_project_is_never_advanced`). | ✅ handled |
| **Cancel racing a *whole-batch failure*** | If the final winding-down batch fails entirely, the job is `Failed` → the advancer (gated on `Succeeded`, `JobProcessor.cs:143`) is skipped, and the Worker skips `Cancelling` (not `IsResumable`). Book **stranded in `Cancelling`**. | ❌ **gap** |
| Top-up mid-batch resume | Advancer re-checks affordability and flips `PausedNeedsCredit → Processing` only when affordable (`BatchAutoAdvancer.cs:112-132`). Proven by `BatchAutoAdvancerTests.A_topped_up_wallet_auto_resumes_a_paused_needs_credit_project`. | ✅ handled |
| Resume while still throttled | `PausedNeedsCapacity` enqueues while *staying* `PausedNeedsCapacity`; the status only moves when `RunBatchAsync` settles it (or re-pauses). Proven via the pipeline by `CapacityPauseTests.The_run_resumes_and_processes_the_same_batch_once_capacity_frees`. | ✅ handled (advancer-path enqueue-while-paused untested — #2018) |
| Retry racing cancel / cap | Retry shares `TryFinalizeCancellationAsync` + the `RunBatchAsync` re-read; retry-vs-cancel proven by `CancellationTests.Retrying_a_cancelled_project_is_a_noop_too`; retry past the cap by `RetryFailedPagesTests.A_page_that_keeps_failing…`. | ✅ handled |
| **Retry racing a batch boundary** | The advancer's in-flight guard is type-scoped to `ProcessNextBatch` (`BatchAutoAdvancer.cs:103`), so a `RetryFailedPages` job is invisible — a retry and an auto-advanced batch can run concurrently against the same project (lost-update / double-settle). | ❌ **gap** |

→ **Spawned: Bug #2014** (finalize `Cancelling` after a failed batch); the retry-vs-batch race folds into **Bug #2013** (make the dedupe type-agnostic).

### 3.4 Partial batch failure; storage hiccup; RunPod timeout vs throttle vs FAILED; the attempt cap

| Scenario | Current handling | Verdict |
|---|---|---|
| Partial batch failure | Per-page failure is caught (`TranslationPipeline.cs:343-349`, `:623-629`), page left `Failed` (retryable), batch continues; book returns to `Processing`. Whole-batch failure rethrows → `JobStatus.Failed`, book unchanged. Proven by `RetryFailedPagesTests.A_failed_page_is_durable_and_does_not_sink_the_rest_of_the_batch`. | ✅ handled |
| RunPod throttle vs FAILED vs timeout | Typed mapping in `RunPodClient.cs`: terminal `FAILED/CANCELLED/TIMED_OUT` → `RunPodException` (`:128-131`); sustained throttle → `RunPodCapacityException` (`:142-152`); cold-start rides the full timeout (`:154-165`). Proven by `RunPodResilienceTests` (classify / fast-fail / cold-start-keeps-waiting). | ✅ handled (FAILED/TIMED_OUT branch untested — #2018) |
| Storage hiccup mid-page | Best-effort publish/figure/checkpoint swallow storage throws (non-fatal, `:100-109`,`:877-928`,`:535-542`); a main-path storage throw becomes a retryable page failure. Correct by construction, **no dedicated test**. | ⚠️ partial (test) |
| Per-page attempt cap | `ProcessingAttempts` capped at `MaxProcessingAttempts=3`; retry past it throws a terminal "won't be retried" message (`TranslationPipeline.cs:276-285`). Proven by `RetryFailedPagesTests.A_page_that_keeps_failing…`. | ✅ handled |
| **All pages past the cap** | **Fixed by PBI #2015.** Once no included page has reachable work left (no `Pending` page anywhere, no `Failed` page still under the attempt cap) but at least one page permanently exhausted its attempts, `TranslationPipeline` lands the book on the terminal `ProjectStatus.Failed` instead of leaving it parked in `Processing` forever — from both `RunBatchAsync`'s settle decision and `RetryFailedPagesAsync`'s exhausted-gate. The #2000 status DTO's `blockedOn=retry`/`nextAction=retry` mapping for `Failed` (already defined in `ProjectStatusDescriber`) is now reachable and truthful. Proven by `RetryFailedPagesTests.A_page_that_keeps_failing_surfaces_a_terminal_message_instead_of_looping_forever` (no other pending work → `Failed`), `A_permanently_failed_page_does_not_block_completion_forever_while_other_pages_are_still_pending` (pending work elsewhere keeps it `Processing` until that work finishes, then `Failed`), and `A_batch_that_both_exhausts_one_page_and_succeeds_on_another_settles_the_book_as_Failed` (the `RunBatchAsync` moreWork path, not just the retry gate). | ✅ **handled** |

→ **Spawned: PBI #2015** (terminal state for an all-pages-past-cap book) — **shipped**; test gaps → **PBI #2018**.

### 3.5 Wallet at exact balance; zero includable pages; retention deletion mid-run; clock skew

| Scenario | Current handling | Verdict |
|---|---|---|
| Wallet estimate == balance | `IsAffordable` uses `estimatedUsd <= wallet.BalanceUsd` (`BillingService.cs:1098`) → **affordable** (spends to $0); gated on the ceiling. Correct, but no exact-equality boundary test. | ⚠️ partial (test) |
| Zero includable pages / all-Approved | All-Approved & subset reach `Complete` (`TranslationPipeline.cs:372-381`); empty selection refused (`SelectPagesAsync:176-179`); excluded pages never block completion. Proven by `PageSelectionTests.Selecting_a_subset_excludes_the_rest…`. | ✅ handled |
| Retention deletion mid-run | `ProjectDeletionService.DeleteAsync` calls `_canceller.CancelForProject` **first**, before blob/job/record removal (`:29-45`) — no zombie writes to deleted data. Ordering correct, **not asserted by any test**. | ⚠️ partial (test) |
| Clock skew / UTC discipline | Repo-wide grep for `DateTime.Now`/`DateTimeKind.Local`/`ToLocalTime`/`TimeZoneInfo.Local` over `src` → **zero matches**. `RetentionPolicy.ExpiresUtc` is UTC-kind-stamped + creation-anchored (`RetentionPolicy.cs:67-68`), test-pinned by `RetentionPolicyTests`. | ✅ handled |

→ **Spawned: PBI #2018** (the estimate==balance, cancel-before-delete, and other test gaps).

### 3.6 Notifications lost/duplicated; a payment webhook that never delivers

| Scenario | Current handling | Verdict |
|---|---|---|
| Completion notification duplicate | Emission deduped on `(OwnerId, JobId, Kind)` via `AddIfNewAsync` + a DB unique index (`NotificationService.cs:167-171`, `EfNotificationStore.cs:18-45`). Proven by `NotificationEmitTests.Notification_is_deduplicated_per_batch_even_if_emitted_twice`. | ✅ handled |
| **Completion notification lost** | Emitted only inside the live job run right after settle (`JobProcessor.cs:98-101`); a crash (or swallowed emit failure) between settle and emit loses it permanently — not re-derivable (project already `Complete`). No outbox/at-least-once (#2002 not built). | ❌ **gap** |
| Webhook duplicate delivery | `GrantCreditsAsync` dedupes on `(Method, ProviderReference)` against a `Confirmed` ledger row (`BillingService.cs:356-363`). Proven by `BillingApiTests.A_signed_card_webhook_grants_credit_exactly_once…` + the CoinPayments regression test. | ✅ handled |
| **Webhook never delivers** | Hosted Stripe/CoinPayments checkouts grant credit *only* on the inbound webhook (`Program.cs:1491-1550`); the checkout stays `Pending` until it arrives. No reconciliation/polling → **user paid, no credit**. (Synchronous saved-card path grants inline, unaffected.) | ❌ **gap** |

→ **Spawned: PBI #2016** (at-least-once completion notification) and **PBI #2017** (webhook reconciliation).

---

## 4. Seeded questions from the state-machine spike (#1999 §7)

1. **Should `ProjectStatus.Failed` ever be set?** — **Yes, and it now is (PBI #2015, shipped).** An
   all-pages-past-cap book no longer sticks `Processing` forever with no terminal (§3.4): once no
   included page has reachable work left, the book settles on the terminal `Failed`.
2. **Same-state settles / whole-batch-fail-in-place** — confirmed intentional: `RunBatchAsync` re-writes the
   same state on a full-batch failure and a capacity re-pause. No client misreads it *today* (the status DTO
   #2000 re-derives from live state each read, not from a change event). The future exactly-once event log
   (#2002) must treat a same-state settle as a real event, not "no change" — flagged there. ✅ no gap now.
3. **Capacity auto-advance keeps `PausedNeedsCapacity`** until the batch runs — confirmed correct (§3.3); the
   #2000 DTO already maps that window (`state:PausedNeedsCapacity` + `activeJob:Queued` → `waitingOn=gpu_capacity`).
   ✅ no gap (advancer-path test → #2018).
4. **`retry` is status-agnostic** — confirmed; retry-vs-cancel and retry-past-cap are handled + tested. The
   one hole is retry racing a batch boundary because the in-flight guard ignores `RetryFailedPages` jobs
   (§3.3) → folded into **Bug #2013**.

---

## 5. Spawned tickets

| # | Type | Sev/Effort | Closes |
|---|---|---|---|
| **#2012** | Bug | High | Crash mid-batch permanently wedges a book — stale-`Running` reaper + stranded-`Processing` reset |
| **#2013** | Bug | High | Manual `/process`/`/retry` lack in-flight dedupe → concurrent enqueues double-charge (make the guard enforced + type-agnostic) |
| **#2014** | Bug | Medium | Cancel racing a fully-failing final batch strands the book in `Cancelling` |
| **#2015** | PBI | 3 | Terminal state for a book whose pages are all past the retry cap (decide + set `ProjectStatus.Failed`) — **shipped** |
| **#2016** | PBI | 3 | Make the translation-complete notification at-least-once (survive crash-after-settle-before-emit) |
| **#2017** | PBI | 5 | Reconcile undelivered payment webhooks — poll pending checkouts so a paid top-up always credits |
| **#2018** | PBI | 3 | Harden lifecycle edge-case test coverage (7 behaviours correct today but untested) |

Each is linked to this spike (#2005) as a `predecessor`. Related: the multi-user serverless concurrency /
cold-start-cost spike (#2009), and the exactly-once event log (#2002).

---

## 6. Verification — verified, not assumed

| Claim | Proof |
|---|---|
| `Running` job never reaped; `NextQueuedAsync` selects only `Queued` | `EfAccountStores.cs:569`; no reaper hosted service exists (`ProductionInfrastructure.cs`) |
| Advancer refuses while a job is `Queued`/`Running` | `BatchAutoAdvancer.cs:102-106` |
| Manual `/process` enqueues with only a status guard | `Program.cs:1966-1976`; keyless path skips idempotency (`Program.cs:2523`) |
| Idempotency replay returns one job | `IdempotencyApiTests.Replayed_key_on_async_process_enqueues_one_job_not_two`; `RedisIdempotencyStore.cs:52` |
| `/cancel` re-reads/re-decides on conflict; `SaveProjectAsync` no-retry by design | `Program.cs:2032-2064`; `EfProjectStore.cs:37-50` (Bug #1726) |
| Cancel mid-batch lands terminal; no zombie resurrects | `CancellationTests.Cancel_during_a_running_batch_…`; `BatchAutoAdvancerTests.A_cancelled_project_is_never_advanced` |
| Cancel + whole-batch-failure strands `Cancelling` | advancer gated on `Succeeded` (`JobProcessor.cs:143`); Worker skips non-`IsResumable` (`Worker.cs:104`) |
| Partial failure keeps the batch alive, page retryable | `RetryFailedPagesTests.A_failed_page_is_durable_and_does_not_sink_the_rest_of_the_batch`; `TranslationPipeline.cs:343-349` |
| RunPod throttle/FAILED/timeout typed + mapped | `RunPodClient.cs:128-165`; `RunPodResilienceTests` |
| All-pages-past-cap now lands terminal `Failed` (PBI #2015) | `TranslationPipeline.cs` `RunBatchAsync` moreWork/hasExhaustedFailure decision + `RetryFailedPagesAsync` exhausted-gate; `RetryFailedPagesTests` (3 tests) |
| Wallet estimate==balance is affordable (`<=`) | `BillingService.cs:1098` |
| Subset/all-Approved reach `Complete` | `PageSelectionTests.Selecting_a_subset_excludes_the_rest_from_processing_and_completion`; `TranslationPipeline.cs:372-381` |
| Delete cancels the in-flight job first | `ProjectDeletionService.cs:29-45` |
| UTC discipline clean; `ExpiresUtc` UTC-kind-stamped | grep `src` → 0 `DateTime.Now`/`Local`; `RetentionPolicy.cs:67-68`; `RetentionPolicyTests` |
| Completion + webhook emission idempotent | `NotificationService.cs:167-171` / `EfNotificationStore.cs:18-45`; `BillingService.cs:356-363` |
| Completion notification emitted only in the live run | `JobProcessor.cs:98-101` (no re-derive path) |
| Webhook-only credit grant; no reconciliation | `Program.cs:1491-1550`; `BillingService.cs:130-173` |

**Related decision records:** [`book-status-state-machine.md`](https://docs.getkenly.com/design/book-status-state-machine) (#1999, the
state vocabulary), the transition-table consolidation (#2001), the status DTO (#2000), and the exactly-once
event log (#2002, not yet built).