Kenly Developers

Decision — Fake model output must never silently reach a real run (fail closed) ​

Spike: #1830 — "confirm whether fake OCR/translation output can silently reach a real (non-dev) run". Parent feature: #1365 (Core Translation & Review Experience). Outcome: a decision (expected vs. gap) + a scoped follow-up. This record promotes the spike note to docs/design/ because the decision changes a documented invariant — the runbook's current "RunPod not configured → Fakes (dev/test default)" default (docs/runbooks/runpod-model-processing.md §TL;DR) — for non-development environments.

✅ Implemented in #1832 (successor to #1830). AddBookTranslation took an allowFakeModelBackend gate and threw when the fakes branch was reached with the gate false; the API + Worker hosts derived the gate from IHostEnvironment. (This gate was superseded by #1840 below.)

✅ Superseded / hardened by #1840: the fakes were removed from the product entirely. There is now no fake model backend anywhere in Infrastructure/Api/Worker — so fake output cannot be served by construction, which is a stronger guarantee than the #1832 startup gate (now removed along with allowFakeModelBackend). AddBookTranslation registers real model stages only when RunPod (or a local OCR worker) is configured and nothing otherwise; the host still starts (the JobProcessor resolves the pipeline lazily) so a translation attempt fails loudly rather than crashing boot. Deterministic model-stage test doubles now live in the test assembly (BookTranslation.Tests/TestDoubles) and are injected into the real host via WebApplicationFactory.ConfigureTestServices (TestModelBackend.AddModelBackendDoubles) — the product ships zero fakes and CI still runs fast + offline. See §Removal (#1840) below.

TL;DR — decisions ​

QuestionAnswerEvidence
Which backend served the 2026-07-04 "Test" screenshot ((fake-translation) [to:en] …, OCR 94%)?Fakes (Milestone-1 placeholders).The (fake-translation) [to:…] prefix is emitted only by FakeTranslationService (FakeAdapters.cs:204); the 94% is the hardcoded PageConfidence: 0.94m in FakeOcrService (FakeAdapters.cs:140). Both are registered only in the final else branch of DependencyInjection.cs:172–181.
Was that a bug, or expected?Expected M1 dev-default. No BookTranslation:RunPod:* was configured for the host that served it, so the precedence chain fell to fakes by design.API host sets runPod = null unless both BookTranslation:RunPod:ApiKey and :EndpointId are set (Program.cs:81–99). M1 is "fake adapters" by design (CLAUDE.md status).
Is there a misconfiguration gap today?No. There is no deployed prod/UAT environment that is expected to route to RunPod but silently falls back — RunPod is opt-in and stood up on demand (scripts/runpod-up.ps1). No environment silently regressed → no Bug filed.RunPod bring-up is a manual, on-demand script; no always-on real environment exists yet.
Is the current signal sufficient?No. The only signal is one startup log line (a Warning), watched by nobody; ModelBackendInfo.IsReal is surfaced on no API endpoint and nowhere in the UI. The lone in-band tell is the literal (fake-translation) prefix — easy to mistake for real content.Whole-repo grep for ModelBackendInfo|IsReal|ModelBackend returns only 3 backend files (DI, announcer, runbook) — zero API/UI surface.
Fix direction (per product owner)?Remove the silent fallback — fail closed. A non-Development host must be explicitly configured with a real backend and refuse to start if it isn't, rather than silently registering fakes. Fakes stay reachable only by explicit opt-in (Development/Testing, or an explicit flag).Scoped as follow-up PBI (successor to #1830, child of #1365); not built in this spike.

The question ​

A Review-page screenshot (project "Test", page 15/30, 2026-07-04) showed the translated pane reading (fake-translation) [to:en] Lorem ipsum dolor sit amet, pagina 15, pars 2. with OCR confidence pinned at exactly 94%, while the project read "Translated" / "Complete". Both strings are Milestone-1 stub hallmarks. Is this the expected dev default, or a wiring gap that could let a real, paying run silently fall back to fakes with no user-visible signal?

Current state (read from the repo, not assumed) ​

Backend selection — AddBookTranslation(...) picks by precedence RunPod > local Python OCR worker > fakes (src/BookTranslation.Infrastructure/DependencyInjection.cs:114–181):

  • runPod is not null → all model stages on RunPod (:121–157).
  • else ocrWorker is not null → local OCR worker + fake language/translation (:158–171).
  • else → all fakes (:172–181), ModelBackendInfo(ModelBackend.Fakes, "FAKES …").

How the API decides — runPod is non-null only when both BookTranslation:RunPod:ApiKey and BookTranslation:RunPod:EndpointId are present (src/BookTranslation.Api/Program.cs:81–99). Absent either, the host silently constructs the fakes graph. There is no environment check — a Production host with a missing/typo'd key degrades to fakes exactly as a dev box does.

The signal — ModelBackendAnnouncer (ModelBackendAnnouncer.cs) logs once at startup: Information for RunPod, Warning for fakes/local-worker. ModelBackendInfo.IsReal (true only for RunPod) exists but is consumed only by that logger. It is on no endpoint and in no screen.

Analysis ​

The screenshot is benign and expected: a dev run with no RunPod config, correctly served by the M1 fakes. Nothing regressed; no Bug.

The latent risk is real and forward-looking. The moment M2/M3 stand up an always-on paying environment, the same silent-degradation path means a missing RunPod key there would serve (fake-translation) placeholder text to a paying user — and, once real billing meters compute (M1 fakes meter $0, so today the wallet isn't charged — see docs/design/cost-estimate-accuracy.md), risk charging for placeholder output. The only thing standing between that and a user is a startup Warning nobody reads and a parenthetical prefix easy to miss.

Two ways to close it were considered:

  1. Surface the active backend (expose ModelBackendInfo/IsReal on the API + a "placeholder output — not a real translation" banner in Review). Makes the state visible but still permits a real environment to run fakes — it only labels it.
  2. Fail closed (chosen). Make fake selection impossible by omission in a real environment: a non-Development host that isn't explicitly configured with a real backend throws at startup rather than registering fakes. Visibility becomes moot because the unsafe state cannot occur in prod.

Option 2 is the product owner's directive ("remove all fallback behavior") and the stronger guarantee. A visible indicator (option 1) is a reasonable complement for dev/demo clarity but is secondary.

Decision ​

Remove the silent fake-backend fallback; fail closed. Concretely, for the follow-up PBI:

  • In a non-Development environment (!IHostEnvironment.IsDevelopment()), if no real model backend is configured (no RunPod, no real OCR worker), the host must throw at composition with an actionable message naming the missing BookTranslation:RunPod:* keys — it must not register the fakes graph.
  • Fakes remain selectable only by explicit opt-in — Development/Testing environments, or an explicit BookTranslation:AllowFakeModelBackend=true (or equivalent) flag — so tests and the bare dotnet run dev loop are unaffected, but fakes can never be reached by config omission in a real deployment.
  • Do not delete FakeAdapters. The M1 test suite and the dev loop depend on them; this is about making their selection explicit and impossible-by-accident, not removing the code.
  • Update docs/runbooks/runpod-model-processing.md (the "Fakes = default when RunPod not configured" TL;DR row) and CLAUDE.md to state the fail-closed invariant.

A user-visible backend indicator (surfacing IsReal on the API/UI) is optional and secondary — it may be folded into the same PBI as a dev-clarity nicety or split out, at the implementer's discretion.

Verification (evidence, not assumed) ​

  • FakeAdapters.cs:140 — OcrResult(PageConfidence: 0.94m, …); :204 — $"(fake-translation) {into} {sourceText}". The screenshot's two hallmarks are these exact literals. ✔
  • DependencyInjection.cs:172–181 — fakes registered only in the no-RunPod/no-worker else. ✔
  • Program.cs:81–99 — API leaves runPod null (→ fakes) unless both RunPod key + endpoint are set; no environment gate. ✔
  • Whole-repo grep ModelBackendInfo|IsReal|ModelBackend → 3 files, all backend; zero API/UI consumers. ✔
  • docs/runbooks/runpod-model-processing.md §TL;DR documents today's "Fakes = dev/test default, WARNING at startup" invariant that this decision revises for non-dev. ✔

Implementation (#1832) ​

The follow-up shipped exactly the fail-closed decision above. The shape:

  • AddBookTranslation(..., bool allowFakeModelBackend = true) — the composition root gains a gate. Its fakes else branch now throws an InvalidOperationException before registering any fake adapter when the gate is false, with an actionable message naming BookTranslation:RunPod:ApiKey / :EndpointId and the BookTranslation:AllowFakeModelBackend opt-in. (src/BookTranslation.Infrastructure/DependencyInjection.cs.)
  • The hosts derive the gate from their environment, so fakes are opt-in only:
    allowFakeModelBackend =
    Environment.IsDevelopment()
    || Environment.IsEnvironment("Testing")
    || config["BookTranslation:AllowFakeModelBackend"] == true
    (src/BookTranslation.Api/Program.cs, src/BookTranslation.Worker/Program.cs.)
  • Why the library default is true, not false: AddBookTranslation is an Infrastructure extension called both by the product's hosts (which always pass the environment-derived gate) and by bare in-process test callers (which have no IHostEnvironment and want the fakes). The safety lives at the host seam — there is no way to run the shipped product without going through the API or Worker host, which is what makes the invariant impossible by construction — so a false default would only churn tests without adding product safety.
  • FakeAdapters are untouched — only their selection is now explicit.
  • Verified by ModelBackendFailClosedTests (src/BookTranslation.Tests) + the runbook/CLAUDE.md updates; the existing suite stays green because it runs under Testing/bare in-process.

Removal (#1840) — no fake model backend at all ​

Product direction (2026-07-04): "remove any remnant of the fakes — let's not have the fakes at all." The #1832 startup gate made a real deploy refuse to serve fakes; #1840 goes further and deletes the fakes, so there is nothing to serve. The shape:

  • Deleted Infrastructure/Fakes/FakeAdapters.cs (fake normalizer/preprocessor/layout/OCR/language/ translation) and removed the fakes DI branch + the allowFakeModelBackend gate + ModelBackend.Fakes. Grep for Fake across Infrastructure/Api/Worker model code returns zero hits (the remaining FakeCardPaymentGateway/FakeFxRateProvider are unrelated, config-gated payment/FX test gateways).
  • AddBookTranslation now registers real model stages only when RunPod (or a local OCR worker) is configured, and nothing otherwise (ModelBackend.None). The one genuine no-op that remained — page preprocessing — is now real product code, PassThroughPagePreprocessor (not a "fake"). The normalizer is always the real PdfPageNormalizer.
  • JobProcessor resolves the pipeline lazily (from IServiceProvider, per job) instead of taking it in its constructor, so a host with no backend still starts — a job then fails with a clear error rather than crashing host startup. (Graceful "GPU starting" UI is #1833.)
    • ITranslationPipeline registration itself is deferred, or lazy resolution is not enough (Bug #1853). In Development, WebApplicationBuilder turns on ServiceProviderOptions.ValidateOnBuild, which walks every singleton's constructor call site at host-build time. A plain AddSingleton<ITranslationPipeline, TranslationPipeline>() therefore had its call site validated eagerly and threw on the unregistered ILayoutDetector — crashing boot before the lazy JobProcessor path ran, so the "host still starts" guarantee held only in Production (where ValidateOnBuild is off). Fix: in the no-backend branch AddBookTranslation registers ITranslationPipeline via a factory that throws an actionable "no model backend configured" error on first use. A factory call site is opaque to ValidateOnBuild, so the host boots; the first resolution (JobProcessor per job, or a /process* endpoint per request) fails loudly. When a backend is configured, the plain type registration is kept (unchanged working path). The test host always hits the no-backend path, so TestModelBackend.AddModelBackendDoubles re-registers the real TranslationPipeline over that factory once its stage doubles are in place.
  • Tests keep running fast + offline: the deterministic model-stage doubles moved to the test assembly (BookTranslation.Tests/TestDoubles/FakeModelAdapters.cs). Direct-construction tests (PipelineTestHarness, ResumabilityTests) reference them there; host-based integration tests inject them via WebApplicationFactory.ConfigureTestServices (TestModelBackend.AddModelBackendDoubles). The product contains zero fake model code. Full suite stays green (544 tests, 0 skipped).

Follow-up ​

  • ✅ #1833 (shipped): model-backend readiness is surfaced on the API and in the SPA — the graceful counterpart to "the host starts even without a live backend." Shape:
    • GET /v1/model-status (anonymous) reports { backend, isReal, ready, state, detail } where state ∈ ready | starting | unconfigured (docs/api-contract.md §5). Backed by an always-registered IModelBackendReadiness that maps the configured ModelBackendInfo + a live health probe to a status.
    • The health probe is IModelHealthProbe, implemented by RunPodClient via the worker's {op:"health"} op on the same /runsync transport that serves real OCR/translation — so it is pod/serverless-agnostic (works unchanged on the dev persistent pod and a production serverless endpoint; only BaseUrl/EndpointId differ). The probe result is TTL-cached + coalesced so a polled endpoint triggers at most one backend health call per window (bounding serverless worker wake-ups).
    • The SPA polls it app-wide (ModelReadinessProvider), shows a top "GPU infrastructure is starting, please stand by…" banner (ModelReadinessBanner) while not ready, and disables the process action until the backend reports ready. No new persistent config toggle — behaviour is automatic.
    • Real-run (pod stopped → banner + gate; pod started → clears) is the same code path as the tested ready/starting probe outcomes; exercising it against a live pod is billable RunPod spend, deferred to an on-demand bring-up.