Skip to content

0028 — Long operations as reconnectable background jobs (cull first)

  • Status: IMPLEMENTED (2026-07-17). Slices 1–4 landed: server job engine + cull runner; mutable notification, Bell, reconnect/cancel wiring; cached + serialised face-analyzer session; export migrated onto the job model (review/removal stay synchronous by design, §8 slice 4). Verified live end-to-end (start/observe/reconnect/cancel for cull; export as a job).
  • Why now: a face-enabled cull is minutes of CPU work; the studio streams it over a single SSE tied to the request, with no heartbeat. Driven from a tablet that sleeps mid-run, the connection drops and the client shows a bare "network error" while the cull keeps grinding server-side — orphaned, invisible, and re-triggerable into a double-run. The user can see neither (A) that a cull is in progress nor (B) that it finished successfully. (Root-caused 2026-07-17; see 0027 §8.1, which this spec supersedes.)
  • Decisions (2026-07-17): (1) go for the robust model — decouple the operation from the connection (background job + reconnectable progress), not just a heartbeat on a connection-bound stream (a heartbeat alone still loses a slept/closed tablet); (2) the cull is surfaced as one mutable notification in the 0025 panel (started → live progress → terminal, updated in place) with an explicit inline Cancel button; (3) the ORT session is cached/reused across culls — firm requirement, not optional; (4) multiple shoots can cull concurrently — single-flight is per-shoot, not global — so every notification is shoot-attributed and the panel persists and reconciles jobs across shoot navigation (the bell is global, not shoot-scoped).
  • Gate resolutions (2026-07-17, §7): Q1 — explicit cancel only (closing the stream/tab never stops the work; the inline Cancel button is the only stop). Q4 — generalise now: the job model is generic over all long ops (cull and export, review, object-removal) in this spec, not cull-first-others-later — the machinery is built kind-agnostic and every op migrates onto it here. Q2 — persist a marker now: a lightweight in-flight breadcrumb in the shoot sidecar (cleared on terminal) so a server that died mid-op is detected and the op offered for re-run — a crash-recovery aid, not a durable history log.
  • Realises: the 0019 §7 robustness family (resumable/observable long operations); hardens the 0003 studio contract.
  • Depends on: 0003 (studio contract / SSE), 0011 (studio auth), 0025 (notifications — the completion surface). Relates to 0027 (the reset failsafe that the same incident motivated).
  • Anchors: new R-JOB-* requirements (to fold into 0002).

1. Why

Long studio operations (today: cull; also export, review, object-removal) can run far longer than a mobile browser will hold an idle-ish HTTP response, especially over the LAN bridge to a tablet. The cull is the acute case and the only one currently streamed: POST …/cull sets the SSE headers, then runs the whole pipeline on the request goroutine with r.Context(). Two consequences:

  1. The work is welded to the connection. If the connection drops (tablet sleep, wifi blip, refresh), r.Context() cancels → the cull aborts or is orphaned, and the client has no way back to it.
  2. No liveness signal. Nothing flows between the (late) first progress event and the (late) done, so the browser can time the connection out during the model-load or a slow stretch.

The fix is to make a long operation a server-side job the client observes, rather than a stream the client hosts.

2. The model

  • Start = enqueue, not stream. POST …/cull starts a cull job in a goroutine with its own context (independent of the request), records it in an in-memory job registry, and returns 202 { jobId } immediately.
  • Observe = subscribe. The client opens an SSE stream on the job (GET /jobs/{jobId}/events) to receive progress + the terminal event. The stream is re-openable: closing/reopening it (reconnect, refresh, a second device) never affects the job. A heartbeat keeps each open stream warm.
  • Reconnect = re-attach. On (re)load, the client asks "is a cull running for this shoot?" (GET /shoots/{id}/jobs?active=1) and re-attaches to the live job, restoring the in-progress UI — so a slept tablet wakes into the running cull, then sees it finish.
  • Single-flight, keyed by scope. At most one job per scope at once, where the scope key is (shootID, kind, target): target is empty for shoot-wide ops (cull, export) — so one cull and one export per shoot — and the frame id for per-frame ops (review, object-removal) — so many frames run at once, but a double-tap on the same frame attaches rather than double-running. A POST for a scope already in flight returns the existing job's id (attach), never a duplicate run — killing the double-run / double-write risk generically.
  • Cancel = explicit. Because closing the stream no longer cancels the work, cancellation becomes an explicit POST /jobs/{jobId}/cancel (cancels the job's own context). This changes today's semantics (client abort → connection close → ctx cancel); see §7-Q1.

3. Job lifecycle & registry

type Job struct {
    ID        string
    ShootID   string
    ShootName string            // for the shoot-attributed notification (§6.4)
    Kind      Kind              // cull | export | review | removal
    Target    string            // "" for shoot-wide (cull/export); frame id for per-frame (review/removal)
    State     State             // pending | running | succeeded | failed | cancelled
    Progress  struct{ Done, Total int }
    Result    any               // e.g. the verdict tally on success
    Err       string            // on failure
    Started, Updated time.Time
    // internal: own context+cancel; a subscriber fan-out for progress events.
}
  • Registry: a mutex-guarded map[jobID]*Job on the Server, plus a map[scopeKey]jobID single-flight index keyed by (shootID, kind, target) (§2). The registry is generic over Kind — each op registers a runner func(ctx, *Job) (result, error) and the registry owns lifecycle, fan-out, single-flight, retention identically for all of them. In-memory — transient runtime state the studio server already holds (ORT session, caches). The durable part is only the crash breadcrumb (§3.1), which lives in the shoot, not the server.
  • Progress fan-out: the job publishes progress/terminal to any number of live SSE subscribers (channel per subscriber, dropped on unsubscribe). A subscriber that attaches after a terminal state immediately receives that terminal event (so a just-too-late reconnect still learns the outcome).
  • Retention: terminal jobs are retained briefly (a few minutes, and until the next job for that shoot supersedes them) so reconnecting clients see the result, then GC'd.
  • Restart: an in-flight job dies with the server (its goroutine). It leaves a crash breadcrumb in the shoot sidecar (§3.1) so the interruption is detected and the op offered for re-run, rather than silently vanishing.

3.1 Crash breadcrumb (durable, in the shoot) — Q2

A running job writes a small marker into the shoot's sidecar so a server that dies mid-op is recoverable:

  • Location: .krites/jobs/<jobID>.yaml (per-job files under a jobs/ dir; a running op's marker is the only durable job state). Records { id, kind, target, startedAt }.
  • Write on start, delete on terminal. So at steady state a marker exists iff an op is genuinely in flight for that server process. Non-destructive: markers are sidecar metadata (like verdicts/analysis), never touch originals/ (0001 §3, §4.6).
  • Reconcile on shoot-open. When a shoot is opened (openRef), any breadcrumb whose jobID is not live in the in-memory registry was interrupted by a crash/restart → surfaced as a synthetic interrupted job in GET /shoots/{id}/jobs?active=1. The client turns that into a "previous cull was interrupted — re-run?" notification with a Retry action, then the marker is cleared (superseded by the re-run's fresh marker, or by an explicit dismiss).
  • Scope of durability. This is a crash breadcrumb, not an audit trail — it holds only in-flight state and is gone the instant the op finishes. It does not make krites stateful in the catalogue sense (0001 §2); the shoot holds its own transient job state, exactly as it holds its analysis cache. The §9 "no durable job history" non-goal stands for terminal jobs; this is the one carve-out and only for the running one.

4. API surface (studio, 0003 §5) — all behind 0011 auth

Start endpoints (each returns 202 { jobId }, or the running job's id under single-flight, §2). The existing synchronous handlers move onto the job model (Q4):

  • POST /api/v1/shoots/{id}/cull — body may carry { reanalyze: bool } (folds in the 0027 re-cull control). Scope (id, cull, "").
  • POST /api/v1/shoots/{id}/export — scope (id, export, ""). (Was synchronous JSON.)
  • POST /api/v1/shoots/{id}/review — the batch review (0009 slice 6), body { frames: [...] }; scope (id, review, batch:<hash of the selection>), so the same shortlist submitted twice joins the running job instead of paying twice. (The single-frame POST …/frames/{frame}/review stayed synchronous — see slice 4 below.)
  • POST /api/v1/shoots/{id}/frames/{frame}/remove — scope (id, removal, frame).

Observation/control (kind-agnostic — one set of endpoints for every job):

  • GET /api/v1/jobs/{jobId}/events → SSE: progress events, a terminal done (with the kind's result payload) or error, and periodic heartbeat comments. Honours Last-Event-ID for resume. Terminal-then-close if the job is already finished.
  • GET /api/v1/jobs/{jobId} → JSON snapshot (poll fallback / non-SSE clients).
  • GET /api/v1/jobs?active=1 → all running jobs, across every shoot and kind (each carries shootId + shoot name + kind). The global reconnect the bell uses on load.
  • GET /api/v1/shoots/{id}/jobs?active=1 → the running jobs for one shoot (the in-review HUD's narrower view), plus any interrupted breadcrumb reconciled on open (§3.1).
  • POST /api/v1/jobs/{jobId}/cancel → cancels the job's context; job → cancelled.

Compatibility. Because export/review/removal change from a synchronous body response to 202 {jobId}, their clients (api.js) must migrate to the observe-a-job flow in the same slice they move (§8). Endpoints keep their paths; only the response contract changes.

Job endpoints are observation/control of already-gated commands; no new MCP surface (R-MCP-2 unaffected — export/remove keep their ExcludeFromMCP gating; the studio job wrapper is not itself an MCP tool).

5. Heartbeat

Each open …/events stream runs a ticker (~10 s) emitting an SSE comment (: ping\n\n) whenever no real event has flushed in the interval. Cheap, standards-friendly, and it alone would have prevented the observed drop on an attentive session — but it is the liveness layer on top of the decoupling in §2, not a substitute for it.

6. Client & UX (0014 components)

The notifications panel (0025) is the primary progress surface — it is always reachable from the chrome, it survives navigating away from the cull screen (the whole point of the bell, R-NOTIF-2), and it is where the user both watches the cull and cancels it. The in-review throughput HUD stays as the in-context view when the user is on the cull screen, but it is no longer the only place the run is visible.

  • Start: startCull → POST …/cull → { jobId } → open EventSource on …/jobs/{jobId}/events (native auto-reconnect + Last-Event-ID). No more connection-bound runCull AbortController.
  • Reconnect on load: when a shoot opens, GET …/jobs?active=1; if a cull is running, re-attach the EventSource and re-hydrate its notification so the panel shows live progress again after a sleep/refresh.

6.1 One mutable notification per cull (not a new one each event)

A cull owns a single notification, keyed by its jobId, that is updated in place — never a stream of separate entries:

  • On start → create a sticky notification: "Culling lyndsey-denis…", progress 0/total.
  • On each progress/heartbeat → update that same notification's progress (done/total → a live bar + count/ETA). No new entry.
  • On terminal → update it again to the outcome: "Cull finished — 112 keep" (success), "Cull failed — ", or "Cull cancelled" — and unstick it so it ages out like any log entry. Its row action becomes a jump-to-shoot link.

A Cancel action lives on the notification itself while the job runs — an explicit inline Cancel button (→ POST …/cancel), distinct from the row's tap-to-jump. This is the user's always-available way to stop a long cull, even after navigating away.

6.2 Required 0025 store + Bell changes

The 0025 store is append-only today (add(); one whole-row action; no progress; no in-place mutation). This spec extends it (owning the change here since the job drives it):

  • Upsert/update by key — upsert(key, spec) / update(key, patch) so a jobId-keyed notification mutates in place across start → progress → terminal. (add() stays for one-shot notices.)
  • A progress field ({done,total} / percent) rendered by the Bell as a live bar.
  • Inline actions — items may carry action buttons (e.g. Cancel), rendered alongside/instead of the current row-click action. The Bell learns to render a progress bar + inline buttons for an active job row.
  • Sticky/active handling — an in-progress job pins and is exempt from the MAX cap and from being aged off, and markAllRead() must not visually dismiss a running job; it settles into the normal transient log only once terminal.

6.3 Finished & failure (B)

On done, the mutated notification is the unmistakable success confirmation (plus the existing toast); on error/cancelled, it carries the real reason (never a bare "network error") and offers Retry. Because the terminal event is replayed to any late subscriber (§3), a client that reconnects after completion still resolves its notification to the correct outcome.

6.4 Concurrent shoots, persistence & attribution

The whole point of decoupling is that more than one shoot can be culling at once (single-flight is per-shoot, §2). The notification layer must treat that as normal, not an edge case:

  • Shoot attribution on every notification. Each job carries { shootId, shootName }, and the notification leads with it — "lyndsey-denis · Culling… 240/556", "harper-tom · Cull finished — 88 keep". Without the label, two concurrent culls are indistinguishable in the panel. (Job struct §3 already holds ShootID; add the display name.)
  • Persist across shoot navigation. The 0025 store is a session-scoped singleton, so it already survives moving between shoots within the SPA — this spec must not regress that by scoping notifications per shoot. The panel shows all shoots' jobs at once, each its own mutable row with its own Cancel.
  • Server registry is authoritative for "what's running." The client store renders it plus terminal history; on load/refresh the client calls GET /jobs?active=1 (global, §4) and reconciles — opening an EventSource per active job and (re)creating its shoot-labelled notification. So a refresh, or landing on a different shoot than the one culling, still shows every live cull.
  • The in-review HUD stays single-shoot (the shoot on screen); the bell is the cross-shoot view. Two surfaces, one job registry.

7. Gate decisions (resolved 2026-07-17)

  • Q1 — cancellation semantics → EXPLICIT. The move from implicit (close-to-cancel) to explicit POST …/cancel is confirmed. Closing a tab/stream never stops a job; the inline Cancel button is the only stop. It's the whole point of decoupling.
  • Q2 — restart persistence → PERSIST NOW. A running job writes a crash breadcrumb into the shoot sidecar (§3.1), cleared on terminal; a restart reconciles it on shoot-open and offers the interrupted op for re-run. A crash-recovery aid, not a durable history log.
  • Q4 — generalise now or later? → GENERALISE NOW. The registry/runner machinery is built kind-agnostic and all four long ops (cull, export, review, object-removal) migrate onto it within this spec. Single-flight is scope-keyed (shootID, kind, target) so shoot-wide and per-frame ops coexist (§2).

8. Phasing

Q4 makes the model generic from slice 1, but the migration order stays incremental so each slice lands green: build the generic machinery with cull as its first kind, then move the other ops onto it. Cull is still the acute bug and lands first.

  1. Slice 1 — the generic job engine + cull as its first kind (server). Kind-agnostic registry + scope-keyed single-flight + own-context goroutine + subscriber fan-out + retention/GC + the crash breadcrumb (§3.1); the observation/control endpoints (…/jobs/{id}/events SSE with heartbeat + Last-Event-ID, …/jobs/{id}, …/jobs?active=1, …/shoots/{id}/jobs?active=1, …/cancel). Register cull as the first runner; POST …/cull → 202 {jobId}, folding in {reanalyze} (the 0027 re-cull control). TDD: job lifecycle / scope single-flight / fan-out / terminal-replay / breadcrumb-reconcile unit tests + a godog scenario for start→observe→reconnect→done.
  2. Slice 2 — the mutable notification (client). Extend the 0025 store (upsert/update, progress, inline actions, sticky-active) + Bell (progress bar + Cancel button); wire startCull to EventSource + reconnect-on-load + the global ?active=1 reconcile; the single started→progress→terminal notification with inline Cancel; the interrupted-op "re-run?" notification. Playwright for the reconnect + cancel UX. (Kind-agnostic so slice 4's ops light up the same UI for free.)
  3. Slice 3 — cache the ORT session. Hold the built face.Analyzer/ORT session on the server and reuse it across culls (kills the per-run model-init dead time). Firm, not optional. Guard session thread-safety under concurrent/t.Parallel use — likely a guarded singleton or a small pool behind the provider seam (mirror the existing inpaint backend cache: inpMu/inpUse RWMutex draining in-flight use before a config-reload swap). Do not reintroduce a package-level mutable hook (the CLAUDE.md injection rule).
  4. Slice 4 — migrate the fire-and-forget long ops onto the job model. Export becomes a registered runner + 202 {jobId} (with per-frame progress + cancellation added to export.Frames); its api.js client moves to the observe-a-job flow and it lights up the same mutable notification + crash breadcrumb as a cull. Review and object-removal stay synchronous by design — a refinement of Q4: the job engine is generic (any op can be a job), but only the fire-and-forget long ops (cull, export) benefit from decoupling. Review returns a critique the loupe shows inline, and removal is an interactive brush preview / commit returning an image — both are short and need their result in-hand at the call site, so the fire-and-forget-with-reconnect model would degrade their UX for no robustness gain. (If a per-frame op ever needs it, the frame-target single-flight key already supports it.)

Partly revised 2026-07-27 by 0009 slice 6. The reasoning above holds for one frame, and the loupe's single-frame review is still synchronous. But the studio's batch review — a selection sent as one call for a comparative best-of-set read — is a slow, paid call over many megabytes, so it is a job (jobs.KindReview, scope (id, review, batch:<hash of the selection>)). The split is by surface, not by capability: "short and needs its result in-hand" was a property of the single-frame case, not of review as such.

9. Non-goals

  • No cross-shoot job queue / scheduler — one active cull per shoot; this is a local, single-user tool (0001 §2).
  • No durable job history / audit log — terminal jobs are ephemeral. The one durable carve-out is the §3.1 crash breadcrumb for a running job, cleared the instant it ends.
  • No change to the cull's actual pipeline, verdicts, or the non-destructive guarantee — this is purely how the run is driven and observed.
  • Not a general websocket/RPC layer — SSE + JSON, consistent with 0003.

10. Requirements (R-JOB-*, to fold into 0002)

  • R-JOB-1 A long op (cull/export/review/removal) runs as a server-side job on its own context, independent of the request; the start endpoint returns 202 {jobId} and never blocks on the work.
  • R-JOB-2 A job is observed over a re-openable SSE stream (…/jobs/{id}/events); opening, closing, or reopening it (reconnect, refresh, second device) has no effect on the job. Each open stream emits periodic heartbeats.
  • R-JOB-3 A subscriber that attaches after a terminal state immediately receives that terminal event (terminal replay) — a just-too-late reconnect still learns the outcome.
  • R-JOB-4 Single-flight per scope (shootID, kind, target): a start for a scope already in flight returns the existing job's id (attach), never a duplicate run. Shoot-wide ops use empty target; per-frame ops use the frame id.
  • R-JOB-5 Cancellation is explicit (POST …/cancel); disconnecting a stream/closing a tab never cancels a job (Q1).
  • R-JOB-6 A running job leaves a crash breadcrumb in the shoot sidecar, written on start and deleted on terminal; on shoot-open a breadcrumb with no live job is reconciled to an interrupted job offered for re-run (Q2). Breadcrumbs are sidecar-only — originals/ are never touched (non-destructive guarantee, 0001 §3/§4.6).
  • R-JOB-7 The job engine is kind-agnostic: registry, fan-out, single-flight, retention, breadcrumb, heartbeat, and the observe/control endpoints are identical for every kind; a new kind is a registered runner, not new machinery (Q4).
  • R-JOB-8 GET /jobs?active=1 returns all live jobs across every shoot and kind (global reconnect); GET /shoots/{id}/jobs?active=1 returns one shoot's live jobs plus any reconciled interrupted breadcrumb.
  • R-JOB-9 The ORT/face.Analyzer session is built once and reused across jobs, guarded for concurrent use, with no package-level mutable hook (slice 3; the CLAUDE.md injection rule).