Skip to content

0020 — Explainable cull (face zoom · focus peaking · reasons heatmap)

  • Status: IN PROGRESS (2026-07-13) — §7 resolved on the recommendations (on-demand peaking; lazy /analysis endpoint; face+focus issues first, exposure regions deferred; accept the AnalysisVersion 4→5 re-cull; peaking toggle). Slice 1 (face zoom panels) DONE · Slice 2 (focus peaking) DONE · Slice 3 (reasons heatmap) DONE — track complete (exposure-region pins deferred, §7-Q3).
  • Depends on: 0004 (face/eye), 0005 (decision capture / analysis cache), 0006/0007 (develop/aesthetic signals), 0014 (Svelte studio), 0009 (review).
  • Realises: 0019 §5.A "explainable cull" + §9.1 Stage-1; 0016 #19 (face zoom), 0017 #3 (focus peaking), 0017 #5 (reasons heatmap).
  • Anchors: new R-XCULL-* requirements below (to fold into 0002).

1. Why

The cull already decides — it does not yet show its work spatially. A reject reads "eyes closed; soft focus" as text, but Hailey can't see which face blinked or what plane is sharp without zooming and panning. Aftershoot's differentiation (Narrative Select) is exactly this legibility. This spec surfaces the spatial evidence the deterministic engine already computes but currently discards:

  1. Face zoom panels — a strip of 100%-crops of every detected face in the frame, each tagged with its eye-open / smile / facing state.
  2. Focus peaking — a bright edge-energy overlay on the loupe image so the in-focus plane is obvious at a glance.
  3. Reasons heatmap — the verdict's reasons pinned to where on the frame they fired (the blinking face, the blown highlights), not just listed as text.

All three are read-only overlays over existing previews — no pixels written, non-destructive guarantee untouched (R-ND-*).

2. The data gap (why this is spec-gated)

The analysis cache (shoot.Analysis, AnalysisVersion 4) persists only aggregate signals per frame (cull.Signals: FaceCount, MinEyeOpen, MinSmile, MinFacing, Sharpness scalar, clip fractions). The per-face boxes (analyze/face.Face.Box + per-face scores) and the sharpness edge map are computed during the cull and thrown away once reduced to aggregates. Reasons (cull.Reason) are text strings with no coordinates.

So all three features need spatial data that isn't persisted. That is a cache shape change → AnalysisVersion bump to 5 (invalidating existing analysis caches; affected shoots re-cull on next open — standard, R-CAP-3). This is the reason the track is gated on review rather than built inline with the Stage-1 frontend work.

3. Data model changes

3.1 Face boxes (slice 1)

Persist the per-face detail the aggregation drops. Proposed addition to Analysis:

// Faces is the per-frame detected faces, kept so the studio can show face-zoom
// panels + the reasons heatmap. Boxes are NORMALISED (0..1 of the frame W/H) so
// they're resolution-independent across preview sizes. Only present when a face
// model ran (mirrors the eye signal's provenance, R-CAP-3).
Faces map[string][]FaceBox `yaml:"faces,omitempty" json:"faces,omitempty"`

type FaceBox struct {
    X, Y, W, H   float64 // normalised 0..1, top-left origin
    Confidence   float64
    EyeOpen      float64 // -1 = unknown (face.EyeUnknown)
    Smile        float64 // -1 = unknown
    FacingCamera float64 // -1 = unknown
}

Normalised (not image.Rectangle) so it maps onto any preview dimension and stays pure data (WASM-safe, no cgo). Populated in pkg/analyze/face where Face.Box already exists; carried through pkg/pipeline into the cache.

3.2 Reason coordinates (slice 3)

Give a reason an optional locus. Reasons stay backward-compatible (text remains); add a parallel structured issue list per frame:

// Issues is the located, structured form of the verdict reasons: each carries a
// kind + optional region so the studio can pin it on the frame. Text `Reasons`
// stay for the CLI + back-compat; Issues is the spatial superset.
type Issue struct {
    Kind   string   // "blink" | "soft-focus" | "clipped-highlights" | ...
    Region *Region  // normalised box, or nil for whole-frame issues
    FaceIx *int     // index into Faces for face-bound issues (blink)
}

Face-bound issues (blink) reference a FaceBox; exposure issues need a coarse clip region (see §7-Q3 — may defer). Whole-frame issues (soft focus) carry no region and render as a frame-level chip, not a pin.

3.3 Focus-peaking edge map (slice 2)

Not persisted (proposed — §7-Q1). The edge-energy map is large (a per-frame raster) and only ever needed for the single focused frame in the loupe. Compute it on demand from the loupe preview, server-side, in pkg/imaging (pure-Go, the same edge operator the sharpness signal uses), and serve it as a transparent PNG mask the frontend overlays. No cache growth.

4. API surface (studio, 0003 §5)

  • GET /frames/{frame}/analysis — per-frame detail: { faces: [...], issues: [...] }. Fetched lazily when the loupe opens a frame — not folded into the 4,000-row /frames list (§7-Q2), which must stay lean for grid load.
  • GET /frames/{frame}/preview?overlay=peaking — the focus-peaking mask PNG (transparent except edges), sized like the loupe preview. New overlay param alongside the existing develop variants (0017#8 added develop=original).
  • The frames-list payload is unchanged. frameView gains nothing; all new data is detail-fetched, so the grid is unaffected.

5. Studio UI (0014 components)

  • Face zoom panels — a horizontal strip under the loupe image: one 100%-crop per face (CSS object-position/background-position off the loupe preview, no new image fetch), badged with eye/smile/facing state and a warning ring when a face tripped a gate. New FacePanels.svelte.
  • Focus peaking — a loupe toggle (key O, per 0017 #3) that overlays the peaking mask at reduced opacity. New overlay in Loupe / small Peaking.svelte.
  • Reasons heatmap — pins/markers positioned from each Issue.Region/FaceIx over the loupe image; hovering a reason chip highlights its pin and vice-versa.
  • Honesty rule (carry-forward): show only the real deterministic signals (eye / smile / facing / sharpness / exposure). The mockups' Emotion / Composition / Lighting sub-scores are Phase-4 aspirational — grey them until the aesthetic model lands (handover loose end; 0007-DEC-1).

6. Phasing

  1. Slice 1 — face zoom panels. Persist Faces (AnalysisVersion 5), the /analysis endpoint, FacePanels.svelte. Highest differentiation, self-contained. DONE (2026-07-13): face.QualifyingFaces + shoot.FaceBox/Analysis.Faces (version 5); pipeline captures normalised boxes through fresh + cache-reuse; GET /frames/{f}/analysis; loupe Faces strip (CSS-crop off the loupe preview, eye-state labels + blink concern ring). Go + unit + e2e.
  2. Slice 2 — focus peaking. On-demand overlay=peaking mask + loupe toggle. Independent of slice 1 (no cache change beyond slice 1's bump). DONE (2026-07-13): quality.EdgeEnergy (per-pixel form of the sharpness Laplacian, shared via laplacian4) + quality.Peaking (transparent green edge overlay); GET /frames/{f}/preview?overlay=peaking serves it on demand as a cached PNG (no analysis-cache growth — §7-Q1); loupe Focus peaking toggle with the O accelerator (shortcuts.js), overlaid screen-blended on the loupe image. Go + unit + e2e; headless visual confirms pixel-exact alignment. Placement note: the operator + overlay live in pkg/analyze/quality (the operator's home, dependency-free/WASM-safe) rather than pkg/imaging as §3.3 proposed; studio composes it with imaging.EncodePNG. Keeps the "same operator" guarantee literal and avoids an imaging→quality dependency.
  3. Slice 3 — reasons heatmap. Issue model + coordinates, loupe pins. Builds on slice 1 (face-bound issues reference Faces). DONE (2026-07-13): cull.Issue{Kind,Text,Face} is now the single source of a resolution's reasons (cull.Reasons == the issues' Text, so CLI text and the located heatmap never drift); exported cull.Issues(s,p). The /analysis endpoint derives located issues from the persisted signals + faces + the shoot's effective profile (face-bound issues carry the offending faceIx) — no Issue persistence, so no AnalysisVersion bump beyond slice 1's. Loupe renders hover-linked reason chips ↔ face pins, the pins mapped through the loupe's cover crop (faceMarker, unit-tested) and hidden when a face falls in the cropped-away margin. Go + unit + e2e; headless visual confirms the design. Refinement vs §3.2: issues are a serve-time projection, not persisted state, and v1 covers face (blink/smile/facing) + whole-frame (focus/exposure/aesthetic as chips) — exposure clip-region pins remain deferred (§7-Q3). Also fixed a latent UX bug found in testing: entering object-removal now scrolls the drawing canvas into view so it is never left under the toolbar.

Each slice is TDD+BDD: golden-image face-box projection + issue-locus resolution unit tests; a godog scenario for the /analysis + overlay endpoints; Playwright for the loupe overlays; docs-as-we-go. just ci + wasm-check green per slice.

7. Open questions (gates — resolve before implementing)

  • Q1 — peaking map: on-demand vs persisted? Recommendation: on-demand (server-computed, not cached) — keeps the analysis cache lean; the map is only needed for one frame at a time. Confirm, or persist a downscaled map if we want it in the grid too (we don't, today).
  • Q2 — face data delivery: detail endpoint vs frames-list? Recommendation: lazy /frames/{frame}/analysis so the 4,000-frame list stays small. Confirm.
  • Q3 — reasons-heatmap scope for v1. Face-bound (blink) and whole-frame (soft-focus) issues are ~free. Exposure issues (clipped highlights/shadows) need a coarse clip region map — extra compute + cache. Include exposure regions in v1, or ship face+focus pins first and defer exposure to a follow-up?
  • Q4 — AnalysisVersion 4→5 re-cull. Bumping invalidates existing analysis caches; affected shoots re-cull on next open (fast, cached decode). Acceptable (standard R-CAP-3 behaviour) — confirm no migration is wanted instead.
  • Q5 — focus-peaking trigger. 0017 #3 proposes hold-O. Hold-to-show or toggle-on? Recommendation: toggle (consistent with the wiper / lights-out toggles just shipped), with O as the accelerator.

8. Non-goals

  • No new ML signal or model (deterministic evidence only; the aesthetic model stays Phase-4).
  • No pixel writes / no change to export or the non-destructive model.
  • No grid-level overlays — all three features live in the loupe/detail view.