Skip to content

0032 — Faithful MediaPipe ROI pipeline (fix the FaceMesh no-lock)

  • Status: IN PROGRESS (2026-07-23). Stage 1 IMPLEMENTED (MR !95 — margin ROI, no-lock 22%→15%). Stage 2 SPIKED & REFUTED — size-compensated square makes coverage worse, not better (see §7). Stage 3 (BlazeFace + rotation) gated on real evidence the residual ~15% hard faces cost keepers; 0032 is parked at Stage 1 until then, with alternative approaches under investigation (§8).
  • Owner: —
  • Realises: closes the coverage gap 0029 §9.4 left open — the mediapipe face strategy's ~22% FaceMesh no-lock. Lifts eye/blink + facing + expression coverage on hard faces toward parity with the (now-dropped) 2d106det path.
  • Depends on: 0029 (mediapipe strategy, mesh eyes/facing), 0004 (face detect/crop — UltraFace), 0013 (ONNX runtime), 0031 (model hosting — a new detector artifact would ride the krites-models release). Anchors: may add R-FACE-* for the ROI transform.

1. The problem

The mediapipe strategy (default since 0029 phase 4) runs MediaPipe FaceMesh on the crop of each UltraFace detection. On this shoot ~22% of detected faces never lock — the mesh presence head returns a confidently-negative logit and the face degrades to Unknown for eyes, facing and expression. That degrade is safe (never a false reject) but it is a real coverage regression versus the 2d106det landmarker, which locked on more of the same faces.

The root cause is that we feed FaceMesh a crop it was not trained to expect. MediaPipe's own pipeline does not hand the mesh a raw detection box — it computes a rotated, square, margin-expanded ROI from the detector's keypoints. We skip all three transforms.

2. Evidence (phase-4 no-lock spike, 2026-07-23)

Measured over 119 detected faces (Lyndsey-Denis), KRITES_TEST_LIMIT=150, via the pkg/face/onnx INT_TEST spike harness (meshlock_spike_integration_test.go, TestMeshPadding_lockRate):

  • UltraFace boxes are consistently portrait — aspect Dx/Dy p10/p50/p90 = 0.63 / 0.75 / 0.90. Scaling that straight into the 256² square input distorts the face horizontally.
  • Uniform padding is a real, cheap lever. No-lock vs pad fraction: 0.00→22%, 0.15→19%, 0.20→18%, 0.25→15%, 0.30→17%, 0.35→18%. Clean optimum at 0.25, recovering 8 of 26 failing faces (~⅓). Cost: mesh-EAR mean shifts 0.187→0.174 (stable across pads), so the mesh anchors re-tighten once, ~[0.11,0.24]→[0.10,0.22] (a re-cull).
  • Naive squaring backfires (72% lock, worse than baseline). Squaring a portrait box adds so much background that the face shrinks in-frame and the mesh loses it — confirming the ROI needs face-size compensation, not just aspect correction.
  • The remaining ~15% is a hard floor for crop tuning. The faces that never lock have presence logits at p50 −8.7 (sigmoid ≈ 0.0002) — confident failures, not borderline, so lowering the presence threshold only admits garbage. These are small / turned / tilted / occluded faces. Beating the floor needs alignment, not more margin.

3. Design — stage the ROI toward MediaPipe fidelity

Each stage is independently shippable and measured by the same spike harness (no-lock rate + locked-face mesh-EAR sanity). Value is highest and certainty greatest at the top.

Stage 1 — uniform margin (the validated quick win)

Expand the detection box by a 0.25 margin before cropFaceNHWC, and re-tighten the mesh EAR anchors to ~[0.10,0.22] (recalibrate with TestMeshAnchorSweep, hold blink recall). No-lock 22%→15%. Pure geometry, no new model, no new dependency. Requires a re-cull. (This is the one stage that could alternatively land as a 0029 phase-4 follow-up; tracked here to keep the ROI story in one place.)

Stage 2 — aspect-correct square ROI with size compensation

Build a centred square ROI (side = long box edge) then scale it so the face occupies a target fraction of the crop (MediaPipe's ~1.5× expand from the tight box), rather than the naive square that shrank the face. This removes the horizontal distortion (which also lets the mesh EAR read closer to the 2d106det scale, shrinking the anchor delta). Measure whether size-compensated squaring beats Stage-1 padding alone or stacks with it.

Stage 3 — rotation alignment (attack the hard floor)

Rotate the ROI so the eyes are horizontal before cropping — the transform MediaPipe uses to make tilted/turned faces present upright to the mesh. This needs a roll estimate, which needs facial keypoints the detector does not currently emit:

  • UltraFace RFB-320 outputs boxes only — no keypoints. So Stage 3 needs either:
  • (A) Adopt BlazeFace as the face detector (MediaPipe-native, emits 6 keypoints incl. both eyes → roll for free; Apache-2.0, consistent with the phase-4 licence-clean goal). A new ONNX artifact hosted via 0031/krites-models. Most faithful; also likely improves detection recall on the hard faces themselves.
  • (B) Keep UltraFace + a cheap keypoint pass to estimate roll (e.g. a tiny 5-point model, or derive roll from a first coarse mesh pass and re-crop). Avoids swapping the detector but adds a stage and its own failure modes.

Rotation is where the confident-negative failures (tilted/profile faces) are recoverable, so Stage 3 is what pushes below the 15% floor — at the cost of a detector change / extra model.

4. Constraints & non-goals

  • Pure-Go geometry. The ROI transform (pad/square/rotate + resample) stays in the pkg/face/onnx adapter, cgo-free; only a new detector model (Stage 3A) is a native/ONNX concern behind the existing session plumbing.
  • Non-destructive / re-cull. Any ROI change alters persisted eye/facing/expression values → AnalysisVersion bump + re-cull. No effect on originals/.
  • Model hosting. Stage 3A's BlazeFace rides a krites-models release (0031); until hosted it is opt-in / cache-resolved, like the 0029 models were.
  • Not a rewrite of the detector for the whole tool — UltraFace stays fine for the cull face-presence/box needs; this is specifically about the crop handed to FaceMesh. If BlazeFace (3A) proves a net detection win, unifying on it is a separate call.

5. Success metric

FaceMesh no-lock < 5% on the Lyndsey-Denis set (from 22%) without regressing the locked-face blink recall or facing agreement (the 0029 §9.4 bars), measured by the meshlock_spike + TestMeshVsLandmark_eyes INT_TEST harnesses. Stage 1 alone reaches ~15%; Stages 2–3 target the floor.

6. Resolved decisions (2026-07-23)

  1. Stage 3 detector → BlazeFace, as a replacement, gated on a spike. BlazeFace is the MediaPipe-native input (emits boxes + score + 6 keypoints → roll for free), Apache-2.0 (in-family), and likely lifts detection recall on the hard faces themselves. Rather than bolt it on for keypoints only (which would mean two detection passes per frame), Stage 3 evaluates it as a full UltraFace replacement; if the detection-quality spike shows it ≥ UltraFace on the shoot, unify on it.
  2. Sequencing → Stage 1 now, Stage 2 fast-follow, Stage 3 gated on real evidence. Ship the validated margin win first (no new dependency), then the size-compensated square (may reach sub-15% with no model), and only take on the BlazeFace lift once the residual hard faces are shown to cost real keepers in Hailey's culls. Stage 1 starts after the phase-4 re-cull is reviewed, so the two validations stay clean.
  3. Reunify anchors → RESOLVED: NO, keep the mesh-specific anchors. The Stage-2 spike confirmed squaring does pull mesh-EAR back onto the 2d106det scale (square ×2.0 → 0.231 ≈ 2d106det's 0.23), so the distortion was indeed what lowered it — but the only square scales that fix the scale (≥×2.0) have unusable lock rates (15%). There is no square operating point that both locks faces and matches the scale, so the mesh keeps its own [0.10,0.22].

7. Implementation plan & sequencing

Every stage is measured by the SAME harness (meshlock_spike_integration_test.go + TestMeshVsLandmark_eyes) — the §5 success metric is the regression gate. Each stage is its own MR + re-cull, incrementing shoot.AnalysisVersion.

Stage 1 — margin ✅ IMPLEMENTED (MR !95, 2026-07-23)

Pure geometry, no new model.

  1. Promote the spike's padBox to production: new pkg/face/onnx/roi.go → faceROI(box, margin, bounds) image.Rectangle (pad + clamp).
  2. meshRunner gains margin (from Config.MeshROIMargin, default DefaultMeshROIMargin = 0.25); mesh.mesh crops cropFaceNHWC(img, faceROI(box, m.margin, img.Bounds()), size). The detector d.box is untouched — the ROI is mesh-input-local, and the eye/facing/ smile signals are crop-space ratios, so nothing needs remapping to image space.
  3. Recalibrate: extend TestMeshAnchorSweep to apply the 0.25 margin, re-derive DefaultMeshEAR{Closed,Open} (~[0.10,0.22]), confirm blink recall ≥ 12/14.
  4. face.mesh.margin config key in faceprovider.analyzerConfig + config docs; bump AnalysisVersion; re-cull; re-run meshlock_spike + TestMeshVsLandmark_eyes to confirm 22%→~15% at no recall cost. Bar: no-lock ≤ 16%, recall ≥ 12/14 (0029 §9.4).

Stage 2 — size-compensated square ❌ SPIKED & REFUTED (2026-07-23)

The spike (meshlock_spike, square ×{1.25,1.5,1.75,2.0} vs Stage-1 padding, 119 faces) shows size-compensated squaring makes coverage monotonically worse, not better:

strategy lock rate mesh-EAR mean
baseline (raw box) 78% 0.187
pad0.25 (Stage 1) 85% 0.174
square ×1.25 80% 0.215
square ×1.5 66% 0.204
square ×1.75 48% 0.206
square ×2.0 15% 0.231

The mesh is far more sensitive to face-size-in-crop than to horizontal distortion: plain padding keeps the face large and adds the margin the mesh wants, whereas squaring shrinks the face in-frame and lock collapses as the scale grows. No square variant beats Stage 1's plain padding. Squaring does fix the EAR scale (decision 3), but only at scales whose lock rate is unusable. Conclusion: plain padding (Stage 1) is the crop-tuning optimum; the residual ~15% need alignment, not aspect/scale tuning — i.e. Stage 3, or a different tool (§8).

Stage 3 — BlazeFace + rotation (the big lift; gated on decision 1)

  1. Decision-gate spike: convert MediaPipe BlazeFace short-range → ONNX; blazeface_integration_test.go compares detection recall/precision + box quality vs UltraFace on the shoot, and validates the eye keypoints.
  2. If BlazeFace ≥ UltraFace: adopt it as THE detector — pkg/face/onnx/blazeface.go (SSD anchor decode + 6 keypoints), face.detector config, host via 0031/krites-models (new signed artifact + Apache-2.0 licence entry).
  3. Rotated faceROI: roll angle from the eye keypoints → affine warp into 256² (golang.org/x/image/draw Interpolator.Transform). Recal, version bump, re-cull. Target no-lock < 5%.

facesettings.go re-pinned face.ear.closed/open on any face-settings save, mis-scaling the mesh eye detection. Fixed by dropping the anchors from the settings view (config-file-only now).

8. Alternative tools & approaches (2026-07-23 investigation)

Before committing to the Stage-3 BlazeFace lift, a survey of permissively-licensed options — in light of the Stage-½ finding that crop tuning caps at ~15% because the residual faces are a model limitation, not a crop one.

Confirmed root cause. MediaPipe FaceMesh has a documented training-data limit: it struggles with faces rotated > ~45°. So no crop transform recovers those faces — alignment or a different model is required. (The survey also confirmed FaceMesh V2 is trained on a 25% margin crop — exactly the Stage-1 MeshROIMargin we landed on empirically. A newer FaceMesh V2 exists and may be more pose-robust; worth a drop-in comparison.)

★ Alternative A (recommended to evaluate) — decouple the eye signal from FaceMesh: YuNet detector + OCEC eye-state, both MIT. - YuNet (OpenCV Zoo, MIT) — a tiny, fast detector that emits box + score + 5 keypoints (both eyes, nose, mouth corners) and is explicitly good on small and side-view faces. Replaces UltraFace: a net detection upgrade and gives the eye keypoints for both ROI rotation and eye-crop extraction. - OCEC (MIT) — a purpose-trained open/closed-eye classifier (24×40 eye crop → sigmoid, F1 ≈ 0.99, 112 KB–6.4 MB ONNX). - The reframe: take the safety-critical blink signal off the FaceMesh entirely — YuNet finds the face + eyes even on hard faces, OCEC classifies open/closed from the eye crops. The 15% mesh-no-lock then only costs expression (opt-in, non-safety-critical), not eyes. This also retires the whole EAR-anchor + margin calibration (mesh-EAR heuristic → trained classifier). FaceMesh stays only for smile/frown/valence (validated). More models, but a more robust and simpler eye path than chasing mesh lock.

Alternative B — YuNet + rotation (Stage 3, but YuNet not BlazeFace). Keep the mesh-EAR eyes; use YuNet's eye keypoints to roll-align the ROI so FaceMesh locks on more faces. YuNet (MIT, OpenCV-supported, ONNX-ready) is likely easier to adopt than converting BlazeFace, and supersedes Stage-3 decision 1: prefer YuNet over BlazeFace as the keypoint detector.

To verify before adopting: (1) the YuNet .onnx weight licence specifically (the OpenCV Zoo repo is MIT; confirm the weights carry no extra terms); (2) YuNet vs UltraFace detection quality on Hailey's shoot (a spike, mirroring the planned BlazeFace spike); (3) OCEC accuracy on real wedding-shot eye crops (blinks, sunglasses, downcast).

Recommendation. If/when the residual hard faces prove to cost keepers, pursue Alternative A (YuNet + OCEC) ahead of the BlazeFace Stage-3 plan — it attacks the safety-critical eye signal directly and sidesteps FaceMesh's fragility, rather than investing in mesh-lock recovery whose ceiling we've now measured.

Feasibility spike (YuNet + OCEC, Python, 2026-07-23)

Ran cv2.FaceDetectorYN (YuNet) + ocec_m.onnx over 150 Lyndsey-Denis frames; baseline the Stage-1 analysis cache (UltraFace 636 faces, 20% mesh no-lock).

  • YuNet detection — strong. Usable eye keypoints on 100% of detections → an eye state on every detected face (the "no-lock" failure mode disappears), including 71 small (<40 px) faces. It detects more faces than UltraFace (~2.7 vs ~1.1 / frame) — coverage is clearly not the problem; the confidence threshold would need tuning to trim background faces.

  • OCEC — capable but crop-sensitive. On faces the mesh calls open, OCEC scores the well-framed eyes correctly (0.97–1.00) but the mis-framed ones 0.00 — a keypoint→box crop (any scale 0.30–0.60 × inter-ocular) frames inconsistently (brow/cheek/blur), so the aggregate open-rate is an implausible ~25%. OCEC trains on tight eye boxes from a dedicated eye detector; YuNet emits eye centres. Reliable eye-state needs proper eye-region extraction (rotation-normalised, tightly framed), not a keypoint heuristic. Refinement — rotation-normalised + framing sweep (still poor). Levelling the eye from both keypoints and sweeping the crop framing (width 0.4–0.65 × iod, aspect 1.6–2.2, vertical shift) only lifts open/closed separation to ~16 points (known-open faces read 57% open, known-closed 41% open). Nowhere near a usable classifier (want ~80-pt separation). OCEC's F1 ≈ 0.99 is measured on eye crops framed by its own DEIMv2 / DINOv3 wholebody detector (classid 17 eye boxes) — a different, heavier input distribution that YuNet's eye centres don't reproduce. And DINOv3 ships under a custom, non-standard licence (commercial-permitted but approval-gated + bespoke terms — not Apache/MIT), so OCEC's proper path would reintroduce exactly the licensing baggage we dropped 2d106det to avoid.

  • Verdict (revised). YuNet is a clear, low-risk, MIT win — adopt it as the detector (better coverage + keypoints, drop-in via OpenCV), superseding BlazeFace. The OCEC eye-decoupling (Alternative A) is downgraded: the cheap YuNet-keypoint→OCEC path does not work, and OCEC's accurate path needs a heavy, non-permissively-licensed eye detector. So the tractable, licence-clean route to lower no-lock is Alternative B — YuNet's eye keypoints roll-align the crop, then the existing (validated) FaceMesh mesh-EAR + expression run on the upright face, keeping everything we already trust and only improving lock. Spike code: scratchpad/yunet-spike/.