Skip to content

0029 — Trained expression signal (MediaPipe blendshapes: smile ↔ frown valence)

  • Status: IMPLEMENTED (2026-07-23) — all phases 1–5 shipped and merged (phases 1–2 MR !76, phase 3+5 MR !90, phase 4 MR !92), each validated by a re-cull of the Lyndsey-Denis shoot. The residual no-lock coverage work continues in 0032. Phase 4 (shared backbone) made the mediapipe strategy self-contained: eyes + facing now derive from the FaceMesh (mesh-EAR on recalibrated [0.11,0.24] anchors, chosen over the trained eyeBlink for its 12/14 blink recall), dropping the non-commercial 2d106det from the default stack (now Apache-2.0/MIT). Known limitation: ~22% mesh no-lock reads eyes/facing/expression Unknown on hard faces (safe, no clean fallback exists — ACCEPTED; the faithful-ROI fix is spec 0032, see §9.4). Phase 5 (gate) added an opt-in soft frown floor (cull.profile.frown_soft, a NEGATIVE valence threshold): a frame whose worst face reads grumpier than the floor demotes to maybe, never rejects (Hailey can always keep it deliberately). Off by default (0 = off), worst-face aggregate (Q4), golden-tested in pkg/cull; the studio pins the grumpy face(s) in the loupe. Phase 3 (calibration) added a serve-time valence remap — face.expr.neutral + face.expr.gain, read in meanFaceValence (studio), hot-reloadable with no re-cull, identity by default; verified live on lyndsey-denis (gain 1.3 spreads a smile 93→100, deepens a frown 31→25, leaves neutral put). Gates resolved with Matt: self-convert the models (Q1), ship smile − frown with config-weighted Duchenne/brow extras (Q2), keep the geometric estimator as the config-selected WASM fallback (Q3).
  • Implementation notes. The estimator lives in pkg/face/onnx as the mediapipe strategy (not a separate pkg/face/mediapipe) so it reuses the private ORT runtime/session/landmarker/crop plumbing rather than exporting it — the "(or a blendshape estimator)" option in §3. It composes: EAR still yields eyes + facing, the FaceMesh → blendshape pass yields smile/frown/valence. Models are converted by scripts/convert-mediapipe-models.py (tf2onnx, opset 17), pinned by checksum in pkg/face/models/models.go, and verified against a golden fixture (pkg/face/onnx/testdata). Hosting follow-up → 0031: the two model artefacts are not uploaded yet; the signed krites-models provisioning repo (0031) is their home (and every other model's). Until it lands the strategy is opt-in and resolves them from the local cache.
  • Why now: the current expression signal is a geometric mouth-curvature heuristic with provisional, uncalibrated anchors (face/onnx SmileClosed=0.00/SmileOpen=0.10), explicitly shipped "verdict-neutral until calibrated on a labelled set" (R-EXPR-5). In practice it under-reads badly: a bride with a broad smile scored 0.40, most frames ~0. Expression (smile and frown) is central to judging people at a wedding, so this needs to be as good as we can make it — a trained classifier, not geometry.
  • Decision (2026-07-17): replace the heuristic with MediaPipe FaceMesh + Blendshapes. Rejected AffectNet-trained models (HSEmotion/EmotiEffLib — best accuracy, but AffectNet bars commercial use of "images and any derived data"; same non-commercial taint as our Places2 inpainting flag, and we will not expose the product to that risk). Training our own on permissive data is a future goal, out of scope here.
  • Realises: 0007 expression layer (upgrades Layer-1), the 0019 §5.A explainable-cull moat; lights up the diverging Expression axis added for 0023/the cull-score rail.
  • Depends on: 0004 (face detect/crop — UltraFace), 0007 (expression seam), 0013 (ONNX runtime), 0023 (score display). Anchors: revises R-EXPR-*.

1. Why the heuristic must go

smile = ramp01(mouthCurvatureRatio, SmileClosed, SmileOpen) from 4 lip landmarks is cheap but weak: the curvature ratio correlates only loosely with a real smile, the anchors were never calibrated (no labelled set), and taking the frame's worst face smile (MinSmile) for a display axis makes every group shot read ~0. The 0028/score-rail work gave us a good display (diverging grumpy↔happy), but it is rendering a bad signal.

2. The spike (evidence)

MediaPipe FaceMesh+Blendshapes run on this shoot's frames (2026-07-17):

  • Correct smile magnitude. The broadly-smiling bride (DSC08284) read smile ≈ 0.78 (old geometric: 0.40); a near-neutral face read ≈ 0.01.
  • Per-face smile and frown on a 7-person group (DSC08069, fed the existing UltraFace crops): the one big smiler 0.94, a genuinely unsmiling face frown 0.10 (valence −0.10), the neutral faces ~0.00. Exactly the grumpy↔happy discrimination we want.
  • Coverage caveat: MediaPipe's own short-range face detector under-detects small/ angled faces (many whole-frame "no face"); and on tight crops ~2 of 7 group faces got no mesh lock. → We must not use MediaPipe for detection (see §3).
  • Licence: Apache-2.0, Google-trained weights — clean for local and the SaaS track.

3. Architecture — UltraFace detects, MediaPipe expresses

Keep the proven detector; add a trained expression estimator behind the existing seam.

UltraFace (detect boxes, unchanged)
  └─► per box: crop (padded)
        └─► MediaPipe FaceMesh (478 landmarks, ONNX)
              └─► Blendshape MLP (52 ARKit coeffs, ONNX)
                    └─► expression = f(mouthSmile*, mouthFrown*, …)
  • Detection stays UltraFace (0004) — its coverage is why the spike fed it crops; MediaPipe's detector is not a regression we accept.
  • New provider pkg/face/mediapipe (or a blendshape estimator) behind the face.Analyzer/expression seam, chosen by config — the geometric estimator stays as a fallback for a model-free / WASM build (§7). No package-level hooks; injected per the provider rules.
  • Two ONNX models (facemesh + blendshapes) resolved and dlopened via the existing ORT runtime (0013), auto-provisioned + checksum-pinned like the current face models.
  • A face whose mesh doesn't lock (small/profile) → ExprUnknown, exactly today's degrade path (skipped from aggregates, never a fabricated 0).

4. The expression signal

Per face, from blendshapes: smile = mean(mouthSmileLeft, mouthSmileRight), frown = mean(mouthFrownLeft, mouthFrownRight), and a valence = smile − frown in [−1, 1] (0 = neutral). Optionally fold in cheekSquint* (Duchenne / genuine smile) and browDown* — a small calibrated combination, tunable in config (0001 §6, hot-reload).

  • Display (now): the diverging Expression axis maps valence → the grumpy↔happy bar (neutral centre). Per-frame it aggregates the mean across faces (not the worst).
  • Gate (later): smile/frown can feed the cull verdict once trusted, but stays display-only until calibrated/validated (honours the R-EXPR-5 posture) — flip it on deliberately, not in this spec.

5. Data model & persistence

Persist the raw per-face blendshape-derived values (smile, frown, and the derived valence) on FaceBox, not a pre-baked ramp. Two payoffs:

  1. The display mapping (neutral point, gain, Duchenne weighting) becomes a serve-time, hot-reloadable calibration — tune and refresh, no re-cull (the "live-tunable" idea, now on a signal worth tuning).
  2. Honest provenance: ExprUnknown where no mesh locked.

This is an analysis-cache shape change → AnalysisVersion bump (re-cull to populate; standard R-CAP-3). The geometric MinSmile gate field stays for back-compat during the transition.

6. Studio

No new UI — it feeds the existing diverging Expression axis (0023/score rail), which now shows a trustworthy value. Consider a small "Expression: how it's read" help topic (0026) and, while calibrating, a provisional marker. Face-zoom panels can optionally badge smile/frown per face.

7. Models, ONNX & the WASM line

  • Conversion: MediaPipe ships .tflite; convert facemesh + blendshapes to ONNX (or adopt an existing Apache port, e.g. py-feat/mp_blendshapes), verify parity against the spike numbers, pin + checksum, auto-download (Apache-2.0 permits redistribution).
  • cgo-free / WASM: the ONNX inference is a native/purego+ORT adapter and lives only in its provider package, never imported by the pure-Go deterministic engine (just wasm-check still guards it). The geometric estimator remains the WASM/no-model fallback so the engine still compiles and runs model-free.

8. Licensing (why MediaPipe, explicitly)

Apache-2.0 code and Google-trained weights (not AffectNet/RAF-DB/CelebA, which are all non-commercial). This deliberately sidesteps the "trained on non-commercial data → derived- data taint" trap we already track for Places2-trained inpainting. Record the model provenance + licence in docs and the provisioning manifest.

9. Phasing

  1. Provider + signal. pkg/face/mediapipe (facemesh+blendshapes ONNX on UltraFace crops) → per-face smile/frown/valence; unknown-on-no-lock. Golden-fixture unit tests against the spike's reference values; provider faked in cull tests.
  2. Persist + display. FaceBox raw values (AnalysisVersion bump), serve-time valence mapping (hot-reloadable), wire the diverging Expression axis to it.
  3. ✅ Calibration pass (DONE 2026-07-23). A serve-time valence remap in meanFaceValence — face.expr.neutral (recentre) + face.expr.gain (spread) — read per request, so tuning hot-reloads with no re-cull and outside the golden parity test. Identity by default (the raw signal already spreads well on real shots: 481 faces, valence −0.57…+0.95, 43% happy / 29% neutral / 14% grumpy); the knobs let Hailey tune to taste. Duchenne/brow (face.expr.cheek_squint / brow_down) remain the analysis-time weights (baked into persisted valence).
  4. Shared backbone — IMPLEMENTED (2026-07-23). The mediapipe strategy is now self-contained: eyes and facing derive from the FaceMesh, dropping the InsightFace 2d106det dependency (non-commercial / research-only — a real commercial-track blocker, see 0031 §3). mediapipe is the primary/default strategy; ear + 2d106det remains an opt-in fallback. The default face stack is now Apache-2.0/MIT end-to-end.
  5. Eyes → mesh-EAR (chosen), not eyeBlink. Both were validated head-to-head against the 2d106det baseline over 150 Lyndsey-Denis frames (93 locked faces, 14 baseline blinks):
    • trained eyeBlink blendshapes: 88% verdict agreement, matched scale — but only 4/14 blink recall (misses real blinks; wrong error for a culler).
    • mesh geometric EAR: 12/14 recall, but the mesh EAR scale is intrinsically lower than 2d106det's (p50 0.19 vs ~0.23), so the shared [0.12,0.26] anchors mis-scaled it (62% agreement, mean eyeOpen 0.49). Box-padding was tested and ruled out as the cause (raw mesh-EAR is padding-invariant; padding only marginally, non-monotonically helps the no-lock rate).
    • Fix: mesh-specific EAR anchors [0.11,0.24] (DefaultMeshEAR{Closed,Open}, strategy-aware withDefaults), recalibrated on the same frames → 80% agreement at 12/14 recall (no recall regression). face.ear.* still overrides. Higher-agreement anchors only buy points by dropping real blinks, so recall was held over agreement.
  6. Facing → 2D yaw-asymmetry proxy on mesh corners (for now). The same eye-corner/nose yaw proxy the ear strategy uses, on mesh indices (mesh yaw agrees with the 2d106det baseline on the bride fixture). True 3D head-pose from the mesh's Z coordinates is a deferred refinement (open q. below).
  7. Known limitation — 22% mesh no-lock (ACCEPTED 2026-07-23). FaceMesh fails to lock on ~1 in 5 detected faces (small/turned/occluded); those read eyes/facing/ expression Unknown → no blink/expression gate for that face (safe: never a false reject). There is no clean fallback — eyeBlink also needs the mesh, and the only other landmark source is the 2d106det we are dropping. So going self-contained trades some blink coverage on hard faces for a licence-clean default. A spike (2026-07-23) measured the levers: uniform crop padding of 0.25 recovers ~⅓ (22%→15%, its optimum), naive squaring backfires, and the rest are confident mesh failures (presence logit p50 −8.7) that need rotation-aligned ROI, not more margin. Decision: accept 22% for phase 4 (safe-Unknown) and track the faithful MediaPipe ROI pipeline (padding → size-compensated square → BlazeFace + rotation) as its own effort — spec 0032. The spike harness (meshlock_spike_integration_test.go, TestMeshPadding_lockRate) is the record.
  8. Sequencing: gated on 0031 hosting the models (a fresh default install must have something to download). As shipped in slices A/B the mediapipe strategy still used 2d106det for eyes/facing — this phase actually drops it.
  9. ✅ Gate integration (DONE 2026-07-23). An opt-in soft frown floor cull.profile.frown_soft — a NEGATIVE valence threshold; a frame whose worst face (Q4) reads grumpier than it demotes to maybe, never rejects (the diverging-axis intent: Hailey can always deliberately keep a grumpy frame). Off by default (0/non-negative = off). Wiring: Signals.MinValence (worst-face valence, sentinel −2) from ExpressionSignal, Profile.FrownSoft, a soft branch in expressionIssues, the cullprofile accessor/Overridable parity, and loupe face-pinning (grumpyFaces). Golden-tested in pkg/cull. Requires a re-cull to populate the new signal. This closes R-EXPR-5.

10. Open questions

  • Q1 — tflite→onnx conversion vs an existing ONNX port? Recommend: convert ourselves and pin (control + provenance), unless a well-maintained Apache port matches the spike parity.
  • Q2 — valence formula. smile − frown only, or add cheekSquint/browDown? Recommend: ship smile − frown, make the extra terms config-weighted for calibration.
  • Q3 — keep the geometric estimator? Recommend: yes, as the WASM/model-free fallback (and a fast path), selected by config.
  • Q4 — per-frame aggregate for the gate. Mean for display is settled; if expression ever gates, worst-vs-mean is a separate call (blink-style worst may fit the gate, mean the display).

11. Non-goals

  • Not training our own expression model (future goal, needs permissive data).
  • Not a full emotion taxonomy (anger/fear/…) — smile↔frown valence is the wedding-relevant axis; richer affect is out of scope.
  • Not switching face detection to MediaPipe (UltraFace stays).
  • No change to the non-destructive guarantee; expression is display-first, gate-later.