Skip to content

0008 — object removal: the Inpainter provider & the remove command

Status: IN PROGRESS (2026-07-04) — the Phase-3 master spec (0001 §11). Companion to 0001-krites.md (§4.5 object removal, §5 providers, §13-Q1 ML runtime / Q3 cloud / Q6 hardware), 0002-interface-contracts.md (R-RM-1..2, R-EXP-1, R-MCP-2, R-PRIV-2, R-GLOBAL-8), and the patterns proven in 0004-face-eye.md (native ONNX adapter behind an interface, purego/ORT, layered build), 0006-develop.md (the edit-record model + the pure-Go export bake this extends), and 0007-aesthetic-expression.md (the seam-ships- inert-first discipline, and the model-provenance bar that gates the adapter).

Status: reviewed; all gating questions resolved; cleared to build. (2026-07-04, Matt.) Q1 (model), Q2 (patch storage) and Q3 (original-frame coords, composite-first) are resolved (§8). The backend is LaMa (self-hosted, pinned, Apache-2.0 ONNX) as the default eraser, MI-GAN to follow as a fast secondary behind the same seam; the Places2 training-data licence risk is a known, documented caveat (§4.1) to be surfaced in docs and code, not a blocker. Slice 1 (the pure spine) and slice 2 (the LaMa adapter) are both unblocked.

1. The shape of the work — and the one load-bearing decision

Object removal is the first generative capability in krites: the user marks a region (brush/box in the studio, a --region on the CLI) and an inpainting model fills it — removing a stray guest, a distraction, an exit sign. Everything before this phase was either pure deterministic maths (quality, dedup, straighten, crop, look bake) or a discriminative model that returns numbers (face/eye, expression). Inpainting produces pixels, and that is what makes the design interesting.

The load-bearing decision that keeps the rest of the system honest:

The inpaint materialises to a stored raster at remove (accept) time; export composites that cached raster purely. The generative step — the one native, non-deterministic, provider-backed operation — happens once, when the removal is accepted, and writes a result raster into the shoot's sidecar cache. The export bake stays a pure-Go, deterministic, WASM-safe pipeline (0006 R-EXP-4): it composites a cached image, it never calls a model. So remove joins cull/develop as provider-backed analysis-time work, and export keeps its single guarantee — deterministic pixels from recorded instructions (R-EXP-2).

This mirrors, exactly, how the whole engine already splits: heavy/native work produces recorded artefacts; the pure core consumes them. It is why removal fits the non-destructive model without weakening it.

2. Package layout & the WASM boundary

Following the 0007 split (aesthetic.Scorer interface pure + faked; the ONNX backend a separate native package):

Package Contents Native? WASM-safe?
pkg/develop (extend edit.go) the Removal instruction on the edit record — plain data (mask + result reference + basis) no yes (pure, like Crop)
pkg/remove the Inpainter interface + the removal orchestration (validate mask → call provider with timeout → store result → record instruction); provider-neutral, faked in tests no (interface + orchestration only) interface yes; orchestration is provider-facing, not in the WASM guard set
pkg/remove/onnx (new, later slice) the LaMa adapter — purego + bundled ORT dylib, reusing the pkg/face/onnx session machinery yes no — never imported by the engine
pkg/develop/bake (extend) composite the cached removal raster first in the pure bake no yes

The instruction and the compositing are pure and WASM-safe; the model is quarantined in its own adapter package behind the Inpainter interface, exactly as 0001 §7 and just wasm-check require. pkg/remove is not added to the wasm-check guard set (it is provider-facing orchestration, not deterministic engine), but it holds no native import itself — the adapter does.

3. The removal instruction (non-destructive, R-RM-1, R-ND-2)

A new optional member on EditRecord (pkg/develop/edit.go), keeping every field optional so an all-nil record stays a no-op (0006 R-EDIT-½):

type EditRecord struct {
    Straighten *Straighten `yaml:"straighten,omitempty" ...`
    Crop       *Crop       `yaml:"crop,omitempty" ...`
    Look       *LookRef    `yaml:"look,omitempty" ...`
    Removals   []Removal   `yaml:"removals,omitempty" ...` // NEW — ordered, additive
}
  • Removal is one accepted removal: a mask + a result reference + a basis. Multiple removals compose in record order.
  • Mask — in normalised original-frame coordinates (fractions in [0,1], origin top-left) so it is resolution-independent and shares one coordinate space with Crop/straighten (0006 R-EDIT-3). Two mask kinds, additive:
  • a box (x,y,w,h) — the CLI's --region (slice 1);
  • a brush mask — a 1-bit PNG stored in the sidecar, referenced by path (the studio's freeform brush, a later slice).
  • Result reference — a path to the materialised inpaint raster in the sidecar cache (.krites/removals/<frame>/<id>.png), stored as the bounding-box patch of the affected region plus its normalised offset (smaller than a full frame; composited back at its box). Full-resolution, so export is quality-faithful.
  • Basis — the provider id + model + params the fill came from (0006 R-EDIT-4 mould), so a re-run is explainable and the Phase-4 learner can relate an override to what the machine produced. Never baked into the instruction as pixels.

Reversibility is free: reset <frame> clears Removals (and the orchestration prunes the now-orphaned .krites/removals/<frame>/ rasters); the original is never touched (R-ND-2). Removals are never written to XMP (0006 R-XMP-1b — Adobe can't represent them; krites-export path only).

4. The Inpainter provider seam (0001 §5, R-RM-1..2)

A narrow, provider-neutral interface in pkg/remove — the fifth seam alongside Decoder, FaceAnalyzer, AestheticScorer, Clusterer:

// Inpainter fills a masked region of an image. Provider-neutral: a local LaMa
// ONNX adapter and a cloud diffusion backend satisfy the same contract (0001 §5).
type Inpainter interface {
    // Inpaint fills the masked pixels of img and returns the filled result.
    // The mask is 1 = fill, 0 = keep. Honours ctx (R-GLOBAL-8 timeout).
    Inpaint(ctx context.Context, img image.Image, mask image.Image) (image.Image, error)
}
  • Injected, faked in tests — no package-level hooks (CLAUDE.md); a fake Inpainter (e.g. fills the mask with a flat colour) drives every unit/e2e test of the orchestration, command and export path, so the whole feature is testable with no model present — the 0007 "ships inert" discipline.
  • Local-first, cloud opt-in (0001 §5, Q3, R-RM-2, R-PRIV-2): the local LaMa ONNX adapter is the default backend; a cloud diffusion backend is a per-capability opt-in selected in config, and when selected the remove command discloses that pixels leave the machine before sending (R-PRIV-2), with the studio's persistent privacy indicator naming the operation (R-UI-19). Secrets stay in the keychain (R-PRIV-3).
  • Timeout — every Inpaint call carries a context deadline (R-GLOBAL-8); a timed-out/failed fill surfaces as a command error and records nothing (the instruction is only written on a materialised result).

The LaMa adapter (pkg/remove/onnx) reuses the 0004 ORT scaffolding: the purego-loaded bundled dylib, the CoreML execution provider on Apple Silicon (0001 §13-Q6), the session/tensor plumbing from pkg/face/onnx. LaMa is object-removal-class, not a generative scene editor (0001 §13-Q1) — the interface lets a cloud diffusion backend slot in later without call-site changes.

The chosen backend (Q1 investigation, 2026-07-04)

  • Default: LaMa (big-lama, Apache-2.0), the reference-quality feed-forward eraser. We ship the lama_fp32.onnx (opset-17, Carve-Photos export re-hosted by opencv/inpainting_lama, Apache-2.0). A real-frame A/B (bench / stone / sign / beach) confirmed LaMa reconstructs cleaner backgrounds than MI-GAN; both degrade on large removals over fine directional texture — a known limit, mitigated by the crop strategy below.
  • CoreML-friendly. The recommended lama_fp32.onnx implements its Fourier layers as matmul-DFT (no ONNX DFT/FFT node), so it runs on the CoreML EP without the CPU fallback a real-FFT graph would force. (The dynamo lama.onnx, opset-18, does carry real DFT nodes — avoid it; it's slower and CPU-bound on CoreML.) End-to-end CoreML partition placement is unverified until profiled on a Mac (R-MLR follow-up).
  • Fixed 512×512. The adapter owns the geometry: for a localised removal it runs a mask-bounded 512 crop (padded), composites the fill back with a feather — sharper than whole-image-downscaled-to-512 and it sidesteps tiling. Large/scattered masks fall back to overlap-tiled 512 with feather-blend.
  • Self-hosted, pinned artefact (R-MLR-4). We do not depend on the HF repo at runtime: mirror lama_fp32.onnx to our own release asset, pin its SHA256 + source URL + licence in the provider config (the pkg/face/models.Model{URL, SHA256} pattern), and self-export once from big-lama in a pinned env as a provenance/repro check (validate against the mirror on fixtures). Carry the Apache-2.0 LICENSE + Samsung ©2021 NOTICE + the Carve/FourierUnit modification note.
  • MI-GAN to follow (R-ASC-style additive slice): a fast, tiny (28 MB), MIT secondary behind the same seam — but its shipped HF artefact carries no licence, so we self-export from the MIT Picsart-AI-Research/MI-GAN repo and attach the MIT text ourselves; never source the GPL-3.0 lxfater mirror.

4.1 Licensing caveat — Places2 training data (MUST document in code + docs)

Both LaMa and MI-GAN are trained on Places2, whose dataset terms are non-commercial / no-redistribution. Whether trained weights inherit their training data's licence is legally unsettled; upstream (Samsung) nonetheless released the LaMa weights under Apache-2.0. For krites' private, self-hosted, single-user use (Hailey) this is immaterial. It becomes a real question only if krites is ever distributed commercially.

Decision (2026-07-04, Matt): proceed with LaMa, and flag this prominently in both the documentation and the code (a NOTICE/attribution file plus a comment at the model-registry entry and the adapter) — establishing good-faith disclosure and temporary cover. If a takedown/removal request is ever received, we comply. Matt is raising the training-data-taint question for further (legal) investigation separately; this spec does not block on it. This obligation is a definition-of-done item for slice 2 (the adapter is not complete without the code + docs flag).

5. The remove command & export compositing

5.1 krites remove <frame> --region <x,y,w,h> (R-RM-1, R-MCP-2)

Scaffolded with gtb --ci generate command --name remove --agentless. It:

  1. resolves the frame + its full-resolution image (via the Decoder — a keeper is RAW-decoded, 0001 §13-Q4);
  2. builds the mask from --region (normalised);
  3. runs the Inpainter with a timeout (disclosing first if cloud-backed);
  4. writes the bounding-box result patch to .krites/removals/<frame>/<id>.png;
  5. appends a Removal to the edit record (reversible).

It is gated off MCP (R-MCP-2): remove carries setup.ExcludeFromMCP at generate time (gtb disable mcp remove + rebuild — never hand-edit the generated cmd.go). It is pixel-producing / destructive-feeling, so it never appears as an MCP tool.

5.2 Export bake order (R-EXP-1, R-EXP-4)

Removals composite first, before the existing geometry/colour bake, so filled pixels are straightened, cropped and graded along with everything else. The pure-Go bake (pkg/develop/bake) becomes:

removals → straighten → crop → look

Each removal composites its cached patch at its normalised box onto the working image — a pure hard raster paste, no model call. The patch is the model's fill for the whole masked box and the backend is trained to blend at the mask boundary, so the box edge is not feathered: feathering it would blend the fill back toward the original object just outside the box and re-reveal a ghost of it (overlaps likewise resolve by record order, later wins). Any boundary softening belongs inside the adapter, over inpainted content (a dilated-margin fill is a possible slice-2 refinement, §4). Export stays the only pixel-producing command (R-EXP-2) and stays deterministic (R-EXP-4): given the same edit record and cached rasters, the bake is reproducible (R-GLOBAL-7).

6. Implementation plan (the layered build)

Ordered so value lands early and the model is the last dependency, per the 0007 discipline (seam + integration ship first, adapter when the model is sourced):

  • Slice 1 — the pure spine (no model). ✅ Landed 2026-07-04. Removal on the edit record + YAML round-trip; the Inpainter interface + pkg/remove orchestration (Fill); the remove command wired to the provider seam and gated off MCP; sidecar patch storage (shoot.WriteRemoval/ReadRemoval/ PruneRemovals); export compositing (hard paste, removals → straighten → crop → look); reset pruning. Unit-tested with a faked inpainter + an e2e feature (features/remove.feature) pinning validation, off-MCP decline and the non-destructive guarantee. Inert until a backend is enabled — mirrors 0007 Layer 1.
  • Slice 2 — the LaMa adapter (pkg/remove/onnx). ✅ Landed 2026-07-04. LaMa lama_fp32.onnx, SHA256-pinned (R-MLR-4, onnx.BigLaMa); the pure mask-bounded-512 crop → tensors → reassemble geometry (composites the fill back into only the masked pixels — no export-side feather); the native Inpaint behind ONNX Runtime; provider wiring in inpaintprovider.Build with the model fetched/verified on enable and the basis stamped from the built provider. Env-gated integration test (INT_TEST=1 + KRITES_TEST_ORT_LIB/ KRITES_TEST_LAMA_MODEL) run green end-to-end (object removed, unmasked pixels byte-identical). The §4.1 Places2 licence flag ships in code (onnx.BigLaMa comment) + docs (NOTICE.md, the remove page). Decisions: (i) the adapter keeps its own minimal ORT session rather than sharing pkg/face/onnx's unexported runtime — isolates the new native code from the verified face path; a shared-runtime extraction is a clean later refactor. (ii) the model is pinned at the Carve/OpenCV HF URL for now; mirroring it to a self-hosted artefact and re-pinning is a hardening TODO before a released build. A once-off self-export provenance check remains outstanding. Incorporates a high-effort code-review pass: a mask-threshold mismatch (a sub-threshold mask silently no-op'd as "filled") and a provenance-id bug (a remove.model override was stamped with the pinned id) are fixed and tested. Open follow-up the review confirmed: the adapter's own ORT session duplicates pkg/face/onnx's runtime — a shared-runtime extraction is worthwhile once both backends are exercised together (verifiable locally, cached lib + face models).
  • Slice 3 — MI-GAN secondary (additive, same seam). ✅ Landed 2026-07-05. The fast/tiny (28 MB) backend behind the same Inpainter seam, selected with remove.backend: migan (default stays lama). The pinned artefact is migan_pipeline_v2.onnx (onnx.MIGANPipeline, SHA256-pinned, opset-17) — the model produced by the MIT repo's own scripts/create_onnx_pipeline.py, so it owns all its 512 geometry internally (bbox crop → resize → normalise → generator → denorm → resize back → feathered blend) and takes/returns full-res uint8 image+mask→result tensors. The adapter (pkg/remove/onnx/migan.go) is therefore thin: pack the frame + the inverted mask (MI-GAN polarity is 255 = keep, 0 = hole), run, and composite only the hole pixels back over a copy of the original — preserving the byte-identical-outside-mask guarantee (§5.2). remove.backend selection wired through inpaintprovider.Build (covers both the CLI and the studio, which share it), the basis stamped Provider: "migan"; an unknown backend is a clear error. Unit-tested with a fake run (pack/invert/composite, empty/sub-threshold no-op, output-size guard) + an env-gated integration test (KRITES_TEST_MIGAN_MODEL). The §4.1 Places2 caveat and the MIT-vs-re-host licence note ship in code (onnx.MIGANPipeline), docs (NOTICE.md, the remove page) and here. Decision (2026-07-05, Matt): pin the andraniksargsyan/migan HF re-host now (real SHA256 → the integration test is runnable today), mirroring how slice 2 temporarily pins the Carve HF LaMa. The re-host carries no LICENSE, so the DoD hardening TODO is to replace the pin with our own self-export from the MIT repo (create_onnx_pipeline.py on the Google-Drive weights), mirrored to an artefact we control and shipped with the MIT LICENSE — never the licence-less re-host, and never the GPL-3.0 lxfater mirror. Same Places2 training-data caveat as LaMa applies.
  • Slice 4 — the studio removal UI (R-UI, 0003): brush/box a region → preview the inpaint (preview-resolution, throwaway) → accept (materialise full-res) / reject. The persisted result for export is always full-resolution (§1).
  • 4a — the backend endpoint. ✅ Landed 2026-07-04. POST /api/v1/shoots/ {id}/frames/{frame}/remove with {region, accept}: runs the fill through the Inpainter seam (a Server option, faked in tests), composites via the same bake.Bake the export uses, returns a loupe preview JPEG; accept caches the full-res patch + records a reversible Removal, preview is throwaway; no backend → 501. httptest-covered; served on the authenticated studio (0011).
  • 4b — the Svelte brush/preview UI. ✅ Landed 2026-07-04. A "Remove object" tool on the loupe: it shows the whole frame (contain) and a drag surface; a box calls the endpoint (accept=false), swaps in the inpaint preview, and offers Accept / Redraw / Cancel. lib/region.js maps the drag to a normalised region (letterbox-aware, unit-tested). 4b-1 (WYSIWYG previews): the studio preview bakes the edit record (versioned cache), so an accepted removal shows across reloads. Verified end-to-end in a headless browser (drag → real LaMa inpaint → preview → accept). Also fixed a 0011-introduced regression: GTB's default 10s WriteTimeout truncated the inpaint and the SSE cull stream — disabled on the loopback studio.
  • Additive later — cloud diffusion backend: a config flip + new adapter behind the same seam, with R-PRIV-2 disclosure; not a rewrite.

7. Non-goals (this spec)

  • A generative scene editor / relight / object insertion — LaMa removes; it does not synthesise new content (0001 §13-Q1). Out of scope.
  • Auto-detected distractions ("remove all exit signs") — removal is user-directed in Phase 3; automatic suggestion is a possible Phase-4 idea, not here.
  • Retouch (0001 §4.4) — the bounded blemish/skin set is a separate Phase-≥2 workstream (R-RT-1), not folded in here despite sharing the edit record.
  • Writing removals to XMP — impossible in Camera-Raw (0006 R-XMP-1b); krites-export path only.

8. Open questions (resolve or defer before the gated work)

  • Q1 — inpaint model provenance. ✅ RESOLVED (2026-07-04, Matt): LaMa (self-hosted, pinned, Apache-2.0 lama_fp32.onnx) as the default; MI-GAN to follow. A four-thread investigation + a real-frame A/B settled it (§4, "The chosen backend"): (i) nothing in the 2023–25 landscape beats LaMa/MI-GAN for a feed-forward, ONNX, permissively-licensed on-device eraser (the rest is diffusion — heavy/NC/not-ONNX — or NC feed-forward); (ii) LaMa out-reconstructs MI-GAN on real frames; (iii) the recommended opset-17 export is matmul-DFT, so it is CoreML-friendly (no FFT-op CPU fallback); (iv) it is fixed-512, handled by a mask-bounded-crop adapter; (v) we self-host a pinned SHA256 mirror and self-export once as a provenance check. Residual risk — the Places2 training-data licence — is accepted as a documented caveat (§4.1), not a blocker, with takedown-compliance and a separate legal follow-up.
  • Q2 — result storage granularity. ✅ RESOLVED (2026-07-04, Matt): the bounding-box patch + offset. LaMa only alters the masked pixels, so the patch is the delta — storing it records exactly what changed (the non-destructive philosophy), keeps the sidecar lean (a local removal is <1 MB vs. a 10–40 MB full-frame PNG at RAW resolution), and lets multiple removals on one frame each carry a small patch. Cost: the box edge is hard-pasted (the backend blends the boundary; the export composite must not feather it, §5.2), and a scattered mask degenerates toward full-frame — both acceptable (§3, §5.2).
  • Q3 — mask coordinate frame vs. develop geometry. ✅ RESOLVED (2026-07-04, Matt): original-frame normalised coords, composited first (before straighten/ crop). The mask is defined against the immutable original, so it survives any later straighten/crop re-edit; it shares the one coordinate space Crop/straighten already use; and removing before geometry means the fill is leveled/cropped/graded with the rest of the frame (§5.2 bake order removals → straighten → crop → look). Cost: inpaint runs at full original resolution (also the quality-correct place) and a later crop may waste a removal — both minor.
  • Q4 — Q6 sub-detail (0001 §13-Q6). Exact M-series chip + RAM only affects how aggressively local LaMa runs vs. degrades to cloud; it does not gate the design (the seam supports both). Track, don't block.

Nothing in slice 1 is gated — the pure spine can be built against a fake the day this DRAFT is accepted. Q1 gates only slice 2.