Skip to content

0035 — Sign krites release binaries (self-update trust anchor)

  • Status: IMPLEMENTED (2026-07-27) — both phases complete. Phase A: key provisioned (§4), minted and published to WKD, signing wired, cert embedded and fingerprint-pinned, and v0.5.2 shipped as the first signed release (verification evidence in §8). Phase B: require_signature is on, so self-update now fails closed — the poisoned well is shut. The posture is pinned by a test (§6) rather than trusted to generated code.
  • Key identity: fingerprint 329F3314FA1D317EDFCCABC28E45CA53DB150302, created 2026-07-26T10:32:32Z (⚠️ pin with --created on any re-mint or the fingerprint changes), UID krites Release Signing <[email protected]>.
  • WKD: bucket hu/8iqmwgrtibiqphio1rr4da61y9qpgbo7 serves exactly that cert; all five pre-existing buckets verified byte-identical to production afterwards (freeze intact).
  • RA cert deliberately absent — and that is forced, not chosen: WKD groups certs by UID↔email match, and the rotation-authority cert's UIDs are release@/release-v2@, so it cannot enter this bucket without an offline RA ceremony. Same as krites-models. This resolves §7-Q1's sub-question.
  • Owner:
  • Realises: closes an unauthenticated self-update path. krites' update command currently installs a downloaded binary with no cryptographic anchor — the poisoned-well risk.
  • Depends on: infra (the krites release-signing KMS key + signer role — ✅ provisioned 2026-07-26, §4), 0034 (the rotation whose tooling and lessons this reuses), and the estate WKD host (Cloudflare Pages, operator-deployed).
  • Reference implementations: go-tool-base (the consumer + producer pattern, and the gtb enable signing generator) and krites-models (0034, the KMS/OIDC release-signing job).

1. The risk

krites inherits GTB's update command, which downloads a release asset and checks its SHA-256 against the release's own checksums.txt. Both come from the same place, so an actor able to publish or tamper with a release controls the binary and the checksum that "verifies" it.

GTB already supports fixing this — and krites has none of it wired:

Control krites today
Embedded trust key (WithEmbeddedKeys) absent — no internal/trustkeys package
properties.signing in .gtb/manifest.yaml absent
update.require_signature unset (defaults false)
signs: block in .goreleaser.yaml absent — releases are Apple-notarized only

The consequence is explicit in setup.verifyManifestSignature:

if s.keyResolver == nil {
    if s.requireSignature { return errors.New("…no signing key is configured") }
    return nil          // ← no resolver, not required: update proceeds UNVERIFIED
}

So today every krites self-update takes the second branch. (macOS notarization covers the .app bundle on that platform only, and is Apple's supply chain, not ours — it is not a substitute for signing the release manifest we publish.)

2. Design

Mirror the chain that already works twice in this estate:

release (tag) ──► goreleaser ──► checksums.txt
                                   └─► gtb sign --backend aws-kms  ──► checksums.txt.sig
                                        (private key never leaves KMS; OIDC-assumed)

krites update ──► fetch checksums.txt + .sig
                   └─► verify sig against the EMBEDDED public key   ──► then trust the SHA
  • Producer: a signs: block in .goreleaser.yaml invoking a scripts/sign-release.sh that shells to gtb sign, plus OIDC + AWS_ROLE_ARN / GTB_SIGNING_KEY_ID on the release job.
  • Consumer: the public half embedded via internal/trustkeys, surfaced to the SelfUpdater as props.SigningConfig{EmbeddedKeys: …}, with update.require_signature eventually true so an unsigned or unverifiable update fails closed.
  • Generated, not hand-written. gtb enable signing does all of the wiring (manifest flag, internal/trustkeys scaffold, root-command wiring, and the GoReleaser signs block when --key-id is recorded). Per CLAUDE.md we must not hand-edit generated files.

2.1 v2-only — no dual-sign

Unlike 0034, this needs no overlap window: krites has never shipped a signed release, so there is no older signature generation any binary expects. One key, one signature.

2.2 Two trust anchors — embedded and WKD (decided 2026-07-26)

krites runs the same composite resolver gtb does — composite[embedded, wkd:openpgpkey.phpboyscout.uk] (--key-source both). The embedded cert ships in the binary; the WKD leg publishes the same cert under the release identity so the trust anchor can be rotated or revoked without shipping a new binary, which is the whole point of having it.

The two anchors interact deliberately:

Situation Behaviour
Embedded and WKD fingerprint sets disagree Always fatal (ErrKeyResolverMismatch) — an active-tampering signal, regardless of config
WKD unreachable (404/outage) Degrades to embedded-only with a warning — require_external_crosscheck ships false, so a CDN blip never bricks updates

So "crosscheck not required" is not "crosscheck not enforced": a disagreement still fails closed; only unreachability is tolerated.

Publication is an operator ceremony, not CI. The WKD host is Cloudflare Pages deployed by hand (cd wkd-staging && wrangler pages deploy .) — a deliberate air gap so a compromised forge or runner cannot rewrite the trust anchor. Buckets are add-only, forever: never modify a published hu/ file, because shipped binaries pin its content. krites' identity is a first-ever key, so it gets a new bucket and touches no existing one.

3. Rollout — two phases, and why it cannot be one

gtb enable signing warns that --require-signature "must only be flipped once a signed release has shipped (the N+1 rollout)". That is a hard ordering constraint:

  • Phase A — sign, don't require. Ship a release whose manifest is signed, with require_signature false. Binaries already in the field have no embedded key and cannot verify; requiring it now would brick their update path. This phase is the trust-on-first-use moment — unavoidable when introducing signing to an already-distributed tool.
  • Phase B — require. Once a signed release exists, re-run gtb enable signing --require-signature so subsequent updates fail closed. Only from here is the poisoned well actually shut.

Shipping Phase B before a signed release exists would make every update fail.

4. Prerequisite — the key ✅ PROVISIONED (2026-07-26)

Done via infra MR !79, released as v0.10.0 and applied to prod (Apply complete! Resources: 3 added, 0 changed, 0 destroyed):

alias alias/krites-release-signing
key ARN arn:aws:kms:eu-west-2:060730976733:key/c4c41ea6-099f-41f2-9f18-6fb9a6e91d5e
signer role arn:aws:iam::060730976733:role/krites-release-signing-signer

Named without a version suffix: a first-ever key is not a rotation, and -v1 names are forbidden in this account (a future rotation becomes krites-release-signing-v2). It is distinct from krites-models-release-signing-v2 — that signs the ONNX model manifests, this signs krites' binaries, and ci_subject_filters stops either signing for the other. Standing up a key like this is now documented in infra/docs/how-to/mint-a-new-release-signing-key.md.

Original requirement (kept for context) `infra/src/main.signing-kms.tf` provisioned three keys — gtb, ffmpeg-wasi, krites-models — and **none for the krites binary**. Required, mirroring `module "krites_models_signing_kms"`: 1. A `signing-kms/aws` module instance for krites, with `ci_subject_filters = ["project_path:phpboyscout/krites:ref_type:tag:ref:v*"]` so only this project's release tags can sign, and `oidc_audience = "sts.amazonaws.com"`. 2. The minted **public** half exported for embedding. 3. **Naming:** per the rotation runbook, "a first-ever key for a project gets a plain identity with no `-vN` (there is no old generation to separate from)" → identity `[email protected]`. The KMS resource name is an infra decision (the existing `-v2` suffixes denote *rotated* generations, which this is not). Until that exists, the krites-side work can be staged but not exercised.

5. Work items

infra (prerequisite)DONE — key provisioned in prod (§4).

operator ceremony (WKD — air-gapped, not CI) 1a. Mint the key's OpenPGP identity (gtb keys mint, UID [email protected]) and generate its WKD bucket (gtb keys wkd); copy only the new hu/<hash> file into wkd-staging (add-only — never touch an existing bucket), then deploy by hand: cd wkd-staging && wrangler pages deploy .. Verify served sha256 == staged for every hu/ file, old and new.

krites — Phase ADONE (v0.5.2, 2026-07-27) 2. ✅ gtb enable signing --key-id alias/krites-release-signing --public-key internal/trustkeys/keys/<name>.asc --key-source both --email [email protected]. Do not pass --require-signature yet, and leave --require-external-crosscheck off (§2.2). ⚠️ This generator overwrote .goreleaser.yaml wholesale, silently dropping the krites-app build, app_bundles, dmg, the ONNX-dylib before-hook and the no-Windows exclusion. Restored in !115; upstream bug filed as go-tool-base issue #4. The signs: block is now hand-maintained — see the warning comment above it. 3. ✅ Drop the minted .asc into internal/trustkeys/keys/, asserting its fingerprint before use (the 0034 lesson: never vendor a cert without checking what it is). 4. ✅ The release job's id_tokens, AWS_ROLE_ARN, AWS_REGION — no scripts/sign-release.sh was needed in the end; the signs: block shells to gtb sign directly. 5. ✅ Bound the upload/sign network calls — the 0034 release job hung silently for a full hour for want of exactly that. 6. ✅ Cut a release; verify checksums.txt.sig is published and verifies against the embedded key.

krites — Phase BDONE (2026-07-27, the release after v0.5.2) 7. ✅ Re-ran gtb enable signing --require-signature (with the full existing posture re-passed, since the generator writes the manifest block from flags, not from what is already there). Effect: require_signature: true in the manifest and verify.DefaultRequireSignature = true in the generated pkg/cmd/root/signing.go. Documented in the config reference under Self-update signature verification. ⚠️ It clobbered .goreleaser.yaml again — this time it did log conflict detected … overwriting modified file, but overwrote regardless, destroying app_bundles, dmg, notarize and both build ids (the whole Mac packaging chain). Restored from git and verified section-by-section before committing. Always diff .goreleaser.yaml after running this generator. Upstream: go-tool-base issue #4.

6. Tests

  • Unit: trustkeys.Keys() returns exactly the embedded cert(s) and its fingerprint matches the expected value (mirrors 0034's embedded-vs-pinned guard, which catches a half-landed cert swap).
  • Release (manual gate, Phase A): download the published checksums.txt + .sig and verify against the embedded key; confirm the signer fingerprint is the intended one.
  • Behavioural (Phase B): an update whose signature is absent/invalid must fail closed. Enforcement itself lives in the framework (setup.verifyManifestSignature) and is tested there; what krites owns — and what regresses silently — is the posture it sets. So pkg/cmd/root/signing_posture_test.go asserts the four effective values: DefaultRequireSignature true, DefaultKeySource both, DefaultExternalKeyEmail the WKD identity, and DefaultRequireExternalCrosscheck false.

It asserts the effective values, not the generated file's text, which is deliberate: the generator only emits non-default values, so the posture depends on framework defaults as much as on our flags. Reading the values catches both a regenerate that drops --require-signature and an upstream change to a default beneath us — neither of which would show up as a diff in hand-written code. Verified to fail by removing the generated line and re-running.

7. Open questions

  1. Trust-anchor source RESOLVED (2026-07-26): both — embedded **and WKD.** That is what the WKD leg exists for, and the estate's IaC + gtb keys tooling already make it routine (§2.2). require_external_crosscheck stays false so an outage degrades rather than bricks; a fingerprint disagreement remains always-fatal.
  2. Scope of signing — sign the checksums.txt manifest only (what self-update verifies), or also publish per-artifact signatures? Manifest-only matches gtb/krites-models and is what the verifier consumes.
  3. Does macOS notarization stay as-is? RESOLVED (2026-07-27): yes, unchanged. The v0.5.2 run performed both — cross-platform notarize of the darwin binaries, native notarize of the .dmg, and the OpenPGP signature — with no interaction between them. They are orthogonal chains (Apple's, and ours) and neither duplicates the other.

8. Phase A verification evidence (v0.5.2, 2026-07-27)

The full trust chain was verified against the published release, not the job's self-report:

Check Result
checksums.txt.sig published ✅ alongside all 5 artifacts + dmg
Signature packet count 1 (correct — first-ever key, no dual-sign; §2.1)
Issuer fingerprint 329F3314FA1D317EDFCCABC28E45CA53DB150302
Verifies against embedded cert GOODSIG + VALIDSIG
WKD bucket cert fingerprint identical to embedded → resolver agrees, no ErrKeyResolverMismatch
WKD content is a real keyring ✅ leading c6 public-key packet, not the catch-all HTML a missing bucket serves
Signed manifest ↔ artifact krites_Linux_x86_64.tar.gz: OK

So the chain WKD ∥ embedded → signature → checksum → artifact is closed and confirmed.

8.1 The release job is the fragile part, not the signing

Signing itself worked first time it was reached. Getting to it took six attempts, none of which failed for a signing reason:

Cause Fix
Windows target won't compile (ONNX loader needs POSIX symbols) config restored (!115)
gtb built into the repo → goreleaser "dirty state" build into $TMPDIR
60-minute default timeout on a cold cache timeout: 2h, declared on the job so it travels with the tag
Git clone reset mid-fetch (twice) GET_SOURCES_ATTEMPTS: "3" (!117)
Runner terminated mid-job (Terminated: 15) runner restart; see below

Two observations worth keeping for the next release. First, a 13-minute stall with zero output appeared mid-run between setting defaults and running before hooks, and a separate attempt spent 16 minutes on a clone that normally takes 2 seconds — both consistent with the macOS runner host suspending rather than with a slow link. If release jobs keep dying oddly, run the runner under caffeinate -dimsu before investigating anything else. Second, a gitlab-runner restart SIGTERMs any in-flight job, so restarting the runner to "fix" a stuck release is what kills that release — check for a running job first.