Sign your own binaries with go-tool-base, part 7: rotation and break-glass

Tue, 23 Jun 2026 00:00:00 +0000

Most signing guides stop the moment the first release goes out the door, which is a shame, because the question that keeps you up at night comes later: what do you do when the key has to change? Keys get rotated on a schedule, keys get compromised, and one day you’ll want to move off RSA onto something newer. This last part of the signing series covers the bit everyone skips, swapping a signing key out from under a fleet of installed tools without locking a single one of them out, and the break-glass key for the day the primary is gone.

By Part 6 you’ve a pipeline signing every tagged release through a KMS key, a public key published over WKD, and that same key baked into the binary as its trust anchor. It all works. The catch nobody mentions is that you’ve now got a key embedded in software sitting on other people’s machines, and that key is the one thing in the whole chain you can’t quietly change. So let’s plan for changing it properly.

Why there’s no auto-rotate button

If you’ve used KMS for encryption, you’ll know it can rotate keys for you on a yearly tick. Asymmetric SIGN_VERIFY keys don’t get that, and the terraform-aws-signing-kms module sets enable_key_rotation = false on purpose. That’s not an oversight to work around; it falls straight out of how the key is built. The private half of a signing key never leaves the HSM and has no export path at all, so there’s no mechanism by which AWS could hand your verifiers a rotated public half and keep the old one verifiable. KMS auto-rotation works for symmetric keys precisely because you never see the key material; the instant you need a public key your clients pin against, rotation stops being something a cloud provider can do behind your back.

So rotation here is a runbook you maintain, not a checkbox you tick. That sounds like the worse deal until you weigh it: you’ve got a key with no exfiltration path, in exchange for rotating it by hand on the rare occasions you must. For a release-signing key that’s exactly the right trade. You don’t want this key rotating itself; you want to be standing right there when it happens.

Rotate by minting a new key, never by changing the old one

Here’s the rule that makes everything else fall into place. You do not edit the existing key. The module’s key_spec and name are immutable by design, and that immutability is a feature: the v1 key is a fixed point that stays verifiable while you stand up its replacement next to it.

So you mint a second key. A second module instance, named acme-release-signing-v2, with its own alias alias/acme-release-signing-v2:

module "signing_kms_v2" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v2"
 oidc_provider_arn = data.aws_iam_openid_connect_provider.gitlab.arn

 ci_subject_filters = [
 "project_path:acme/acme-cli:ref_type:tag:ref:v*",
 ]

 key_administrator_arns = [/* ... */]
 automation_role_arn = data.aws_iam_role.automation.arn
}

Apply that and the v1 key carries on signing, undisturbed. Now mint its public half exactly the way you minted v1 back in Part 4, only the alias changes:

gtb keys mint \
 --backend aws-kms \
 --kms-region eu-west-2 \
 --key-id alias/acme-release-signing-v2 \
 --name "Acme Releases" \
 --email release@acme.dev \
 --created "2026-06-08T00:00:00Z" \
 --output signing-key-v2.asc

You’ve now got two real signing keys, both live, neither one a threat to the other. The whole rotation is the careful business of moving traffic from one to the other while every installed client keeps verifying.

The dual-publish, dual-sign window

This is the part that does the work, and it leans on one fact about how the verifier treats your keys. The trust anchor isn’t a single key, it’s a set: v1, v2, and the rotation authority all sit in internal/trustkeys/keys/ and all get embedded together. A release verifies if any key in that set validates its signature. That’s what makes a handover window possible, because during it a client might hold v1, or v2, or both, and verify happily whichever it has.

So you run an overlap. Three moves, in order.

First, publish both public keys over WKD. Same email bucket, same command you already know from Part 4, just with v2 added to the file list:

gtb keys wkd \
 --domain acme.dev \
 --email release@acme.dev \
 --output ./wkd-staging \
 signing-key-v1.asc signing-key-v2.asc rotation-authority.asc

Both signing keys share release@acme.dev in their UID, so they land in one hu/ bucket together, and the rotation authority rides along as it always has. Deploy that staging tree the way you deployed it before. Now the WKD endpoint serves the new trust anchor, so a client that fetches keys on its next update picks v2 up automatically.

Second, embed v2 alongside v1 and ship a release. Drop signing-key-v2.asc into internal/trustkeys/keys/ next to v1, cut a release, and that build now ships knowing about both keys. Installed clients pick the new trust anchor up as they update through the window. This is the slow bit, and it should be: you want the new key spread far and wide before you stop signing with the old one.

Third, sign with v2, wait, then retire v1. Point the pipeline’s GTB_SIGNING_KEY_ID at alias/acme-release-signing-v2 so new releases are signed by the new key. A client that’s already updated verifies against the embedded v2; one that’s lagging still has v1 in its set and, because you’re still publishing v1 over WKD, can still fetch and trust it. Nobody’s locked out.

Leave that overlap running long enough that you’re confident the slow movers have updated, weeks, not hours, depending on how often your users actually run the thing. Only then do you retire v1: drop its module instance, pull signing-key-v1.asc from the embed directory and the WKD file set, and ship a release that knows only about v2. The rotation is done, and at no point did an installed tool see a signature it couldn’t check.

The break-glass key, for when there’s no handover

Everything above assumes the old key can hand over to the new one: it’s still there, still able to sign, and you’re rotating on your own terms. The nightmare is the other case. The KMS key is gone, access revoked, or you’ve reason to think it’s compromised and you daren’t sign anything with it again. There’s no handover, because the thing that would do the handing is exactly what you’ve lost.

That’s what the rotation-authority key is for, and it’s why Part 4 had you mint it the moment everything else was calm. It’s an offline ed25519 key, generated once on a trusted machine, and the go-tool-base how-to, generate the rotation-authority key, walks the storage in detail. The short version: the private half never lives on a networked box. You print a paper backup with paperkey, write it to an encrypted USB stick, and the two go in a safe; the local copy gets shredded. The how-to even has you type the paper backup back in once before you walk away, because discovering your printer ate a stripe of pixels is a problem you want now, not eighteen months from now when the building’s on fire.

The public half, though, has been in your trust set the whole time, embedded in the binary and served over WKD right alongside the signing keys. That’s the trick. Because every installed client already trusts the rotation authority, it can vouch for a brand-new signing key outside the normal sign-with-the-old-key path. You bring the private half out of the safe, use it to authorise the new key, ship that, and installed tools adopt the replacement on their next update, all without the dead primary key ever having to sign a thing.

It is, deliberately, a key you hope never to touch. But a break-glass key you forgot to cut is just a pane of glass.

The whole chain, end to end

That’s the series. Step back and the shape is one clean line: a signing key is born inside KMS and never leaves it (Part 2); its public half is minted out of the HSM without the private bytes ever surfacing (Part 4); that public key is published off-platform over WKD, somewhere your release host can’t quietly rewrite (Part 4), and embedded into the binary as a required trust anchor (Part 5); every tagged release is signed through the key over short-lived OIDC credentials with no stored secrets (Parts 3 and 6); a stranger’s copy of your tool verifies its own updates against that anchor before trusting a byte (the cross-check); and when the day comes, the whole thing is rotatable without locking anyone out. A key that can’t be stolen, can’t be forged, and can still be replaced.