Posts on PHP Boy Scout

Two bugs that taught me the rules

Wed, 20 May 2026 00:00:00 +0000

Some bugs are interesting because they’re subtle. These two were interesting because they were the exact opposite… in each case the tool had a hard rule I simply didn’t know about, and its error message couldn’t be bothered to tell me what that rule was. Both came out of building the infrastructure toolchain, both cost me a good deal more time than they had any right to, and both are the sort of thing that looks blindingly obvious the moment you know it and utterly baffling until you do.

So here they are, written down, partly to save you the bother and partly so I don’t go and forget them myself.

Bug one: the rule-less job that skips your merge requests

The cicd gate components, in their first cut, shipped with no rules: block. They were dead simple jobs: lint, scan, validate. No conditions, because they should just always run. Obviously.

They ran on branch pipelines. On merge requests, they didn’t run at all! The gates that were the entire point of the components were simply absent from the one place you’d most want to see them… the merge request.

The cause is a GitLab CI rule that’s remarkably easy to go years without ever learning: a job with no rules: block runs only on branch and tag pipelines. It does not run on merge-request pipelines. So “no conditions” doesn’t mean “runs everywhere” at all. It means “runs everywhere except a merge request”, which is about the least intuitive default I can think of.

The fix is faintly absurd, and that’s exactly what makes it stick. You add an unconditional rule: rules: [{ when: on_success }]. The content of that rule does precisely nothing. It always matches. What actually matters is that the job now has a rules: block at all, because merely having one is what makes a job eligible for merge-request pipelines. A rule whose content is meaningless, added solely so the block exists. That’s the fix. I’ll admit I stared at it for a moment.

Bug two: the import block that only works at the root

The second one came from terraform-aws-security-baseline. The account-hardening module needed to adopt a resource that already existed in the account, which is exactly what OpenTofu’s import {} block is for. So an import block went into the account-hardening module, right next to the resource it was adopting. The natural home for it, surely.

OpenTofu disagreed, and rejected it outright. The rule: an import block is only allowed in the root module. It can’t live inside a child module. A module that wants one of its own resources imported can’t declare that import itself… the import has to be declared up at the root, and the root caller does the adopting.

The fix was to take the import block out of the module and document caller-side adoption instead. The module describes the resource, and the root configuration that calls the module is where the import actually lives.

Two unrelated bugs, in two completely different tools, and the same shape sitting underneath both of them.

In each case the tool has a hard structural rule. Where a block is allowed to live. What makes a job eligible for a particular kind of pipeline. And in each case the error told me the tool was unhappy without telling me which rule I’d broken, so the obvious next move (debugging my own logic) was the wrong move entirely. There was nothing wrong with the logic. The thing was simply in a place the tool doesn’t allow, or missing a block the tool quietly insists on.

The lasting lesson here isn’t the two specific rules, useful as they are to know. It’s the reflex. When something that should obviously work just doesn’t, and the error is unhelpful, stop debugging your logic and start suspecting a structural rule about where something is allowed to be, or whether a thing is eligible in the first place. GitLab CI and OpenTofu both have a handful of these, and you mostly learn them the hard way, by tripping over them. Knowing the shape of the category at least means the next one costs you an hour instead of a whole afternoon.

Worth remembering

Two bugs from building the toolchain, one shape. A GitLab CI job with no rules: block runs on branches and tags but silently not on merge requests, and the fix is an unconditional rules: block whose content does nothing and whose mere existence is the entire point. An OpenTofu import block gets rejected inside a child module, because imports are only legal at the root, so the caller adopts and the module just describes.

Neither error named the rule it was enforcing, and that’s the category to watch for. When sound logic fails against an unhelpful error, suspect a structural rule about where a thing may live or whether it’s even eligible… not a bug in what you actually wrote. It’ll save you an afternoon. It certainly cost me a couple.

Reviewed, then applied

Mon, 18 May 2026 00:00:00 +0000

The genuinely dangerous moment in infrastructure-as-code isn’t the apply. It’s the gap between the plan a human read and approved, and the change that actually runs a moment later. If those two are different computations (and by default they are) then nobody really reviewed the thing that touched your account. The infra repo closes that gap from both ends.

The gap between “reviewed” and “ran”

Here’s the moment in infrastructure-as-code where things go wrong.

Someone opens a merge request. CI runs tofu plan and the output is there to review: these three resources change, this one is destroyed. A human reads it, decides it’s correct, approves, merges. Then apply runs.

The trap is in what apply actually applies. If apply does its own fresh tofu plan and then applies that, the change that runs is not necessarily the change that was reviewed. State can have moved. A provider can have drifted. Someone else can have applied something in between. The reviewed plan and the applied change are two separate computations done at two different moments, and every difference between those moments is a change nobody looked at.

infra closes that gap from both ends.

Plan as an artifact

The first end is making the reviewed plan and the applied plan the same object.

The tofu-plan component runs the plan and saves it. It writes tfplan.cache, OpenTofu’s binary plan file, as a CI artifact. It also writes tfplan.json, which GitLab renders as a plan widget right in the merge request: the add, change and destroy summary, there to review without leaving the MR.

The tofu-apply component then does not re-plan. It applies that saved tfplan.cache. And OpenTofu itself enforces the safety net: applying a stale plan file, one captured against a state that has since moved, is rejected by the tool. So what reaches the account is provably the plan that was reviewed, or it’s nothing at all. There’s no third option where something unreviewed slips through.

Applying is a human decision

The second end is when apply runs.

infra is trunk-based: it dropped the develop branch and works on main. But a naive trunk setup auto-applies every push to main, which means there’s no human gate at all, just whatever the last merge happened to contain.

So the gate is built explicitly. releaser-pleaser keeps a release merge request open against main. Ordinary merges to main run plans but apply nothing. The apply happens only when a person merges the release MR. Merging it cuts a release tag, and the tag pipeline is what runs tofu-apply, against the plan banked by the latest main pipeline.

The effect is that the act of applying to the account is the deliberate, visible act of merging the release request. Nothing reaches the account because a commit landed. It reaches the account because a person decided a release should go out and merged it. (Which, after the accidental v2.0.0 that kicked off the whole GitLab move, is a discipline I’d freshly relearned the value of.)

The guard on the gate

There’s one more piece, because a gate is only as good as its precondition.

A verify-main-plan job blocks the release MR from being mergeable unless the latest main pipeline is green. You can’t cut a release, and therefore can’t apply, on top of a main whose plan didn’t even succeed. The human gate has its own gate: the thing you’re about to merge has to be standing on a known-good plan before you’re allowed to merge it.

The bottom line

The risk in infrastructure-as-code is the gap between the plan a human reviewed and the change that runs, because a re-plan at apply time is a different computation from the one that was approved.

infra closes it twice over. tofu-plan saves the plan as a tfplan.cache artifact and renders it as a merge-request widget; tofu-apply applies that exact artifact, and OpenTofu rejects it outright if the state has moved underneath it. And applying is gated on a human merging a releaser-pleaser release request, not on a push, with a verify-main-plan check making sure that request can only be merged on top of a green plan. What gets applied is what was reviewed, when a person decided it should be.

One graph, not micro-stacks

Sun, 17 May 2026 00:00:00 +0000

Once an infrastructure repo has a few concerns in it (account hardening, the security baseline, the signing stack still to come) there’s a steady pressure to split them into separate stacks with separate state, and Terragrunt is right there to help you do it. The infra repo keeps everything in one OpenTofu graph instead. The reason comes down to who enforces your dependency ordering: the engine, or you.

The pressure to split

The infra repo’s src/ has several concerns in it, and more coming, the signing stack among them. Once a repo reaches that point, there’s a steady pressure to split: one stack per concern, each with its own state file.

It’s an appealing pressure. Separate stacks feel modular. Each apply touches less, so the blast radius of any one run is smaller. And Terragrunt exists, popular and well-regarded, precisely to orchestrate a fleet of separate stacks. The path is well trodden.

infra didn’t take it. src/ is a single OpenTofu root stack: each concern is a module block, in its own main.<concern>.tf file, all sharing one state and one graph.

What one graph gives you

The thing a single graph gives you is engine-enforced truth about ordering and data.

Inside one OpenTofu graph, the tool builds the full dependency DAG itself. When the signing stack needs a value the security baseline produced, you reference it directly, module.baseline.something, and OpenTofu guarantees two things: the baseline is created before the thing that depends on it, and the value handed across is the current one from this same apply. Ordering and data-passing aren’t things you arranged. They’re facts the engine checks and enforces, every plan, every apply.

What splitting costs

Split src/ into per-concern stacks with separate state, and that guarantee is the thing you spend.

Now one stack reads another’s outputs through terraform_remote_state. That’s a lookup of a snapshot: the other stack’s last applied state, whatever it was, whenever that was. It’s not a live edge in a graph. Ordering is no longer enforced by the engine either; it becomes something you arrange yourself, in CI stage sequencing or in Terragrunt’s own dependency blocks.

That’s the trade, stated plainly. You give up a strong, engine-checked guarantee, and you buy back a weaker, hand-arranged imitation of it. Terragrunt is a good tool for managing that weaker world tidily. But the question worth asking first is whether you should be in the weaker world at all.

When splitting is genuinely right

This isn’t an argument that splitting is always wrong. Separate states genuinely earn their place when concerns have different change cadences, different access boundaries, or different teams owning them: when you actively want an apply of one to be unable to touch another, and you want different people holding different state.

infra has none of those. It’s a single account, a single operator, one cohesive set of concerns. The only thing splitting would buy here is a smaller per-apply blast radius, and that’s better handled by reviewing the plan before it applies, which the next post is about, than by fragmenting the dependency graph. So src/ stays one graph, and Terragrunt was considered and deliberately not adopted.

If ordering between graphs is ever needed

If infra ever does genuinely need more than one stack, the plan isn’t Terragrunt. It’s to keep each stack a single strong graph internally, and to sequence the stacks with CI stages. Keep the engine-enforced guarantee where it’s strongest, inside each graph, and reach for hand-arranged ordering only at the one seam where it’s unavoidable.

Boiling it down

A multi-concern infrastructure repo feels like it should be split into per-concern stacks, and Terragrunt is right there to manage the result. infra keeps src/ as one OpenTofu graph instead.

Inside one graph, OpenTofu enforces dependency ordering and passes current values across module boundaries as checked facts. Split into separate states and that becomes a terraform_remote_state snapshot lookup plus ordering you arrange by hand: a weaker version of what you gave up. Splitting is right when concerns have different cadences, boundaries or owners; for a single-account, single-operator repo none of that applies, so the strong guarantee is worth keeping, and Terragrunt is the tool for a problem infra chose not to have.

CI you include, not copy

Sat, 16 May 2026 00:00:00 +0000

Every infrastructure repo runs the same CI: lint the OpenTofu, scan it, validate it, plan, apply. The first repo, you write that .gitlab-ci.yml by hand. The second, you copy it. By the third, you’ve got three copies of the same pipeline quietly drifting apart, which is the exact problem you’d never tolerate in application code. The cicd repo is the fix, and it’s just the library-first instinct pointed at the pipeline.

The `.gitlab-ci.yml` you keep copying

The infrastructure repos in this series all run the same CI gate jobs: format and validate the OpenTofu, lint it, scan it for security issues and secrets, and on the deploy side, plan and apply.

The first repo, you write that .gitlab-ci.yml by hand. The second repo needs the same jobs, so you copy it. The third repo, you copy it again. Now there are three copies of the same pipeline, and they do what copies always do. They drift. A fix you make in one repo’s CI doesn’t reach the other two. A tightened scan rule lands in the repo you were working in and nowhere else. It’s the copy-paste problem, exactly as it shows up in application code, just written in YAML and therefore that bit easier to pretend isn’t code.

GitLab has a feature for exactly this

GitLab CI/CD Components are the answer to that problem. A component is a reusable, versioned piece of pipeline that you publish, and other projects pull in with an include: pinned to a version:

include:
 - component: gitlab.com/phpboyscout/cicd/tofu-lint@v0.5.0

That’s a library import, for pipeline. The component has a defined interface, a version, and a home in GitLab’s CI/CD Catalog. A consuming repo includes it instead of carrying its own copy, and when the component improves, the consumer moves a version pin rather than re-copying YAML.

Why a monorepo of components

The cicd repo holds all of the components together: tofu-lint, tofu-security, tofu-validate, tofu-plan, tofu-apply, and more. One project, not one project per component.

That’s a deliberate call, and the reason is how GitLab versions things. A version is a tag, and a tag belongs to a project. A component’s version is its project’s tag. So a monorepo of components, versioned together as one tag stream, is the natural unit: a consumer pins @v0.5.0 and gets a known-good set of components that were tested together, rather than juggling a separate version for each one.

Authoring discipline

A component is a file under templates/, and it opens with a spec: inputs: block: the typed inputs, their defaults, the component’s public interface.

The discipline that keeps the library usable is that a component must be consumer-agnostic. It never hardcodes a token, and it never names a particular consumer’s variable. Inputs have sensible defaults, and a consuming repo overrides them. A component that reaches out and assumes something about the repo including it is a component that works in one repo and surprises the next. An authoring guide in the repo keeps that consistent across everyone who adds a component.

The self-test you cannot fully write

The cicd repo tests its own components with a self-test pipeline. It’s worth knowing where that self-test stops.

When a repo tests its own components by running them in child pipelines, GitLab masks $CI_PIPELINE_SOURCE as parent_pipeline. A component’s rules:, which often branch on the pipeline source to behave differently for a merge request than for a branch or a tag, therefore can’t be exercised honestly by the self-test: the source they’d branch on has been flattened. The self-test covers what it can, and the component rules: are, in the end, validated by real consumers using them for real. That’s a genuine limit, and naming it is better than pretending the self-test proves more than it does. (It’s also, not coincidentally, the exact rules: quirk that bit me in one of the two bugs I closed the series with.)

The same instinct, again

This blog keeps circling the same instinct. go-tool-base exists because the same CLI scaffolding kept getting rewritten, so it was extracted into a library. cicd is that instinct pointed at the pipeline: the same gate jobs kept getting copied between repos, so they were extracted into a versioned, included library.

Stop copy-pasting. Publish, version, include. It’s true for CLI code, and it turns out to be just as true for the YAML that builds and ships it.

The gist

Every infrastructure repo needs the same CI, and copying the .gitlab-ci.yml between them produces copies that drift apart. GitLab CI/CD Components fix it: reusable, versioned pipeline that a repo include:s and pins, instead of carrying its own copy.

cicd is a monorepo of those components, versioned together as one tag stream, because GitLab tags a project and a component’s version is its project’s tag. Components are authored consumer-agnostic, with typed spec: inputs: and no hardcoded assumptions, and their rules: are validated by real use because the self-test can’t see the pipeline source. It’s the library-first instinct, applied to CI: publish it once, include it everywhere, fix it in one place.

One image for the whole toolchain

Fri, 15 May 2026 00:00:00 +0000

Every CI gate job across the infrastructure repos reaches for the same pile of tools: OpenTofu, tflint, trivy, checkov, gitleaks, terraform-docs, the AWS CLI. Installing that pile per job is both slow and quietly dangerous, because nothing pins it consistently. infra-tools is the obvious fix (one image, one source of truth for versions), but two of its build decisions are less obvious and worth a look: it publishes with crane instead of a second build, and it deliberately lets its own vulnerability scan fail.

The same pile of tools, in every repo

Every infrastructure repo in this series runs the same CI gate jobs: format and validate the OpenTofu, lint it, scan it for security problems and secrets, check the docs. Those jobs need a specific set of tools, and it’s the same set in every repo.

Install them per job and you pay twice. You pay in time, because every pipeline downloads and installs the whole set again. And you pay in drift, because unless every repo pins every tool identically, the repos slowly diverge on which version of trivy or tflint they actually run, and a check that passes in one repo fails in another for no reason anyone can see.

One image, one source of truth

infra-tools is the answer: a single Debian-based container image with the whole toolchain baked in. Every CI job in every repo uses it with one image: line.

The real value isn’t the convenience. It’s that the image is the one place tool versions are pinned. The Go-based tools are pinned in a mise.toml. checkov, which has no mise plugin, is pinned in a requirements file installed with pipx. The AWS CLI is pinned by a build argument. Three mechanisms, because the tools come from three kinds of source, but one image, and every pin wired to Renovate so a version bump arrives as a reviewable pull request. There’s exactly one answer to “what version of trivy does the toolchain use”, and it lives here.

Publishing with crane, not a second build

A build-pipeline detail that took a real bug to discover.

The pipeline builds the image with kaniko, which builds images without a privileged Docker daemon, something that matters a great deal on shared CI runners. Then it scans the image, then it publishes it.

The obvious way to write the publish stage is “build the image and push it”. But kaniko has no mode for “just push this tarball I already built”. A second kaniko invocation re-executes the entire Dockerfile from the top, including a second mise install, which makes a fresh round of calls to GitHub’s API to fetch tools. GitHub’s anonymous API limit is low and shared by IP, so on a CI runner that second install reliably trips a 403 rate-limit. (Yes, another 403. They do get everywhere.)

So the publish stage doesn’t rebuild. It uses crane to push the exact image tarball the build stage already produced. The image is built once. And because the published bytes are the same bytes the scan stage scanned, there’s no gap between “the image we checked” and “the image we shipped”.

Soft-failing the scanner on purpose

The decision that looks wrong until you see the reasoning: the pipeline scans the image with trivy, and trivy is allowed to fail without failing the pipeline.

A vulnerability scanner that doesn’t gate the build sounds like a scanner switched off. It isn’t. It’s a scanner pointed at something it can’t helpfully gate.

The tools in the image are prebuilt Go binaries. trivy inspects them, reads the version of the Go runtime each was compiled with, and reports every known CVE in that Go runtime. Those findings are real, but they aren’t mine to fix. The only fix is the upstream tool rebuilding itself against a patched Go. With seven such tools in the image, at any given moment one of them is usually a little behind on its Go version.

A hard gate would mean the image becomes unpublishable whenever any single upstream lags, over a CVE in code I don’t own and can’t patch. That’s not a security control; it’s a way to be unable to ship. So the scan is allow_failure. The findings stay fully visible, and the residual count is genuinely useful as a metric for how far behind upstream the toolchain has drifted. It just doesn’t block shipping an image whose only “vulnerabilities” are other people’s build timelines.

What it comes down to

The infrastructure repos all run the same CI gate jobs, needing the same tools, so infra-tools bakes the whole toolchain into one image and pins every version in one place, wired to Renovate.

Two build choices are worth copying. The publish stage uses crane to push the already-built, already-scanned tarball, because a second kaniko build would re-run mise install and hit GitHub’s anonymous rate limit, and because pushing the scanned bytes means shipping exactly what was checked. And the trivy scan is deliberately allow_failure, because it reports Go-runtime CVEs in prebuilt upstream binaries that no change to this repo can fix, so a hard gate would only make the image unshippable over someone else’s lag.

A 403 you can't fix in IAM

Thu, 14 May 2026 00:00:00 +0000

The OIDC post explained the handshake that lets a GitLab pipeline deploy to AWS with no stored key. This is the story of the first time I got it wrong, and spent an afternoon fixing the wrong thing. The error was a flat 403 from AWS, and the maddening part is that no amount of editing the IAM policy was ever going to fix it.

A 403 on the first real run

The OIDC post covered the handshake: GitLab CI mints a signed token, AWS exchanges it for short-lived credentials against a role whose trust policy names the pipeline. During the GitLab migration I wired exactly that up for the infra repo, including a trust policy condition meant to let merge-request pipelines run a plan.

The first merge request that should have triggered tofu-plan didn’t run it. The job failed, and the error from AWS was a flat AccessDenied. A 403.

The instinct, and why it wastes an afternoon

The instinct on an IAM 403 is immediate and almost always right: the policy’s wrong, so go and edit the policy. Tighten the condition. Loosen the condition. Check the wildcard. Re-read the sub pattern character by character.

All of that was wasted, and it was wasted for a reason that took me far too long to see. The trust policy wasn’t matching the wrong value. It was matching a value that does not exist. No amount of editing a condition makes it match a thing that’s never present.

What is actually in the token

GitLab’s OIDC token has a sub claim that encodes the pipeline’s context, and part of that encoding is a ref_type. I’d assumed ref_type could be branch, tag, or mr, because a pipeline can certainly be a branch pipeline, a tag pipeline, or a merge-request pipeline. So the trust policy, for the plan job, matched a sub containing ref_type:mr.

That assumption was wrong. GitLab’s ref_type is branch or tag. That’s the entire set. There is no mr.

A merge-request pipeline doesn’t run against a merge-request ref. It runs against the source branch. So its token’s sub carries ref_type:branch, like any other branch pipeline. The trust policy condition asked for ref_type:mr, GitLab never puts mr in a token, the condition was therefore never true, and every merge-request pipeline got a 403. Forever, until the policy stopped asking for a claim that isn’t real.

The fix, and the lesson worth more than the fix

The fix is small once it’s visible: match ref_type:branch and narrow it down by branch name or project path instead. An afternoon of policy edits, and the actual change is one word.

The lesson is the part worth keeping. When an OIDC trust fails, the useful question is never “is my policy clever enough”. It’s “what’s actually in the token”. An OIDC trust policy can only ever match the claims the identity provider genuinely asserts, and the gap between what a provider asserts and what you assumed it asserts is precisely where this class of bug lives.

So the move, when an OIDC handshake 403s, is to get hold of a real token and decode it. Look at the actual sub, the actual claims, the actual values. Match what’s there. A 403 that survives every sensible edit to the policy is usually not a policy that’s too loose or too strict. It’s a policy matching a claim that was never going to be in the token.

The habit it left behind

I wired an OIDC trust policy to let merge-request pipelines plan, by matching a sub claim with ref_type:mr. The first real merge request got a 403, and no edit to the policy fixed it, because GitLab’s ref_type is only ever branch or tag. A merge-request pipeline runs on a branch ref, so the mr value the policy demanded was never in any token.

The fix was one word. The habit it left behind is the valuable bit: when an OIDC trust fails, stop editing the policy and go and read a real token. A trust policy can only match what the provider actually asserts, and “what I assumed it asserts” is where the 403 was hiding the whole time. (If this shape of bug feels familiar by the end of the series, that’s not an accident: I come back to it with two more from exactly the same family.)

Pure-Rust Git, no git binary

Wed, 13 May 2026 00:00:00 +0000

go-tool-base’s VCS support has two halves that get confused for one. One half talks to forge APIs (GitHub, GitLab) for releases and pull requests. The other talks to the .git directory on disk: clone, history, diff, status. This post is mostly about the second half, and specifically about a question that turns out to have three answers in Rust, only one of which I’d recommend: how do you actually do Git from inside a program?

A VCS subsystem with two halves

go-tool-base has a VCS subsystem, and it does two distinct jobs.

The first is forge APIs. GitHub and GitLab, Enterprise and nested group paths included. It authenticates, lists releases, fetches release assets, manages pull requests. The self-update machinery sits on this half, and it’s what a tool uses to ask “what’s the latest release?”

The second is local Git. go-tool-base also carries a RepoLike object, an abstraction over an actual Git repository on disk: clone it, read its commit history, diff two trees, check its status. This half doesn’t talk to a hosting service at all. It talks to the .git directory.

It would be easy to assume the second half grew out of the first. It didn’t, and where it actually came from is the part worth telling.

A capability ahead of its consumer

The RepoLike object wasn’t built for go-tool-base. It came from another project, where it had already proved itself, and it was pulled into go-tool-base on purpose, with a specific future consumer in mind: the code generator.

The plan is for the generator to use Git directly. When it scaffolds a new tool, that tool should start life as a Git repository, with a git init and an initial commit. When you later regenerate, the generator should diff the regenerated template output against your working tree to detect drift, the same idea as respecting your edits. Both of those are local Git operations, not API calls, so the generator needs a repository abstraction to call into.

That wiring isn’t finished yet. The generator doesn’t drive RepoLike today. But the capability is in place, deliberately, ahead of the consumer that will use it, because the alternative is bolting Git support on later under deadline pressure, and that’s how you end up with the wrong abstraction.

So when rust-tool-base was built, a repository abstraction was never in question. The Rust port carries the same capability for the same reason: a Repo type with init, open, clone, walk, diff, blame, status, commit, fetch and checkout, present and ready for the generator to wire into. The open question was never whether to have it. It was how to do Git from inside a Rust program, and there are three answers, only one of which is any good.

Three ways to do Git, and the one worth picking

Shell out to git. Run the git binary as a subprocess and parse its output. It works until it doesn’t. The binary might not be installed. It might be a different version with different output. Its output is formatted for humans and changes between releases, so parsing it is a standing liability. You’ve made an undeclared dependency on a program you don’t ship.

Link libgit2. libgit2 is the C library that reimplements Git as something you can call from code, and git2 is the Rust binding to it. It’s solid and widely used. But it’s a C dependency, which means a C toolchain in the build, and it’s consistently the single biggest source of cross-compilation pain in the Rust Git ecosystem. The musl builds, the Windows builds, the static linking: libgit2 is where they tend to break.

Use gix. gix is a reimplementation of Git in pure Rust. No C library, no subprocess. It’s just Rust code, and it compiles and cross-compiles like any other crate, because that’s all it is. It’s also generally faster, and being pure Rust it fits the no-unsafe-in-first-party-code story far more comfortably than dragging a C library along.

rtb-vcs is gix-first. The Repo type is built on it. There’s no git binary dependency, and there’s no libgit2 in a default build.

gix is still maturing, and a few write paths, push in particular, aren’t ready in it yet. For those, git2 stays available as an opt-in fallback behind a Cargo feature. Off by default, so the libgit2 C dependency and its cross-compile cost only land in builds that explicitly ask for push support. The common case, a tool that clones, reads history, diffs and commits, pays none of it. (Which is exactly the feature-flag story from a couple of weeks back, doing real work.)

`Repo` is a foundation, not a façade

One design decision is worth calling out, because it came straight from a go-tool-base lesson.

It would have been easy to build Repo as a narrow façade exposing exactly what the scaffolder and the release-notes feature need today, and nothing else. That was rejected on purpose. go-tool-base’s RepoLike is itself the cautionary tale: it arrived from another project, settled into a sensible abstraction, and is already lined up to carry a consumer, the generator, that wasn’t driving its design when it was first written. A repository abstraction gets used by code that doesn’t exist yet. Build one as a narrow façade around today’s needs and you’ve guaranteed a rewrite the first time a downstream tool wants something slightly different.

So rtb-vcs’s Repo is built as a foundation: a sensible, reasonably complete vocabulary of Git operations that a tool author can compose richer behaviour on, without re-importing gix directly and re-deriving the framework’s auth and concurrency conventions. The errors back this up. gix’s error types aren’t leaked through the public API; they’re wrapped in semantic RepoError variants, so the backend could be swapped, gix to git2, or to something else entirely, without breaking a single downstream caller.

Stepping back

go-tool-base’s VCS support has two halves: forge-API calls for releases and pull requests, and a RepoLike object for local Git operations. The repo half arrived from another project and is wired in ahead of its intended consumer, the code generator, which will use it to initialise repositories for scaffolded tools and to diff regenerated output for drift.

rust-tool-base carries the same capability on purpose. Its Repo type is built on gix, a pure-Rust Git implementation, so there’s no dependency on an installed git binary and no libgit2 C library in a default build, which keeps cross-compilation clean. git2 stays an opt-in fallback for the few write paths gix can’t do yet. And Repo is built as a foundation for downstream tools, with the backend wrapped behind its own error type so it can be replaced without breaking callers.

Routing security findings without the noise

Tue, 12 May 2026 00:00:00 +0000

Turning on GuardDuty and Security Hub gives you threat detection. It also gives you a firehose. And an alert system that dutifully forwards everything in that firehose isn’t monitoring, it’s a very efficient way of training your team to ignore alerts. So the alerts module’s real job isn’t detection at all. It’s deciding what’s actually worth interrupting a human for, and the interesting part is everything it deliberately throws away.

Detection is the easy half

Switching on threat detection in an AWS account is a few resources. GuardDuty, Security Hub with its standards, IAM Access Analyzer: the security baseline does exactly that. From then on, the account is generating findings.

And it generates a lot of them. Plenty are low-severity, informational, or simply the normal texture of a cloud account. If you wire every finding to an email or a pager, you haven’t built monitoring. You’ve built noise. And noise has a specific failure mode: people stop reading it, and the one finding that genuinely mattered scrolls past unread alongside two hundred that didn’t.

So the valuable work isn’t detection. It’s routing: deciding what’s worth interrupting a human for, and letting the rest sit quietly in a console for whenever someone reviews it.

Forward the severe, leave the rest

The alerts module routes findings with EventBridge rules into an SNS topic that emails out. The rules are deliberately picky. GuardDuty findings are forwarded only at severity 7 and above. Security Hub findings are forwarded only at HIGH and CRITICAL.

Everything below those thresholds isn’t discarded. It’s still in GuardDuty and Security Hub, where someone doing a review will see it. It just doesn’t get to interrupt anyone’s day. The threshold is the line between “look at this now” and “look at this sometime”.

The duplicate you would otherwise send twice

Here’s the subtle one, and it’s the kind of thing you only find by looking closely at where findings come from.

Security Hub is an aggregator. It pulls findings in from other services, GuardDuty among them. So a single GuardDuty finding can show up in two places: in GuardDuty itself, and again in Security Hub as an aggregated copy.

A rule on GuardDuty findings and a rule on Security Hub HIGH/CRITICAL findings would therefore both fire for the same underlying GuardDuty finding. One event, two emails. Do that across an account and a meaningful fraction of your alert volume is just the same findings counted twice, which is its own kind of noise.

So the Security Hub rule explicitly excludes findings whose ProductName is GuardDuty, with an anything-but match. GuardDuty findings come through the GuardDuty rule. The Security Hub rule handles everything Security Hub adds that GuardDuty didn’t already report. One finding, one alert, regardless of how many services it passed through.

Two tripwires on the root account

Findings are about threats the detectors recognise. The module adds two alarms about something simpler: the root account doing anything at all.

One CloudWatch alarm fires on a root console sign-in. The other fires on any root API call that isn’t a console login. In a well-run AWS account, the root user does almost nothing after initial setup: day-to-day work happens through roles. So root activity isn’t a “finding” to be assessed for severity. It’s a tripwire. Any of it, in an account that should be silent, is worth an immediate look, and the two alarms say so directly.

Why a quiet alert stream matters here

This is monitoring for the account that’s going to hold the release-signing key, and that raises the stakes on getting the routing right.

If a key-bearing account ever does come under attack, the alert that says so has to be seen. An alert stream that’s mostly noise and duplicates is, functionally, no alerting at all, because the people who’d act on it have long since tuned it out. Routing the stream down to “severe, deduplicated, plus root tripwires” is what keeps it something a human will still read on the day it finally matters.

The short version

GuardDuty and Security Hub make detection easy. The hard, valuable part is routing: forwarding what deserves to interrupt someone and leaving the rest in a console.

The alerts module forwards GuardDuty at severity 7-plus and Security Hub at HIGH/CRITICAL, and it drops the duplicate that aggregation creates by excluding GuardDuty-sourced findings from the Security Hub rule, so one finding is one alert. Two CloudWatch alarms act as tripwires on root-account activity, which should be near-zero. For the account that will hold the signing key, a quiet, trustworthy alert stream isn’t a nicety. It’s the difference between monitoring and theatre.

Why go-tool-base left GitHub for GitLab

Mon, 11 May 2026 00:00:00 +0000

A botched version bump made me stop and actually look at where go-tool-base lived, and I didn’t much like what I saw. GitHub had spent months quietly falling over, and when Mitchell Hashimoto (GitHub user #1299, no less) publicly walked Ghostty off the platform, it stopped feeling like just my problem. I’ve been a GitLab fan for years, so the move was less a leap and more an overdue nudge. This is the why, not the how.

It started with a wrong number

Every migration has a trigger, and mine was embarrassingly small. A commit landed on main carrying a BREAKING CHANGE: footer it didn’t really deserve. Semantic-release did exactly what it’s told to do with that footer: it cut a major version. go-tool-base lurched from the v1 line straight to v2.0.0, and a chain of things that keyed off the version went sideways with it.

It was fixable. It wasn’t a disaster. But it was the kind of small, stupid breakage that makes you stop and actually look at your setup instead of just patching it and moving on. And when I looked, the version bump wasn’t the thing that bothered me. It was everything around it.

The platform had been quietly failing

I’d been losing time to GitHub for months. Not dramatically. No single outage you’d write home about, just a steady drip of Actions queues that wouldn’t drain, pull requests that wouldn’t merge, the occasional morning where the thing simply wasn’t there. You absorb it. You re-run the job. You make a coffee and try again. You tell yourself it’s a blip.

The trouble with a steady drip is that you stop counting it. It becomes weather.

The canary left the mine

Then, in late April, Mitchell Hashimoto (co-founder of HashiCorp, creator of Vagrant, Terraform and the Ghostty terminal) published Ghostty Is Leaving GitHub, and The Register picked it up a day later under the headline “GitHub ’no longer a place for serious work’”.

This is not a man with a casual relationship to GitHub. He’s, by his own account, user #1299, joined February 2008. He called it “the place that has made me the most happy”. And he still wrote this:

This is no longer a place for serious work if it just blocks you out for hours per day, every day.

The detail that landed hardest for me wasn’t a quote, it was a habit. He’d kept a journal for a month, marking an “X” on every day a GitHub outage had cost him working time. Almost every day had an X. Reading that, I realised I’d been having the same month. I’d just never been disciplined enough to write it down. He’d turned my vague “it’s been flaky lately” into a row of crosses on a calendar.

I want to ship software and it doesn’t want me to ship software.

When the person who’s been on the platform for eighteen years and loves it says that out loud, it stops being your private grumble. It’s the canary, and the canary has stopped singing.

Why GitLab, and not just “somewhere else”

Being annoyed at GitHub is a reason to leave. It is not, on its own, a reason to pick a destination. The destination has to be a positive choice.

For me GitLab was an easy one, because I’ve been a fan for years. Long enough, in fact, to have also been a reliable grumbler about their pricing tiers, which is how you know it’s a real relationship and not a honeymoon. What I’ve always rated is the model: GitLab treats source hosting, CI/CD, the package registry, releases and Pages as one integrated product, not a marketplace of bolted-on parts you assemble yourself.

That integration is the actual prize. On the old setup, “CI” meant a folder of separate GitHub Actions workflow files, each pinned, each its own little world. On GitLab it’s a single .gitlab-ci.yml pipeline with proper stages (lint, test, security, docs, release) and the release stage talks to the built-in package registry and Pages without me wiring up a single external credential. The CI job that builds the project can authenticate to the things the project needs because they’re the same platform.

There’s a second-order benefit too. A migration is a rare licence to fix things you’d never otherwise touch. Moving gave me the cover to reset go-tool-base’s versioning cleanly (back to a sensible v0.x line, the accidental v2.0.0 left behind as a cautionary tale) and to move the module path to its new home in one deliberate change rather than a thousand apologetic ones.

What I’m not going to claim

I’m not going to tell you GitHub is finished, or that GitLab never has a bad day, because it does, everyone does. This isn’t a teardown. GitHub gave go-tool-base a perfectly good home for its first year, and the archived mirror is still sitting there, read-only, pointing anyone who finds it at the new place.

What changed is simpler than a grand verdict. The friction crossed a line, someone I respect said the quiet part loudly enough that I couldn’t keep filing it under “weather”, and the place I’d have moved to anyway was sitting right there with a better model. Sometimes the prudent move and the move you secretly wanted turn out to be the same move, and you just need a wrong version number to give you permission.

Boiling it down

go-tool-base moved from GitHub to GitLab in May 2026. The proximate cause was a self-inflicted version-bump mess; the real cause was months of GitHub unreliability that I’d stopped consciously noticing until Mitchell Hashimoto’s very public departure named it for me. GitLab was a positive pick, not just an escape hatch: its integrated CI/CD, registry, releases and Pages are one product rather than a kit, and that integration is genuinely worth having. The migration also bought a clean versioning restart as a bonus.

If you’ve been absorbing a steady drip of friction and telling yourself it’s normal: try the calendar trick. Mark the X’s for a month. The page will tell you something you already half-know.

Why I hand-rolled every module

Sun, 10 May 2026 00:00:00 +0000

There are well-known community module libraries for AWS: Cloud Posse, the terraform-aws-modules collection, plenty more. Both terraform-aws-bootstrap and terraform-aws-security-baseline use almost none of them. Every sub-module is hand-rolled from raw AWS resources, and before you accuse me of not-invented-here syndrome (a perfectly fair first guess), hear me out, because the same evaluation kept landing the same way for a real reason.

The promise of a wrapper module

The community module ecosystem makes an appealing offer. Don’t write raw aws_s3_bucket and aws_s3_bucket_policy and aws_s3_bucket_public_access_block and the rest. Call a tested, popular module, pass it a handful of inputs, and get a correct, well-configured bucket. Less code in your repo, and the code you don’t write has been exercised by thousands of other users.

For a lot of infrastructure that’s a genuinely good deal, and I take it often. For the two infrastructure modules in this series, I took it almost never. Every sub-module is built from raw AWS resources. That wasn’t a reflex. It was the same evaluation, made over and over, landing the same way.

What kept going wrong

For each place a wrapper module could have fitted, I looked at the wrapper. And the recurring finding was one of two things. Either using the wrapper correctly, with all the overrides my posture needed, came to more configuration than the raw resources would have. Or the wrapper’s abstraction leaked the instant I needed something it hadn’t anticipated, and I was now writing code to fight it.

The CloudTrail bucket, concretely

The clearest example is the bucket that holds CloudTrail logs.

There are popular modules that set up CloudTrail and bundle an S3 bucket for the logs. Convenient. But that bundled bucket isn’t the bucket I want. It doesn’t carry lifecycle { prevent_destroy = true }, and its bucket policy is weaker than the one the state bucket taught me to want: TLS-only, SSE-KMS-only, wrong-key-denied.

So to use the wrapper I had two options. Accept a weaker audit-log bucket than the rest of the account, which rather defeats the point of an audit log. Or fight the wrapper: disable its bucket, create my own, wire it back in. Fighting the wrapper is more work than simply writing the fifty-odd lines of raw aws_s3_bucket plus policy that give me exactly the posture I’d already designed once. The wrapper didn’t save code. It added a negotiation.

A wrapper is a deal, and deals have terms

This isn’t an argument that community modules are bad. It’s an argument about when the deal is good.

A wrapper module is a good deal while its abstraction holds: while what it assumes you want matches what you want. The moment you need something it didn’t anticipate, the deal inverts. Now you’re working against the abstraction, and an abstraction you’re fighting costs more than no abstraction at all. (Regular readers will recognise that line from the LangChain argument; it’s the same principle in a very different language.)

Infrastructure that holds signing keys is precisely the case where you need to control the specifics: every encryption setting, every lifecycle rule, every line of every bucket policy. That’s a domain where wrapper abstractions leak fast, because the whole job is the details the wrapper smoothed over.

The cost, paid on purpose

Hand-rolling isn’t free. It’s more lines of HCL in the repo, up front, than a one-line module call.

What those lines buy is worth the price for this kind of infrastructure. There’s no transitive module-version churn to track. There’s no abstraction between me and the resource when something behaves oddly. And every line is one I can read, and defend, in a security review, because I wrote it and it says exactly what it does. For a foundation that will hold the most sensitive key in the system, “readable and mine” beats “short and someone else’s”.

That’s a deliberate trade, not a universal rule. For an internal tool on a deadline, reach for the wrapper. For the security-critical base of everything else, the raw resources won every time I checked.

To sum up

The community module ecosystem offers less code that more people have tested, and for plenty of infrastructure that’s the right call. For terraform-aws-bootstrap and terraform-aws-security-baseline it almost never was, because each wrapper turned out to be more configuration than the raw resources once my posture was accounted for, or it leaked the moment I needed a specific.

The CloudTrail log bucket is the pattern in miniature: the bundled bucket lacked prevent_destroy and a strong policy, so using the wrapper meant either a weaker bucket or fighting the module. A wrapper is a good deal while its abstraction holds and a bad one the moment you fight it, and security-critical foundation infrastructure is all specifics. Hand-rolling cost more lines and bought code I can read and defend. For this, that was the trade worth making.

Hardening the account that will hold the keys

Sat, 09 May 2026 00:00:00 +0000

Bootstrapping the account got it ready: somewhere to store state, an identity to deploy as, enough for the next tofu apply to run. Ready is not the same as safe. An account with no audit trail, nothing watching it, and no considered way for a human to get in is fine for experimenting and absolutely not where you put the most sensitive key in the system. So before the signing key goes anywhere near it, the account gets a security baseline.

Ready is not the same as safe

The bootstrap post ended with an account that was ready: it had somewhere to store state and a CI identity to deploy as. The next tofu apply could run.

Ready is not safe. That account still has no audit trail, so nobody could tell you afterwards what happened in it. It has no threat detection, so nothing is watching. Its defaults are AWS’s defaults, which are not a security posture. There’s no considered way for a human to get in. An account in that condition is fine for experimenting. It’s not somewhere you put the most sensitive key in the whole system.

So before the signing key is anywhere near it, the account gets a security baseline.

The baseline, in one downstream stack

terraform-aws-security-baseline is that baseline, and it’s exactly the downstream stack the bootstrap post promised: applied through the automation role bootstrap created, not bootstrapped specially.

It’s six sub-modules, each behind an enable_* toggle: account-hardening (IAM password policy, account-wide S3 public-access blocking, default EBS encryption), audit-logging (a multi-region CloudTrail with log-file validation), aws-config, threat-detection (GuardDuty, Security Hub, IAM Access Analyzer), alerts, and operator-role. Together they turn a bare account into one that records what happens, watches for trouble, and controls who gets in.

Most of those are the expected baseline. The operator role is the one worth slowing down on, because it’s built backwards from how people usually think about an admin role.

The operator role, and the inversion

InfraAdmin is the human way into the account: the role a person assumes to do operator work. Two things define it.

The trust policy decides who may assume it. It trusts only the account root principal, and it requires multi-factor authentication: the assume call must carry aws:MultiFactorAuthPresent, and aws:MultiFactorAuthAge bounds how recently that MFA was performed. No MFA, no role. So far this is a careful but ordinary admin role.

The inversion is a second, separate inline policy, and it’s almost entirely Deny. It denies, using NotAction, anything where aws:RequestedRegion falls outside an allowed set of regions. The role’s power comes from an admin grant. This inline policy fences that power.

That’s the part worth holding onto. People picture an admin role as a list of what it can do. This one is better understood by what it cannot: it cannot act outside its permitted regions, full stop. A fat-fingered command, or a compromised session, cannot quietly spin resources up in some region nobody’s watching. The fence is as much the point of the role as the grant is.

The carve-out, because honesty

There’s a fiddly detail, and it’s the kind of thing that makes the region fence real rather than theoretical.

Some AWS services are global. IAM, CloudFront, Route 53 and friends have no region, and they don’t honour aws:RequestedRegion. A naive region-deny would therefore deny calls to IAM, and you’d lock yourself out of the very service you manage access with. (A close cousin of the kind of self-inflicted lockout I’ll come back to in a later post.)

So the Deny carries explicit carve-outs for the global services. It isn’t elegant, and it can’t be: the global-versus-regional split is just a fact of AWS, and a correct region fence has to account for it. The carve-out list is the honest cost of the control working.

Harden the room, then move the keys in

There’s an order to all of this, and the order is the argument.

The account that will hold the signing key has to be audited before the key arrives, so that from day one every call against it is in CloudTrail. It has to be watched before the key arrives, so GuardDuty is already looking. It has to be access-controlled before the key arrives, so the only human path in is MFA-gated and region-fenced.

You don’t move something valuable into a room and then think about locks. You build the room, fit the locks, check they work, and then move the valuable thing in. The security baseline is fitting the locks. The signing key comes later, into a room already built for it.

Worth remembering

Bootstrapping an account makes it ready for the next deploy. It does not make it safe to hold anything that matters. terraform-aws-security-baseline is the downstream stack that closes that gap: audit logging, AWS Config, threat detection, account hardening, and an operator role, applied through the CI role bootstrap created.

The operator role is the piece to study. It’s MFA-gated on the way in, and then fenced by a separate, almost-all-Deny inline policy that confines it to permitted regions, with carve-outs for the global services that have no region. An admin role defined as much by its fence as its grant. Harden the room first; the keys move in afterwards.

No access keys in CI

Fri, 08 May 2026 00:00:00 +0000

A long-lived AWS access key, sitting in a CI system, is just about the single credential I’d most like to be rid of. It’s powerful, it never expires unless someone remembers to rotate it (nobody remembers to rotate it), and it lives in one of the most attractive targets in the whole supply chain. For infrastructure that’s eventually going to hold a release-signing key, it’s exactly the wrong place to start. So the phpboyscout infrastructure has no AWS access key in CI at all. None.

The access key you don’t want

A CI pipeline that runs tofu apply against AWS needs AWS credentials. The traditional way to give it some is an IAM user with an access key pair, pasted into the CI system as a masked variable.

Look at what that key is. It’s long-lived: it works until someone remembers to rotate it, and rotating it is a chore, so mostly nobody does. It’s powerful: it can apply infrastructure, so it can do nearly anything. And it’s sitting in a CI system, which is one of the most attractive targets in your whole supply chain. You’ve taken your highest-value credential and stored a permanent copy of it in a place built for running automated jobs.

For infrastructure that’s going to hold a release-signing key, that’s precisely the wrong starting point. So the phpboyscout infrastructure has no AWS access key in CI at all. Not a well-guarded one. None.

Federation instead of a stored secret

The replacement is OIDC federation, and the shape of it is worth walking through, because it’s genuinely different from “a secret, but better”.

A modern CI platform can mint an OIDC token. GitLab does this with an id_tokens: block: at job time, GitLab issues a short-lived JSON Web Token, signed by GitLab, that asserts a set of facts. This is project X. This is pipeline Y. This is running on ref Z, of this type.

AWS can consume that. The sts:AssumeRoleWithWebIdentity call takes such a token and, if it satisfies an IAM role’s trust policy, returns short-lived AWS credentials for that role. The trust policy is where the control lives: it names GitLab as a trusted token issuer, and it constrains the token’s sub claim so that only the specific project, and the specific refs, you intend can assume the role.

Put it together: the pipeline asks GitLab for a token, hands it to AWS, and gets back credentials that last about an hour and are scoped to one role. Nothing long-lived is stored anywhere. The credential exists only for the job that needs it, and it can’t be stolen from a CI variable store, because it was never in one.

Two halves of one handshake

That handshake is built by two of the repos in this series, each owning one side.

terraform-aws-bootstrap builds the AWS half, in its automation-iam module: it registers GitLab as an OIDC identity provider in the account, and it creates the automation role with the trust policy that decides which pipelines may assume it.

The CI components build the consuming half: the id_tokens: block that asks GitLab for the JWT, and then simply letting the AWS provider’s own credential chain perform the exchange. The pipeline doesn’t call sts by hand. It presents the token; the SDK does the rest.

The gotcha: don’t set a profile

There’s one quiet way to break this, and a stack can look completely correct while doing it.

The AWS SDK finds credentials by walking a chain of sources in order. The web-identity path, the one that uses the OIDC token, is one link in that chain. It triggers off environment variables the CI sets up automatically.

But if the aws provider block has a hardcoded profile = "...", the SDK takes the profile link of the chain instead, and never reaches the web-identity link. A profile line is the sort of thing that ends up in a provider block from someone’s local development setup, where it’s exactly right. Committed and run in CI, it silently short-circuits the federation. The pipeline either fails to find credentials, or finds the wrong ones.

The rule is simple once you know it: the provider block that runs in CI must not name a profile. Leave the chain free to find the web identity. It’s the kind of bug that teaches you to be precise about which link of the credential chain you’re actually relying on.

The bottom line

Giving CI an AWS access key means storing your most powerful, longest-lived credential in one of your most exposed systems. OIDC federation removes it entirely. The CI platform mints a short-lived signed token, AWS exchanges it via AssumeRoleWithWebIdentity for hour-long credentials against a role whose trust policy names the exact pipeline, and nothing permanent is stored.

terraform-aws-bootstrap builds the AWS side, the identity provider and the trust policy; the CI components build the consuming side, the token request. The one trap is a hardcoded profile in the provider block, which short-circuits the SDK’s credential chain before it reaches the web-identity path. Get that right, and a pipeline deploys to AWS as a verifiable, short-lived identity, with no key to steal.

Secrets that scrub themselves from RAM

Wed, 06 May 2026 00:00:00 +0000

A while ago I worked out where a CLI should keep your API key: env var, OS keychain, or, grudgingly, a literal in the config file. That answers where the secret lives. It says nothing about what happens to it once it’s loaded and sitting in your process memory, which is the half where secrets actually tend to leak. Rust, it turns out, can do something about that half that Go simply can’t.

What go-tool-base already settled

A while back I wrote about where a CLI should keep your API keys. The answer go-tool-base settled on was three storage modes, in a fixed precedence: an environment variable reference (the recommended default), the OS keychain (opt-in), or a literal value in the config file (legacy, and refused outright when CI=true).

rust-tool-base keeps that design unchanged. Same three modes, same precedence, same refusal of literal secrets in CI. A tool embeds a CredentialRef in its typed config, and a Resolver walks env, then keychain, then literal, then a well-known fallback variable, first hit wins. That part is a straight carry-over, because where to keep the secret was design, and design survives the port.

But storage is only half the life of a secret. The other half is what happens to it once it’s resolved and sitting in your process memory. That’s where Rust can do something Go can’t, and rust-tool-base takes the opening.

The two ways a secret leaks after you’ve loaded it

You’ve resolved the API key. It’s a value in memory now. Two very ordinary things can leak it from there, and neither involves your storage being wrong.

The first is the log line. Somewhere a developer writes a debug print of a config struct, or an error includes the struct that holds the key, or a panic dumps it. The secret is a string like any other string, so it renders like any other string, straight into a log aggregator that a lot of people can read.

The second is the leftover bytes. The key sat in a heap allocation. The variable goes out of scope, the allocation is freed, and on most runtimes “freed” just means “returned to the allocator”. The bytes are still there until something else writes over them. A core dump taken in that window contains your key. So does the next allocation that happens to land on that memory and gets logged before it’s overwritten.

A Go string can’t really defend against either. Go strings are immutable, so you can’t zero one in place; the runtime copies them freely, so you can’t even track every copy; and there’s no compile-time barrier stopping anyone printing one. You can be disciplined, but discipline is all you’ve got.

`SecretString` closes both

rust-tool-base routes every secret through secrecy::SecretString, and the crate is explicit that taking a plain &str or String for a secret is a type error, not a style preference.

For the log line, SecretString has its own Debug implementation, and it prints [REDACTED]. Always. A config struct holding a SecretString can be debug-printed, put in an error, caught in a panic, and the secret field shows up as [REDACTED] every single time. You don’t have to remember not to log it. The type already won’t.

For the leftover bytes, SecretString zeroes its memory when it’s dropped. When the value goes out of scope, before the allocation is handed back, the bytes are overwritten. The window where a freed allocation still holds your key is closed. A core dump taken afterwards finds zeroes.

There’s a third leak SecretString blocks that’s easy to miss. It deliberately doesn’t implement Serialize. You cannot serialise a SecretString. That sounds like an inconvenience until you see what it prevents: a tool that loads config, changes one setting, and writes the whole struct back would, with an ordinary string, faithfully write the resolved secret to disk in plain text. Because SecretString can’t be serialised, CredentialRef can’t be either, and that accident is structurally impossible. Writing a secret back is a deliberate, separate path, never a side effect of saving config.

When code genuinely needs the raw value, to drop it into an Authorization header, it calls expose_secret(). The name is the point. Getting at the plaintext is one explicit, greppable, reviewable call, and everywhere else the secret stays wrapped.

Discipline versus the type system

The honest framing is this. None of these leaks are exotic. Logging a struct, a core dump after a free, re-saving a config file: they’re all routine, and they’re all how real credentials end up somewhere they shouldn’t.

go-tool-base’s storage design is good, and rust-tool-base kept it. But in Go, not leaking the secret once it’s in memory comes down to every developer being careful every time. In Rust, SecretString makes the type system carry it. The redaction, the zeroing, the un-serialisability aren’t things you remember to do. They’re things the secret does to itself because of what it is. That’s the part Go structurally can’t match, and it’s why the port didn’t just copy the storage modes across, it tightened the handling underneath them.

The gist

go-tool-base settled where a CLI keeps a secret: env var, keychain, or literal, in a fixed precedence. rust-tool-base keeps that design and hardens what happens once the secret is loaded.

Every secret is a secrecy::SecretString. It debug-prints as [REDACTED], so it can’t fall into a log by accident. Its memory is zeroed on drop, so it doesn’t survive in freed heap. It isn’t serialisable, so it can’t be written back to config by a blanket save. Getting the plaintext is one explicit expose_secret() call. Go can only ask developers to be careful with a secret in memory; Rust lets the type be careful for them.

The chicken-and-egg of remote state

Wed, 06 May 2026 00:00:00 +0000

Here’s a puzzle that every infrastructure-as-code setup hits exactly once, right at the very beginning, and then never again. An OpenTofu stack stores its state in a backend. The bootstrap stack I wrote about last time has a particular job, and part of that job is to create the backend that remote state lives in. So where does the bootstrap stack store its own state, on the very first run, before it’s built the place state is supposed to go?

Where does the state of the thing that makes the state store live?

That’s the puzzle, and it’s a real ordering deadlock rather than a riddle.

An OpenTofu stack keeps a state file, and for anything shared that state file lives in a remote backend: on AWS, an S3 bucket. Fine. But the bootstrap stack has a particular job, and part of that job is to create the S3 bucket that remote state lives in.

So walk through the first run. Bootstrap has never been applied. The state bucket doesn’t exist, because creating it is what bootstrap is for. Bootstrap needs somewhere to store its own state. The only place that would make sense is the bucket it’s about to create, which isn’t there yet. The thing that builds the state store can’t store its state in the state store.

Run local, then migrate

The way out is a two-step that OpenTofu supports directly.

Bootstrap starts configured with a local backend: backend "local" {}. State is just a file on the operator’s machine. With that in place, the first tofu apply runs. It creates the S3 bucket and the KMS key, and records all of it in the local state file.

Now the bucket exists. So the backend configuration is rewritten to point at it: an s3 backend block naming the new bucket. Then tofu init -migrate-state. OpenTofu sees the backend has changed, picks up the local state file, and copies it into the S3 bucket. From that point on, bootstrap’s own state lives in the bucket that bootstrap created. The egg has laid the chicken.

The local backend was a scaffold. It existed for exactly one apply, to break the ordering deadlock, and then the state moved off it and it was never used again.

It happened twice

The infra repo actually did this migration twice, and the second time is the proof that the pattern is general rather than a one-off trick.

The first migration was the one above: local to S3, at the very start. The second came later, during the move from GitHub to GitLab. GitLab offers a managed HTTP state backend, and infra chose to use it. So the backend block was rewritten again, this time from s3 to http, and tofu init -migrate-state ran again, copying the state from the S3 bucket to GitLab’s backend.

The same move, twice, against three different backends. That’s the useful lesson hiding in the chicken-and-egg story. State is portable. The backend is just where you currently keep it, not a property of the stack itself, and moving it is a routine, supported operation rather than surgery.

Why this is the honest answer, not a hack

It’s easy to look at “apply once with a local backend, then migrate” and feel it’s a bit of a smell, a workaround for something that should have been cleaner.

It isn’t. It’s the honest answer to a real ordering problem, and the alternatives are worse.

The obvious alternative is to create the state bucket by hand, in the console, before running bootstrap at all. But then the most important bucket in the account is unmanaged. It exists outside every OpenTofu graph, nobody’s code describes it, its encryption and policy and prevent_destroy are whatever someone clicked that day, and it drifts. The local-then-migrate dance avoids exactly that. The bucket is created by bootstrap, described in code, and tracked in bootstrap’s own state from its very first apply. It’s managed from birth.

The chicken-and-egg isn’t a flaw to be embarrassed about. It’s just the shape of the problem when a stack has to build its own foundations, and OpenTofu’s -migrate-state is the supported tool for exactly that shape.

Pulling it together

Every OpenTofu stack needs a backend to store state, and the bootstrap stack’s job is to create the backend, so on its first run the bucket it needs doesn’t yet exist.

The resolution is to run bootstrap once with a local backend, let that apply create the bucket and key, then rewrite the backend configuration and tofu init -migrate-state the state into the bucket bootstrap just made. The infra repo did it twice, local to S3 and later S3 to GitLab, which shows the real point: state is portable, and the backend is just where you keep it. Doing it this way, rather than hand-creating the bucket, is what keeps that critical bucket managed in code from its very first day.

A state bucket that defends itself

Sat, 02 May 2026 00:00:00 +0000

OpenTofu’s remote state file is, quietly, the most sensitive thing in an infrastructure repo. It’s a plain JSON document listing every resource you manage, every ID, and, depending on your providers, the odd secret in clear text. So the S3 bucket that holds it can’t just be a bucket. It has to actively defend itself, on three separate fronts.

The most sensitive file in the repo

OpenTofu, like Terraform, keeps a state file: a JSON document recording every resource the stack manages, its real-world ID, and its attributes. It’s how the tool knows what already exists. It’s also, quietly, the most sensitive file in the whole repo. It can hold resource identifiers an attacker would value, and depending on the providers in play it can hold secret values in clear text.

Three bad things can happen to it. It can be deleted, and now the tool has forgotten everything it manages. It can be read by someone who shouldn’t. It can be corrupted by two runs writing at once. The bucket that holds remote state has to defend against all three, and terraform-aws-bootstrap’s state-backend module is built around doing exactly that.

The DynamoDB lock table is gone

Start with the corruption problem, because the answer changed recently.

The long-standing pattern for remote state on AWS was an S3 bucket plus a DynamoDB table. S3 held the state; the DynamoDB table held a lock, so two apply runs couldn’t write at once. Everyone who’s done Terraform on AWS has provisioned that table, probably more times than they’d care to count.

OpenTofu 1.10 made it unnecessary. The S3 backend gained use_lockfile, which does the locking with a small lock object in the same bucket, using S3’s conditional-write support. No separate table. The state backend is now genuinely one bucket and one key, with the lock living beside the state. It’s one fewer resource to create, one fewer thing to pay for, and one fewer moving part to reason about. The module takes the new path, and the DynamoDB table simply isn’t there.

A bucket you can’t delete by accident

Deletion is guarded with lifecycle { prevent_destroy = true } on the bucket. With that set, OpenTofu refuses to produce a plan that would destroy the bucket. A stray tofu destroy, a refactor that drops the resource, an accidental rename: all of them fail loudly instead of quietly taking the state bucket with them.

This is also why the state-backend module is hand-rolled from raw aws_s3_bucket resources rather than wrapping a community module like terraform-aws-modules/s3-bucket. prevent_destroy has to sit on the actual resource, and a lifecycle block isn’t something you can pass into a wrapper module as an input. Hand-rolling the bucket keeps prevent_destroy somewhere you can put it and, just as importantly, somewhere the next reader can see it. (There’s a whole post coming on why I hand-rolled every module; this is one of the reasons in miniature.)

Reject anything encrypted wrong

Confidentiality is the subtle one, because the obvious control isn’t enough.

The bucket has a default encryption configuration: server-side encryption with the customer-managed KMS key. But default encryption is a default. A client making a PutObject call can override it per request, asking for plain AES256 or a different KMS key, and S3 will honour the override.

So the module doesn’t rely on the default. The bucket policy explicitly denies the upload it doesn’t want. It denies any request not over TLS. It denies any PutObject that isn’t using SSE-KMS. And it denies any PutObject that names the wrong KMS key. The default encryption config says “this is what you get if you don’t ask”; the bucket policy says “and you’re not allowed to ask for anything else”. State can only ever land encrypted, in transit and at rest, under the one key the module controls.

One small companion setting: bucket_key_enabled. With per-object SSE-KMS, every object operation is also a KMS API call, which costs money and can throttle. An S3 Bucket Key collapses those into far fewer KMS calls, cutting per-object KMS traffic by well over ninety per cent. It’s a one-line setting the module turns on and most people forget exists.

In short

Remote state is the most sensitive file an infrastructure repo has, and the bucket that holds it has to defend against deletion, disclosure and corruption.

terraform-aws-bootstrap’s state backend handles corruption with OpenTofu 1.10’s use_lockfile, dropping the old DynamoDB lock table entirely. It guards deletion with prevent_destroy, which is also why the bucket is hand-rolled rather than wrapped. And it guards confidentiality with a bucket policy that denies non-TLS traffic and denies any upload not encrypted with the right KMS key, because default encryption is only a default and a client can override it. The state bucket isn’t just a place to put state. It’s built to refuse every wrong thing that could happen to it.

Supporting a provider, or actually using it

Sat, 02 May 2026 00:00:00 +0000

If your CLI tool talks to an AI model, you don’t want to hard-wire one vendor. So you reach for a single client interface over several providers, which is the right call. The trap is the next step: build that interface on only what every provider has in common, and you quietly throw away the very features that made you want a particular provider in the first place. rust-tool-base’s rtb-ai refuses to make that trade.

The pull toward one interface

If your CLI tool talks to an AI model, hard-wiring one vendor is a poor bet. One user has an Anthropic key, another an OpenAI key. Someone’s on Gemini. Someone runs Ollama locally because their data can’t leave the building. Someone points at an OpenAI-compatible endpoint from a provider you’ve never heard of. You don’t want a separate code path for each, so you want one AiClient that all of them slot behind.

rtb-ai gets that unification from the genai crate, which already speaks to Anthropic, OpenAI, Gemini, Ollama and OpenAI-compatible endpoints. One interface, five providers, the tool author picks one in config. The Go sibling makes the same bet: go-tool-base’s chat package also unifies several providers, behind an interface deliberately kept to four methods. So far this is the obvious design, and if it were the whole design there’d be nothing to write about.

What “unified” quietly costs you

Here’s the catch in any unified interface. It can only expose what every provider behind it has in common.

The common subset is plain chat. Messages go in, text comes out, optionally streamed token by token. That’s real and it’s useful and every provider does it. But the common subset is also the floor, and the features that make a particular provider worth choosing are almost never on the floor. They’re the things only that provider does.

Anthropic is the sharp example, because it has three features that matter and not one of them is common-subset.

Prompt caching. You can mark the stable parts of a request, the system prompt and the tool list, as cacheable. The provider keeps them warm, and on the next turn you aren’t billed to re-send and re-process text that didn’t change. On a long agent loop, where the same large system prompt rides along on every single turn, that’s a substantial saving in both cost and latency.

Extended thinking. The model works through a hard problem in a visible, budgeted reasoning pass before it commits to an answer, and you can see that reasoning.

Citations. Structured references back to source material in the response.

A client built strictly on the common subset can’t express any of those. It has no field for them, because four of the five providers wouldn’t know what to do with the field. So a purely lowest-common-denominator client would “support” Anthropic and then use it badly, leaving its best features unreachable. Support as a checkbox, not as the point.

The escape hatch

rtb-ai’s answer is to not choose. It runs two implementations under one interface.

For OpenAI, Gemini, Ollama and OpenAI-compatible endpoints, calls route through genai, the unified path. For Anthropic, every method drops to a direct reqwest implementation straight against the Messages API. Same AiClient on the surface, a different implementation underneath, selected by which provider the config names.

And the request type has deliberate room for the difference:

pub struct ChatRequest {
 pub system: Option<String>,
 pub messages: Vec<Message>,
 pub temperature: Option<f32>,
 pub max_tokens: Option<u32>,
 /// Anthropic-only: enables prompt caching at every stable point.
 /// Ignored on non-Anthropic providers.
 pub cache_control: bool,
 /// Anthropic-only: extended-thinking budget. `None` disables.
 /// Ignored on non-Anthropic providers.
 pub thinking: Option<ThinkingMode>,
}

Set cache_control and the Anthropic-direct path inserts cache breakpoints at the three stable points: the system prompt, the tool list, and the first message. Set thinking and it adds the thinking block, and streaming surfaces a separate ThinkingToken event so you can show the reasoning apart from the answer. On a non-Anthropic provider, both fields are simply ignored. The interface carries them; only the implementation that understands them acts on them.

A hatch, not a leak

It’s worth being precise about why this isn’t the thing it superficially resembles, which is a leaky abstraction.

A leaky abstraction is one where implementation details bleed through that you didn’t intend and can’t reason about. The abstraction quietly fails to abstract, and you’re left guessing which provider you’re really talking to.

This is the opposite of that. The two Anthropic-only fields aren’t a leak. They’re named, documented as Anthropic-only, inert everywhere else, and right there in the public type for anyone to see. The interface is uniform for the common case and deliberately, visibly non-uniform at exactly the points where uniformity would have cost you the good features. You opt into provider-specifics by setting a field. You stay fully portable by leaving it at its default. Nothing bleeds; you decide.

The same design line explains what does stay in the unified path. Structured output, chat_structured::<T>, sends a JSON Schema derived from your Rust type with the request and validates the reply against it before handing you a typed T. That’s a portability win that costs nothing across providers, so it belongs in the common interface. The split isn’t “Anthropic versus the rest”. It’s “features that are free to unify go in the unified path; features that aren’t get a designed door”. Prompt caching and extended thinking get the door, because flattening them away would be the expensive kind of convenient.

To sum up

A CLI tool that integrates AI wants one client over several providers, and a unified interface can only expose what those providers share. The shared floor is plain chat, and the features worth choosing a provider for, like Anthropic’s prompt caching, extended thinking and citations, are never on the floor.

rtb-ai keeps both. genai provides the unified path across five providers; an Anthropic-direct reqwest path drops below the abstraction for the features genai can’t reach, and ChatRequest carries the Anthropic-only fields openly, ignored elsewhere. Uniform where uniformity is free, with a designed escape hatch where it isn’t. That’s the difference between supporting a provider and actually using it.

Errors without an error handler

Fri, 01 May 2026 00:00:00 +0000

In the porting post I said go-tool-base’s error handler was one of the bits that didn’t survive the move to Rust, and promised to come back to it. Here’s the come-back. The short version is that Rust hands you, for free, the single consistent error exit that go-tool-base had to build a whole component to get.

What go-tool-base built

A while ago I wrote about error handling in go-tool-base. The core of it: an error should carry a hint, a separate field of human guidance telling the user what to do next, kept apart from the error’s identity so code can still match on it.

The other half of that post was about consistency. Every go-tool-base command returns its errors the idiomatic Cobra way, and they all funnel into one Execute() wrapper at the root, which routes every error through one ErrorHandler. One door out. Presentation decided in exactly one place, so no command can render a failure differently from its neighbour.

That handler is a real object. It exists, it’s wired in, it’s the thing every error passes through. Building it was a deliberate piece of work, and it was the right call for Go.

When I rebuilt this in Rust, the handler didn’t survive the move. Not because consistency stopped mattering. Because Rust gives you the single exit for free, and an object to enforce it would just be re-implementing something the language already does for you.

The shape of a Rust error

Start with the type. In rust-tool-base every crate defines its own error enum, and every one of them derives two traits:

#[derive(Debug, thiserror::Error, miette::Diagnostic)]
pub enum ConfigError {
 #[error("config file not found at {path}")]
 #[diagnostic(
 code(rtb::config::not_found),
 help("run `mytool init` to create one, or set MYTOOL_CONFIG"),
 )]
 NotFound { path: PathBuf },
 // ...
}

thiserror::Error makes it a proper error type. miette::Diagnostic is the interesting one. A Diagnostic is an error that also carries the things you’d want when presenting it: a stable code, a severity, a help string, and optionally source labels pointing at spans of input. The help line is the same idea as go-tool-base’s hint, the recovery step, except here it’s an attribute on the variant rather than a field threaded through a wrapper.

So the guidance lives on the error, structured, from the moment the error is created.

There is no handler, there’s a convention

Here’s where Rust does the work go-tool-base’s handler was built to do.

A rust-tool-base main looks like this:

#[tokio::main]
async fn main() -> miette::Result<()> {
 rtb::cli::Application::builder()
 .metadata(/* ... */)
 .version(VersionInfo::from_env())
 .build()?
 .run()
 .await
}

main returns miette::Result<()>. Every command’s run returns a Result too. In between, errors propagate with the ? operator: a function that hits an error returns it upward, immediately, and the caller does the same, all the way to main. Nobody writes a “check this error” call. ? is the propagation.

And when an error reaches main and main returns it, something has to render it for the user. That something is a report hook. rust-tool-base installs one at startup, and from then on any Diagnostic that exits main is rendered through it: the code, the severity, the help text, the source labels, with colour. One renderer, installed once.

Look at what that adds up to. Every error in the program flows to one place, main. It’s rendered by one thing, the hook. Presentation is decided in exactly one location and no command can deviate from it. That’s precisely the property go-tool-base’s ErrorHandler was built to guarantee. The difference is that nobody built it. The single exit is just where ? propagation ends, and the single renderer is one hook. The language’s own convention for returning errors from main is the funnel.

Errors are values, all the way

The thing that took me a moment to fully trust is that there’s no funnel to maintain, because there’s no funnel as an object. go-tool-base’s handler is a component: it can drift, it has to be kept in the path, a command could in principle be wired to bypass it. The Rust version cannot be bypassed, because bypassing it would mean a command not returning its error, and an error you don’t return is a compile-time warning at best and dead-obvious wrong code at worst.

So the model is just: errors are values, you return them, ? carries them up, main hands the last one to the hook. The consistency isn’t enforced by a guard. It’s the only thing the shape of the language really lets you do.

go-tool-base reaches a single, consistent error exit by building one and routing everything through it. rust-tool-base reaches the same exit by having errors be ordinary return values and letting them fall out of main. Same outcome. One of them is a component you own; the other is a convention you inherit.

Worth remembering

go-tool-base funnels every error through one ErrorHandler so presentation stays consistent. That handler is a deliberately built component, and it’s the right design in Go.

rust-tool-base has no handler. Every crate’s error type derives miette::Diagnostic, carrying its code, severity and help text. Errors propagate with ? to main, which returns miette::Result, and a framework-installed hook renders whatever comes out. The single consistent exit is the end of ? propagation, and the single renderer is one hook. The funnel go-tool-base built by hand is, in Rust, just the language’s return-from-main convention.

The bootstrap that does almost nothing

Fri, 01 May 2026 00:00:00 +0000

A brand-new AWS account is a slightly nerve-wracking thing. It can do almost anything, it’s hardened against almost nothing, and the list of stuff you ought to set up before you trust it with anything real is long. The natural instinct is to write one big “set up the account” module that does the whole list in a single apply. I want to talk you out of that, because the bootstrap module I’m happiest with does almost nothing, on purpose.

The first-apply problem

A brand-new AWS account is not ready for anything serious. Before you’d responsibly run real infrastructure into it, you want an account baseline: a password policy, account-wide S3 public-access blocking, default EBS encryption, CloudTrail, AWS Config, GuardDuty, alerting, a sensible human operator role. It’s a long list, and all of it matters.

The instinct, faced with that list, is to write one big “set up the account” module and have it do everything. One tofu apply, a fully prepared account, done.

That instinct is worth resisting, and terraform-aws-bootstrap resists it deliberately.

Three things, and a hard line

terraform-aws-bootstrap does three things:

state-backend, an S3 bucket and a customer-managed KMS key to hold remote Terraform state.
automation-iam, an OIDC identity provider and an IAM role that CI assumes to apply everything else.
nuke-config, which renders an aws-nuke configuration scoped to the account, for tearing a throwaway account back down.

That’s the whole module. Account hardening, CloudTrail, AWS Config, GuardDuty, the operator role, the alerting: none of it is in here. And it’s not absent by accident. The README has a section headed “what’s deliberately NOT in scope” that lists those exclusions out loud. The boundary is written down, because the boundary is the design.

Why the line is exactly there

The reason the line sits where it does is the most useful idea in the module.

Everything bootstrap excludes belongs in a separate stack, applied through the automation role bootstrap creates. Bootstrap’s only job is to get the account to the point where the next tofu apply can run properly: somewhere to store state, and an identity to run as. Once those two things exist, hardening the account isn’t a special bootstrapping act. It’s just another apply, done the normal way: in CI, reviewed, versioned, deployed through the role.

So the account baseline doesn’t need to be bundled into the bootstrap. It needs to be downstream of it. Bootstrap builds the on-ramp; it doesn’t also have to be the motorway.

A narrow module stays re-runnable

There’s a practical payoff to the narrowness, and it’s about fear.

Bootstrap is the one stack that can’t be applied through CI, because it’s what creates the CI identity in the first place. It runs locally, by a human, rarely. That’s exactly the kind of operation you want to be small, boring, and safe to repeat.

A bootstrap module that also did account hardening would be a large, stateful thing managing dozens of resources. Re-running it would be a held-breath operation. Keeping it to three concerns keeps it the opposite: a small stack you can read top to bottom, re-run without anxiety, and reason about completely. The narrowness isn’t minimalism for its own sake. It’s what keeps the one human-applied stack trustworthy.

The boundary is the feature

It’s tempting to judge a module by how much it does. A bootstrap module is the case where that’s exactly backwards. Its value is in how cleanly it stops.

terraform-aws-bootstrap does the bare minimum to make an account ready for the next apply, writes down everything it refuses to do, and hands off to a downstream stack for all of it. The next post follows the trickiest of its three jobs: the state backend has a genuine chicken-and-egg problem, because it has to store Terraform state in a bucket Terraform hasn’t created yet.

Where this leaves us

A fresh AWS account needs a long list of things before it’s safe, and the obvious move is one big module that does the lot. terraform-aws-bootstrap deliberately does only three: a state backend, a CI identity, and an account-scrub config. Everything else is written down as out of scope.

The boundary is the design. The excluded work belongs in a downstream stack applied through the CI role bootstrap creates, so hardening is just a normal reviewed apply rather than a bootstrapping special case. And keeping the one human-run, locally-applied stack small is what keeps it safe to re-run. A bootstrap module is judged by where it stops.

Two kinds of feature flag

Thu, 30 Apr 2026 00:00:00 +0000

go-tool-base has feature flags: switches that decide which built-in commands are live in a given run. rust-tool-base has those too. But it also has a second, completely separate kind of flag, and the difference between them is one of those distinctions that’s obvious the moment you see it and dangerously easy to conflate before you do. One decides what a command does. The other decides whether a chunk of code is in the binary at all.

A workspace of crates

Before the flags, the shape that makes them possible. go-tool-base is one Go module with packages under pkg/. rust-tool-base is a Cargo workspace of seventeen crates: rtb-app, rtb-config, rtb-cli, rtb-vcs, rtb-ai, rtb-mcp, rtb-docs, rtb-telemetry, and so on, with an umbrella crate called rtb that re-exports the public surface.

That isn’t tidiness for its own sake. Each subsystem being a separately compilable crate is what gives you a unit you can include or exclude wholesale. Hold onto that, because it’s the hinge for everything below.

The flag go-tool-base already has

go-tool-base has feature flags, and I’d describe them as runtime flags. A tool built on it can enable or disable built-in commands:

props.SetFeatures(
 props.Disable(props.InitCmd),
 props.Enable(props.AiCmd),
)

At startup the framework resolves that set and decides which commands are reachable for this run. The init command might be present in the binary but switched off; the ai command might be switched on. It’s about the user-facing surface: which commands exist for someone typing --help.

rust-tool-base keeps this idea. A command carries a CommandSpec with an optional feature field, and the runtime decides whether a feature-gated command is reachable. Same purpose: shape the surface per invocation.

If that were the whole story, there’d be nothing to write. The reason there’s a post is the other kind of flag, which Rust makes available and Go really doesn’t.

The flag Rust adds

Cargo features are a compile-time mechanism. The rtb umbrella crate declares them like this:

[features]
default = ["cli", "update", "docs", "mcp", "credentials", "tui"]
cli = ["dep:rtb-cli"]
update = ["dep:rtb-update"]
ai = ["dep:rtb-ai", "rtb-docs?/ai"]
vcs = ["dep:rtb-vcs"]
telemetry = ["dep:rtb-telemetry"]
full = ["cli", "update", "docs", "mcp", "ai", "credentials", "tui", "telemetry", "vcs"]

Each subsystem is an optional crate dependency, and a feature switches it on. This is a different kind of switch entirely, and the difference is the whole point.

A runtime flag decides what a command does while the program runs. The code is in the binary either way; the flag just gates it.

A Cargo feature decides what’s in the binary in the first place. Build a tool without the vcs feature and rtb-vcs is not compiled. Its dependencies are not compiled. gix, the pure-Rust Git implementation rtb-vcs pulls in, roughly two and a half megabytes of it, is not compiled and not linked. It isn’t switched off in the binary. It was never in the binary. The compiler never even saw it.

That’s something a runtime flag cannot do, because by the time anything runs, the binary already exists with everything in it.

Two axes, kept separate

So rust-tool-base has two flag systems answering two genuinely different questions.

Cargo features answer: what is this binary made of? They’re decided when you build the tool, in Cargo.toml. They control compilation, binary size, dependency surface, and compile time. A tool that never touches Git builds without vcs and is smaller, faster to compile, and has a smaller dependency tree to audit. A tool that wants everything turns on full.

Runtime feature flags answer: what can the user do in this run? They’re decided as the program starts. They control which commands appear, which paths are reachable.

These could have been mashed into one mechanism, and it would have been a mistake. The app-context design notes are blunt about it: feature gating doesn’t belong on the per-command context object, because a feature-gated command “either exists or doesn’t” rather than changing its behaviour mid-run. Compile-time composition is one decision, made by the person building the tool. Runtime gating is another, made per invocation. Conflating them would mean you couldn’t reason cleanly about either.

The Go version of this had to be hand-built

This isn’t a thing Go simply lacks. I wrote a whole post about how go-tool-base keeps its optional keychain dependency out of binaries that don’t want it, using a blank import and the linker’s dead-code elimination. It works. But it was a piece of deliberate engineering for one dependency, and getting it right took care.

Cargo features make that same outcome a first-class, declarative thing, and not for one dependency but for every subsystem the framework has. You don’t engineer the exclusion. You name a feature and leave it off. The crate, and its whole subtree, stays out. Rust’s build system was designed for exactly this, and rust-tool-base leans on it across the entire workspace rather than hand-rolling it once.

Where this leaves us

go-tool-base has runtime feature flags: they decide, per invocation, which built-in commands are reachable. rust-tool-base keeps that, and adds a second kind that Rust makes available.

Cargo features decide what the binary is compiled from. Each of the framework’s seventeen crates is an optional dependency, and a feature switched off means that crate and its entire dependency subtree are never compiled or linked. A runtime flag gates what code does; a Cargo feature gates whether code is there at all. Two axes, two questions, deliberately kept as separate systems.

A framework that contains no unsafe

Tue, 28 Apr 2026 00:00:00 +0000

“It’s written in Rust” gets thrown around as if it were a memory-safety guarantee. It mostly isn’t. Rust is memory-safe by default, which is a wonderful thing, but the unsafe keyword exists precisely so any crate, any module, can step outside that default when it needs to. So “written in Rust” really means “mostly safe, probably”. rust-tool-base makes the stronger claim about its own code, and gets the compiler to enforce it.

Safe by default is not the same as safe

People reach for Rust because of memory safety, and the reputation is earned. Write ordinary Rust and the compiler will not let you have a use-after-free, a data race, or a buffer overrun. That’s the default, and it’s a very good default.

But it’s a default, and defaults can be turned off. Rust has an unsafe keyword precisely so that, when you genuinely need to, you can dereference a raw pointer, call into C, or tell the compiler you’ve upheld an invariant it can’t check itself. Inside an unsafe block, the guarantees are yours to maintain, not the compiler’s to enforce.

That keyword has to exist. Some of the most foundational crates in the ecosystem are built on it, carefully. But it means a fact worth being precise about: a project being “written in Rust” tells you its code is mostly safe. It does not tell you the project’s own code contains no unsafe. Those are different claims, and only the second one is a guarantee.

rust-tool-base makes the second claim about its own code, and has the compiler back it up.

`forbid`, not just `deny`

The mechanism is one line at the top of every crate:

#![forbid(unsafe_code)]

unsafe_code is a lint, and Rust lints have levels. The interesting choice is forbid rather than deny, because the two are not the same strength.

deny makes the lint an error. But it’s an error a downstream module can locally override. Anyone can write #[allow(unsafe_code)] on a function or a block and the deny is lifted right there. As a policy, deny is “don’t do this unless you really mean to”, and “unless you really mean to” is a door.

forbid is the strict one. It makes the lint an error and it makes that error impossible to override from inside the crate. A module cannot #[allow] its way back out. Once a crate root says #![forbid(unsafe_code)], there’s no unsafe anywhere in that crate, and no local exception can be carved out. The compiler simply refuses.

So every rust-tool-base crate that ships in a built tool forbids unsafe at its root. Not “discourages”. Cannot contain it.

The one honest subtlety

There’s a wrinkle, and it’s worth showing rather than hiding, because it’s where the design got specific.

The workspace sets unsafe_code = "deny" as the baseline for everything, including test files. But test code occasionally has a real need for unsafe. In the 2024 edition, std::env::set_var became unsafe, because mutating the process environment isn’t thread-safe, and a test that exercises environment-driven configuration has to call it.

So the split is deliberate. The workspace-wide level is deny, which a test file can locally #[allow] when it genuinely needs that one environment call. But every production lib.rs and main.rs additionally carries #![forbid(unsafe_code)], and forbid cannot be relaxed. Test scaffolding gets a controlled, visible exception for a specific standard-library call. Shipping code gets none. The guarantee that matters, “the code in the binary contains no unsafe”, holds, and the place it’s slightly loosened is exactly the place that never reaches a user.

What the guarantee is actually worth

Two things, one for users and one for reviewers.

For users: an entire family of bug is ruled out of first-party code mechanically. Use-after-free, double-free, data races on shared memory, reading off the end of a buffer. These are the classic memory-safety vulnerabilities, and in a crate that forbids unsafe they cannot originate, because the constructs that produce them cannot be written. That’s not careful coding. It’s the compiler refusing to build anything else.

For reviewers: the cost of an unsafe block is mostly the review burden it carries. Every one is a spot where a human has to check, by hand, that an invariant holds, and has to re-check it whenever nearby code changes. A crate that forbids unsafe has zero of those. There’s no unsafe block to audit, ever, because the compiler guarantees there isn’t one.

I’ll be straight about the boundary: this is a promise about rust-tool-base’s own code. Its dependencies are another matter, and some of them do contain unsafe, correctly. Keeping that side honest is a different job, done by vetting the dependency tree and gating it in CI. Within first-party code, though, the guarantee is real, and there’s no Go equivalent to it. Go has an unsafe package, but nothing that lets a codebase prove, to the compiler, that it never touches it.

The bottom line

Rust is memory-safe by default, but the unsafe keyword exists so that default can be set aside. “Written in Rust” therefore does not by itself mean a project’s own code contains no unsafe.

rust-tool-base makes that the stronger claim. Every crate root carries #![forbid(unsafe_code)], and forbid, unlike deny, cannot be overridden from inside the crate. Test files get a narrow, visible deny-level exception for the one standard-library call that needs it; shipping code gets none. The payoff is a whole class of memory-safety bug ruled out of first-party code by construction, and not one unsafe block left for a reviewer to audit.

Reloading config without a restart

Mon, 27 Apr 2026 00:00:00 +0000

A config file changes. Someone edits a setting, rotates a credential, flips a feature flag. How does the running process find out? For most processes the answer is blunt: it doesn’t, until you restart it. For a short-lived CLI that’s completely fine. For a long-running service, “just restart it” is a much bigger ask than it sounds.

The default answer is a restart

Configuration lives in a file. The file changes: someone edits a setting, rotates a credential, flips a feature flag. How does the running process find out?

Overwhelmingly, the honest answer is that it doesn’t. A process reads its config once, at startup, and that snapshot is frozen for the life of the process. Change the file and nothing happens until you restart, at which point a fresh process reads the fresh file.

For a short-lived CLI invocation that’s completely fine. It reads config, does its job, exits, and the next invocation reads whatever the file says then. But the same frameworks are also used to build long-running services, and for a service “just restart it” is not the small thing it sounds like.

What a restart actually costs

Restarting a long-running service means every open connection drops. Any in-flight request is lost, or has to be retried by whoever sent it. Caches that took real time to warm are cold again. There’s a window, short but real, where the service simply isn’t serving.

If the thing you changed was a log level, or a feature flag, or a timeout, you’ve paid a disruption wildly out of proportion to the change. And the calculation only gets worse as the service gets more important, because the services you least want to bounce on a whim are exactly the ones that matter most.

Hot-reload: re-read in place

Hot-reload is the alternative, and both go-tool-base and rust-tool-base support it.

The process doesn’t read config once and freeze it. It watches the config file. When the file changes, it re-reads it, re-applies it, and carries on running. No new process, no dropped connections, no cold start. The change lands in the live process.

The shape is the same in both frameworks:

A file watcher notices the config file changed. Underneath, this is the operating system’s own file-notification facility, inotify on Linux and its equivalents elsewhere. rust-tool-base reaches it through the notify crate; go-tool-base, through the watcher built into Viper.
A debounce step waits for the writes to settle. Saving a file is often several separate operations, and you don’t want to reload three times for one edit.
The config is re-parsed from disk.
The new config is swapped in atomically.
Observers are notified, so the subsystems that care can react.

Steps four and five are the ones worth slowing down on, because they’re where a naive hot-reload quietly goes wrong.

The two details that make it safe

The atomic swap. You do not mutate the live config object in place. A reader on another thread, partway through reading it, would see a torn mix of old and new values, and that’s a genuinely nasty class of bug. Instead the process builds a new, complete config value and swaps the pointer to it in a single atomic operation. Any reader sees either the entire old config or the entire new one, never a blend. rust-tool-base does this with arc-swap; go-tool-base does the equivalent. Reads stay cheap and lock-free, and an update is one pointer swap.

The observer notification. Re-reading the file isn’t the end of the job. Some subsystems have to do something when config changes: a connection pool resizes, a logger changes level, a rate limiter takes a new ceiling. So a hot-reload system has to let those subsystems subscribe. rust-tool-base hands observers a watch::Receiver, a channel that always holds the latest value; go-tool-base exposes an Observable interface. A subsystem subscribes once and reacts every time config changes, for the life of the process.

Where this earns its keep: a Kubernetes pod

Hot-reload is a nicety on a developer’s laptop. Inside a Kubernetes pod it becomes genuinely valuable, and the reason is a neat fit between how Kubernetes delivers config and how a file watcher works.

In Kubernetes you don’t usually bake configuration into the container image. It lives in ConfigMap and Secret objects, and the clean way to consume them is to mount them as volumes. Mount a ConfigMap as a volume and each key becomes a file in the pod’s filesystem.

Here’s the part that connects to everything above. When you update that ConfigMap or Secret, Kubernetes does not restart your pod. The kubelet notices the object changed and rewrites the projected files inside the still-running pod. The files on disk change underneath a process that never stopped.

That file rewrite is exactly the event a hot-reload watcher exists to catch. So the whole chain becomes:

You kubectl apply an updated ConfigMap, or rotate a Secret.
The kubelet updates the projected files inside the pod.
The framework’s file watcher sees the write.
The config is re-parsed, swapped in atomically, and observers are notified.
The new configuration is live, and the pod never cycled.

You’ve changed a running service, in a running pod, with no rollout, nothing terminated and recreated, no dropped traffic. Rotate a database credential, raise a log level to debug an incident in progress, flip a feature flag: all of it live. For a service where a restart is the very thing you’re trying hard to avoid, the kind of long-running service these frameworks are built for, that’s the difference between a config change being routine and being an event.

The honest caveats

Two things, so this doesn’t read as magic.

First, not everything can be hot-reloaded. Some configuration genuinely needs a restart: the port a server binds to, the size of a thread pool, anything wired up exactly once at process start. Hot-reload covers the large category of settings a subsystem can re-read and re-apply; it doesn’t abolish restarts. A config system worth its salt is clear about which settings are live and which are not.

Second, a Kubernetes gotcha that catches people out. The in-place file update happens for ConfigMaps and Secrets mounted as volumes. Consume the same ConfigMap as environment variables instead, and those are fixed when the container starts and never update, short of a restart. If you want hot-reload in a pod, mount config and secrets as files, not env vars. And even with volumes the update isn’t instant: the kubelet syncs on a period, around a minute by default, so a reload is “within a minute or so”, not “the moment you hit apply”.

What it comes down to

A config file changes, and the default way to pick it up is to restart the process. For a long-running service that restart costs dropped connections, lost work and a cold start, often for a change as small as a log level.

go-tool-base and rust-tool-base both support hot-reload instead: a file watcher catches the change, the config is re-parsed and swapped in atomically so no reader sees torn state, and observers are notified so subsystems can react, all in a live process. The setting where it pays off most is a Kubernetes pod, where ConfigMaps and Secrets mounted as volumes are rewritten in place by the kubelet and the watcher catches that write directly. Mount them as volumes rather than env vars, allow for the kubelet’s sync delay, accept that some settings still need a restart, and within those limits “the config changed” stops meaning “cycle the pod”.

A signing key needs somewhere to live

Sun, 26 Apr 2026 00:00:00 +0000

I left a door open a couple of posts ago, and it’s been quietly bothering me ever since. When I wrote about verifying your own downloads, I was honest that a checksum sitting next to the binary only catches accidents. Anyone who can compromise the release platform can swap the binary and the checksum together, and the tool will happily verify one fake against the other.

Closing that gap needs a signature. And a signature, it turns out, needs a surprising amount of infrastructure standing behind it. This is the first post about building that.

The door the last post left open

A while back I wrote about verifying your own downloads: go-tool-base’s self-update command now checks the SHA-256 of every binary it downloads against the release’s published checksums.txt before installing it.

That post was honest about its own ceiling. A checksum file hosted next to the binary it describes shares a trust root with that binary. Both come from the same release, on the same platform. Corruption, truncation, a CDN serving a stale object: a same-origin checksum catches all of those, because they’re accidents and the checksum wasn’t part of the accident. What it can’t catch is an attacker who’s compromised the release platform itself. Someone who can replace the binary can replace checksums.txt in the same breath, and the tool will cheerfully verify the malicious download against the malicious checksum and call it good.

The post named the fix and then deferred it: a signature whose trust root sits somewhere the release platform can’t reach. “That’s the next phase of this work.” This series is that phase.

What a signature actually needs

It’s worth being precise about why a signature helps where a checksum doesn’t, because it’s easy to wave the word “signature” around and assume it settles everything.

A signature closes the gap only under two conditions. The verifying key, the public half, must reach the user by a path the release platform doesn’t control. And the signing key, the private half, must live somewhere the release platform can’t reach.

The second condition is the one people skip. If the signing key sits in the same CI system that builds the release, you’ve gained almost nothing. An attacker who owns the CI owns the key, and a key they own will sign whatever they hand it. The signature verifies perfectly and means precisely nothing. A signature is only worth the distance between the signing key and the thing being signed. Put them in the same place and the distance is zero.

So the signing key has to live in a different security domain from the release pipeline. Not a different folder. A different account, with a different blast radius, that the release platform has no standing access to.

“Just sign the binary” is not a small feature

That reframes a line item that sounds tiny. “Sign the release binary” unpacks into a list:

there must be a private signing key;
it must live outside the release platform, in its own security domain;
it must be access-controlled, audited, and protected from exfiltration;
only the release pipeline may ask it to sign, and only by proving a short-lived, federated identity, never by holding a copy of the key.

That’s not a feature you bolt onto a CLI. That’s infrastructure.

The shape of it: a cloud account, with the key held in a managed key service so the private key material never exists as a file on a disk that anyone, me included, can copy. The release pipeline authenticates to that account as itself, briefly, and asks the key service to produce a signature. The key never moves.

But an account you’re going to trust with a signing key is itself something you have to get right first. An account with a weak baseline, no audit trail, and long-lived credentials lying around is not a safe home for the most security-sensitive key in the whole system. Before the key can move in, the house has to be built and the locks have to actually work.

What this series builds

So this turned into a rather longer project than “add a signature”, and the series follows it in order.

It starts with bootstrapping a fresh AWS account: the deliberately minimal first tofu apply, and the remote state backend that has a genuine chicken-and-egg problem. Then the credential question, which is the heart of it: how a CI pipeline deploys to AWS with no stored access key at all. Then hardening the account, so it’s genuinely safe to hold something valuable. Then the discipline of deploying changes to it: plans reviewed before they’re applied. Then the shared tooling that makes all of it repeatable.

Every one of those pieces exists for the same reason. The signing key needs somewhere to live, and somewhere safe is not a default you’re handed. It’s a thing you build, deliberately, before you have anything worth protecting in it.

The series ends where the verifying-downloads post pointed: a signing service whose key the release platform can’t touch, so a self-updating tool can finally verify that the binary it’s about to become is genuinely the one I published.

The upshot

go-tool-base’s self-update verifies downloads against a checksum, and a same-origin checksum stops accidents but not a compromise of the release platform. The fix is a signature, and a signature is only worth the distance between its signing key and the release pipeline.

Holding that key safely means a private key that never leaves a managed key service, in a separate cloud account, reached only by a short-lived federated identity. That’s infrastructure, and a safe account is something you build before you trust it with anything. The rest of this series builds it, piece by piece, right up to the signing service itself.

Waivers with an expiry date

Sun, 26 Apr 2026 00:00:00 +0000

A vulnerability scanner gives you a yes or a no. Is there a known advisory on a path you actually use? Yes, or no. That’s genuinely useful, and you should run one. But it’s a snapshot, taken on the day you ask, and supply-chain risk in a framework is a bigger and more ongoing thing than a single yes-or-no can capture.

So rust-tool-base treats its whole dependency tree as something to have a policy about, not something to scan and forget.

A scanner answers one question

When I had go-tool-base security-audited, part of the routine was running a vulnerability scanner over the dependencies. Go has a good one. It looks at your dependency graph, cross-references known advisories, and tells you whether any of them reach code you actually call.

That’s useful and you should do it. But notice the shape of what it gives back: essentially a yes or a no. Either there’s a known vulnerability on a reachable path or there isn’t. It answers one question, on the day you ask it.

Supply-chain risk in a framework is broader than that one question, because a framework drags its entire dependency tree into every tool built on it. rust-tool-base treats the whole tree as something to have a policy about, and the tool for that is cargo-deny.

A gate, not a scan

cargo-deny reads a deny.toml and checks the dependency graph against four kinds of rule.

Licences. There’s an allowlist: MIT, Apache-2.0, the BSD variants, ISC, a handful of others. Every transitive crate’s licence has to be on it. A dependency that pulls in something copyleft, or something with no licence at all, fails the build. You find out the first time it enters the tree, not during a release scramble when someone finally reads the legal implications.

Advisories. It checks the RustSec advisory database, and yanked crates are set to deny, so a dependency that’s been pulled from the registry stops CI.

Bans. Wildcard version requirements (version = "*") are denied outright, because a dependency that floats to whatever’s newest is a supply-chain hole by construction. Duplicate versions of the same crate get surfaced too.

Sources. Crates may only come from the official registry. An unknown registry or a stray git dependency is denied. Nothing sneaks in from a URL.

That’s a gate. It encodes, as rules in a file, what the project will and won’t accept into its dependency tree, and it enforces them on every build instead of once an audit.

The honest part is the waiver list

Here’s the thing every real project runs into. Sooner or later there’s an advisory you genuinely can’t fix this week. It’s against a crate three levels down your tree. The fix needs an upstream release that hasn’t happened. The crate is scheduled to be reworked two milestones from now anyway. The gate is going to fail, and the work to satisfy it honestly isn’t available to you yet.

The lazy response is a blanket ignore: silence the advisory, move on, forget. Now your gate has a hole in it that nobody remembers opening.

rust-tool-base’s deny.toml does something better. Every waiver in the ignore list is a documented record. Each one carries a comment that names the crate, traces the exact dependency path that reaches it, gives the reason, and names the condition that lifts it:

ignore = [
 # `instant` - reached via async-openai -> backoff -> rtb-ai (v0.3).
 "RUSTSEC-2024-0384",
 # `paste` - reached via ratatui -> rtb-docs (v0.2) / rtb-tui (v0.4).
 "RUSTSEC-2024-0436",
 # ...
]

The file states the policy out loud: “Every waiver points at a deferred stub crate that will be reworked before its ship milestone. Lift each waiver when the owning crate lands its v0.1.”

Some waivers go further and carry a structured reason field, so the why travels with the entry rather than living only in a comment above it:

{ id = "RUSTSEC-2025-0140",
 reason = "gix-date via gix is a stub dependency; rtb-vcs v0.5 will upgrade" },

Read that list and you don’t see a project that quietly stopped caring about seven advisories. You see seven advisories the project knows about, can trace, and has tied to a specific milestone. The waiver has an expiry condition. When rtb-vcs reaches v0.5, that gix entry is meant to come out, and the comment is the reminder that it should.

Why this is the bit to copy

A gate that can’t be relaxed is a gate people route around. They’ll find the broadest possible ignore and use it, because the alternative is being blocked on someone else’s release. The pressure to do that is real, and it’s not unreasonable.

So the design that actually holds up isn’t a stricter gate. It’s a gate with an honest, structured escape hatch: you can waive an advisory, but a waiver costs you a documented record with a dependency path and an expiry condition. That price is small enough that nobody routes around it, and high enough that waivers don’t accumulate silently. The ignore list stays readable, and every line in it is something you could defend out loud.

Supply-chain hygiene framed this way isn’t an audit you survive once a year. It’s bookkeeping: a ledger of what you accepted, why, and when each exception is due to close. Which, now I write it down, is just the Boy Scout rule again, pointed at a dependency tree. Leave it tidier than you found it, and write down the bits you couldn’t tidy yet.

Where this leaves us

A vulnerability scanner answers one question on one day. cargo-deny is a standing policy gate: licences against an allowlist, advisories and yanked crates denied, wildcard versions banned, sources restricted to the official registry, enforced on every build.

The part of rust-tool-base’s setup worth copying is the waiver list. Every advisory that can’t be fixed yet is recorded with its crate, its dependency path, its reason and the milestone that removes it. A waiver is a dated note, not a shrug, and that’s what keeps the gate honest enough that nobody actually wants to bypass it.

A builder that won't compile if you forget a field

Sat, 25 Apr 2026 00:00:00 +0000

go-tool-base configures things with functional options, and if you forget a required one, the best case is a runtime failure and the worst case is an empty value sailing silently into everything downstream. Most builder patterns share the same hole. rust-tool-base closes it in a way I find genuinely delightful: the .build() method simply doesn’t exist until you’ve set every required field.

When is a required field actually required

Every framework has constructors with a mix of required and optional inputs. An Application in rust-tool-base needs tool metadata and a version. It optionally takes a custom config type, extra commands, feature toggles. The metadata needs a name and a summary; a description and a help channel are optional.

The interesting question is when “required” gets enforced. There are really only two moments available: when the program runs, or when it compiles. Most APIs pick the first without ever framing it as a choice.

How go-tool-base does it

go-tool-base uses functional options, the standard Go pattern:

tool := props.New(
 props.WithName("mytool"),
 props.WithVersion(version),
)

New takes a variadic list of options and applies them. It’s flexible and it reads well. But look at what the type actually says. New accepts zero or more options. The signature is satisfied by passing nothing at all. If WithName is required, nothing in the type system knows that. Forget it and the code compiles cleanly, and you find out when the program runs, or worse, when it doesn’t visibly fail but quietly carries an empty name into everything downstream.

A plain builder is no better here. builder.name("mytool").build() and builder.build() are both perfectly valid calls as far as the compiler is concerned. The builder hopes you set the name. It can check at the end and return an error, but that check still happens at runtime.

In every one of these the required-ness of a field is a fact that lives in documentation and in the author’s head, not in the code.

Typestate: putting “required” in the type

rust-tool-base builds these with bon, and the pattern it generates is a typestate builder. The idea is that the builder’s type changes as you call it, and that type tracks which required fields you’ve set so far.

let metadata = ToolMetadata::builder()
 .name("mytool")
 .summary("my CLI tool")
 .build();

ToolMetadata::builder() returns a builder in a state that records “name not set, summary not set”. Calling .name(...) consumes that builder and returns a different type, one whose state records “name set”. Calling .summary(...) does the same for the summary.

The part that matters is .build(). It isn’t a method on the builder in general. It only exists on the builder type that represents “every required field has been set”. So this:

let metadata = ToolMetadata::builder()
 .summary("my CLI tool")
 .build();

doesn’t compile. Not because a runtime check fired, but because in the state “name not set” there’s no .build() method to call in the first place. The compiler stops you, and the error points straight at the missing .name(...).

Optional fields stay optional. You can call .description(...) or skip it, and .build() is reachable either way, because the description was never part of the state that gates it. The required and the optional are genuinely different in the type, which is exactly the distinction the functional-options version could only keep in a comment.

Application::builder() works the same way. It won’t produce an Application until it has metadata and a version, and “won’t” there means the method is absent, not that a check returns Err.

Why the moment matters

Moving the check from run time to compile time changes who finds the mistake, and when.

A runtime check finds it when that code path executes, which might be in a test, might be in CI, might be on a user’s machine at the worst possible moment. A compile-time check finds it the instant you write it, in the editor, before anything has run at all. The same mistake, caught at the cheapest possible point instead of one of the expensive ones.

It also changes what the API documents about itself. A functional-options constructor can’t tell you, from its signature alone, which options you must pass. A typestate builder can, because the set of methods available to you at each step is the documentation. You literally cannot reach .build() without having been walked past every required field on the way.

This is one of those places where Rust’s type system earns its reputation. The builder isn’t more careful than the Go version. It’s that “this field is required” stopped being a convention and became something the compiler enforces. (Another entry, if you’re keeping score from the porting post, in the column of outcomes that survived while the Go mechanism got left behind.)

The short version

Required fields have to be enforced somewhere. Functional options and ordinary builders enforce them at runtime, if at all, because .build() is always callable and the type system never learns which inputs were mandatory.

rust-tool-base uses typestate builders generated by bon. The builder’s type changes as you set fields, and .build() only exists once every required field is present. Forgetting one is a compile error that names the missing call, not a runtime surprise. The required-versus-optional distinction stops being a comment and becomes part of the type.

Registering commands without life before main

Fri, 24 Apr 2026 00:00:00 +0000

I ended the last post promising to show how a Rust command registers itself when the language flatly refuses to run any of your code before main(). This is that post, and it’s a lovely example of reaching the same outcome by a completely different road.

The outcome I wanted to keep is self-registration.

What self-registration buys

A command in go-tool-base lives in its own file, and that file puts the command into the framework itself. There’s no central list of commands to keep in sync. You add a file, the command appears. You delete the file, it’s gone. Nothing else changes.

That property is worth protecting. The alternative, a hand-maintained registry that every new command has to be threaded into, is exactly the sort of central file that turns into a merge-conflict magnet and quietly falls out of date. So when go-tool-base moved to Rust, self-registration was firmly in the column of things that had to survive.

The way Go did it was not.

How Go does it

A Go package can declare an init() function, and the runtime guarantees every init() runs before main() starts. A go-tool-base command file uses this to append itself to a package-level slice:

func init() {
 registry.Register(&DeployCommand{})
}

By the time main() runs, every command file’s init() has already fired and the registry slice is populated. It’s a tidy trick, and it leans entirely on a Go feature: code that executes before main().

Rust doesn’t have that

Rust has no init(). There’s no language-blessed phase that runs your code before main(). This is a deliberate decision, not an oversight. Code running before main() across many files has no well-defined order, and a startup phase whose ordering you can’t see is a classic source of subtle, miserable bugs. Rust closed that door on purpose.

Which leaves a real question. If nothing runs before main(), how does a command file insert itself into a registry without a central list editing it in?

Distributed slices

The answer is a crate called linkme, and the mechanism is the linker rather than a runtime phase.

You declare a slice the framework will collect into:

#[distributed_slice]
pub static BUILTIN_COMMANDS: [fn() -> Box<dyn Command>];

A command file then contributes one entry to it:

struct Greet;

impl Command for Greet { /* ... */ }

#[distributed_slice(BUILTIN_COMMANDS)]
fn register_greet() -> Box<dyn Command> {
 Box::new(Greet)
}

Here’s the part that makes it work. The #[distributed_slice] attribute doesn’t generate any code that runs at startup. It places each entry into a dedicated section of the compiled object file. When the linker builds the final binary, it gathers everything in that section and lays it out as one contiguous array. BUILTIN_COMMANDS is that array.

So by the time the program exists as a binary on disk, the registry is already assembled. main() doesn’t build it. No init() builds it. The linker built it, statically, as part of producing the executable. At runtime the framework iterates a slice that was complete before the process ever started.

What you get from it

The outcome is the one Go’s init() gave, and then a bit more.

A command still lives in one file and still self-registers. Adding a command is still adding a file. There’s still no central list.

But there’s no startup phase to reason about, because there isn’t one. There’s no global mutable slice being appended to as init()s fire, because nothing is appended at runtime; the slice is immutable and finished. There’s no ordering question, because the linker isn’t running your code, it’s collecting data. And it costs nothing at runtime: assembling the registry happened at link time, so program start just reads it.

It’s the same idea go-tool-base had, expressed by the tool Rust actually gives you. Go reaches the registry through a controlled phase before main(). Rust reaches it without any phase at all, because the linker did the assembly while the binary was still being built. Two roads, one destination… which, if you’ve been following along, is becoming the whole theme of the Rust side of this project.

In short

Self-registration, where a command file inserts itself into the framework with no central list, is a property worth keeping. go-tool-base achieves it with a package-level init(), leaning on Go’s guarantee that such functions run before main().

Rust has no equivalent and wants none, because code running before main() has no clear ordering. rust-tool-base uses linkme distributed slices instead: each command is placed into a dedicated linker section, and the linker assembles them into one contiguous, immutable slice as it builds the binary. The registry is complete before the program runs. Same outcome as Go’s init(), with no life before main required.

Verifying your own downloads: how I solved it for self-updating CLI tools

Fri, 24 Apr 2026 00:00:00 +0000

Way back in the introduction I promised I’d come back to the self-update integrity checks. Here we are. And the honest starting point is a slightly uncomfortable admission: for a good long while, go-tool-base’s update command was the most trusting line of code in the entire tool.

The most trusting line of code in the tool

Self-update is a lovely feature. The user runs yourtool update, the tool fetches the latest release, swaps itself out, and they’re current. go-tool-base has had this since early on, wired to GitHub, GitLab, Bitbucket, Gitea and a few others.

But look closely at what that feature actually does. It reaches out to the internet, pulls down a file, and then replaces the executable that’s currently running with that file. The next time the user invokes the tool, they’re running whatever those bytes turned out to be.

The original implementation downloaded the release asset over HTTPS and extracted it. HTTPS gets you transport security: the bytes weren’t tampered with in flight. It tells you nothing about whether the bytes were right when they left, or whether they’re even the bytes you meant to fetch. A truncated download, a CDN cache serving a mangled object, a release asset that got swapped after the fact… HTTPS waves all of those straight through. For the one operation in the whole tool that replaces the binary, “we didn’t check” is an uncomfortable place to be sitting.

GoReleaser already does half the job

The good news is that the build side was already producing exactly what I needed. GoReleaser, which builds go-tool-base’s releases, generates a checksums.txt for every release: one SHA-256 per published artefact, the same format sha256sum emits. It was sitting right there as a release asset and nothing was reading it.

So Phase 1 of the integrity work is exactly that: read it.

When update downloads the platform binary, it now also fetches checksums.txt from the same release, looks up the entry for the asset it just pulled, and compares the SHA-256 of the downloaded bytes against the expected hash before anything gets extracted or installed. Mismatch, and the update aborts before it has so much as touched the installed binary. The hash comparison runs in constant time, which is more defence-in-depth than strictly necessary here, but it costs nothing and means every hash comparison in the codebase is the same and reassuringly audit-boring.

Fail open, or fail closed?

The interesting design question wasn’t the hashing. It was: what do you do when there is no checksums.txt?

Plenty of older releases predate this feature. A release might have been cut by hand without GoReleaser. If go-tool-base flatly refused to update whenever a manifest was missing, the very act of shipping this feature would brick the update path for every existing tool the moment they upgraded into it. That’s a cure worse than the disease.

So the default is fail-open: no manifest, log a clear warning, proceed. It matches how the existing offline-update path already behaved with its optional .sha256 sidecar, and it keeps upgrades working.

Fail-open as a default is not the same as fail-open being right for everyone, though. A security-sensitive tool should be able to say “no manifest, no update, full stop”. Two ways to get there:

Tool authors flip a compile-time switch (setup.DefaultRequireChecksum = true in main()) and their binary ships fail-closed from day one.
End users override either way through config (update.require_checksum) or an environment variable.

go-tool-base itself ships with the strict setting turned on, because a tool whose entire job is being a careful framework should hold itself to the stricter bar.

The honest caveat

Here’s the part I want to be straight about, because security features oversell themselves constantly.

A checksum hosted next to the binary it describes protects you from accidents. Corruption, truncation, a CDN serving stale junk, a release asset that got partially clobbered. It does not protect you from a determined attacker who’s compromised the release platform itself. If someone can replace the binary, they can replace checksums.txt in the same breath, and your tool will cheerfully verify a malicious download against a malicious manifest and pronounce it good.

That’s not a flaw in the implementation. It’s the inherent ceiling of same-origin integrity: the manifest and the artefact share a trust root, so they fall together. Closing that gap needs a signature whose trust root is somewhere the release platform can’t reach, a key the attacker doesn’t have. That’s the next phase of this work, and it’s a bigger piece: GPG-signing the manifest, with the public half both embedded in the binary and published independently so a single platform compromise isn’t enough.

Phase 1 is the floor, not the ceiling. But it’s a floor worth having, because the overwhelming majority of real-world “the download was wrong” incidents are accidents, not attacks, and accidents are exactly what a same-origin checksum catches.

Pulling it together

The update command is the most trusting code in a self-updating tool: it fetches bytes from the internet and then becomes them. go-tool-base now verifies the SHA-256 of every self-update download against the release’s own checksums.txt before installing. It fails open by default so shipping the feature doesn’t strand anyone on an un-updatable version, fails closed for tool authors who ask (go-tool-base itself does), and stays honest that a same-origin checksum stops accidents, not a platform compromise.

Verifying your own downloads is a low bar. The point is that the previous height of that bar was zero.

What survives a port, and what doesn't

Thu, 23 Apr 2026 00:00:00 +0000

Rebuilding go-tool-base in Rust turned out to be the most honest design review I’ve ever sat through, and I didn’t have to do anything except keep going. Porting a framework into a language with completely different idioms forces a separation you can’t fake: the parts that survive the move are design, and the parts that don’t are just habit.

Two columns

When you port a system between languages that don’t share idioms, every piece of it sorts itself into one of two columns, without you having to make the call.

In the first column is the outcome a piece of the design produces: every command receives the framework’s services, configuration is layered with a fixed precedence, commands register themselves, errors carry guidance to the user. In the second column is the mechanism that produced that outcome in the original language.

Things in the first column survive the port. You rebuild them, differently, because the tool genuinely needs them. Things in the second column do not survive. You find their replacement, and the Go version turns out to have been one valid implementation of an idea, not the idea itself. Doing this for go-tool-base, mechanism by mechanism, was more honest about my own design than any amount of sitting and staring at it would have been.

The container

go-tool-base hands every command a Props struct. It carries the logger, the config, the assets, the filesystem handle. Some of it is reached through loosely-typed accessors. It works well, and I wrote a whole post about it.

The outcome is column one: a command should receive one object, and that object should carry the framework’s services so the command doesn’t go assembling them itself. That survived. RTB hands every command an App.

The loosely-typed accessors were column two. In Rust an App is a plain struct with concrete fields, each one an Arc<T> so a clone is a few atomic increments rather than a deep copy. Nothing is keyed by string. Nothing is fetched by name and asserted to a type. The thing the container is for survived; the way Go expressed it did not.

Registration

A go-tool-base command self-registers using a package-level init() function, which Go runs before main() and which appends the command to a global slice.

The outcome, column one, is that a command lives in its own file and inserts itself into the framework with no central list to edit. That’s genuinely worth keeping.

The init() mechanism is column two, and Rust doesn’t even offer it: Rust deliberately has no code that runs before main(). The replacement is link-time registration through distributed slices, which gets its own post next. Same outcome, no global mutable state, assembled by the linker rather than by a startup phase.

Configuration

go-tool-base layers configuration with a precedence: flags over environment over file over defaults. Some of it is read back through key lookups.

The layering and the precedence are column one. They survived exactly. RTB layers config with the same ordering.

The key lookups were column two. In Rust the merged configuration is deserialised into your own serde struct, so a config value is a typed field you access like any other field, and a typo is a compile error instead of a missing key at runtime. The precedence survived; reading values back out of a string-keyed bag did not.

The error path

go-tool-base routes every error through one handler so presentation is consistent, which I also wrote up.

One consistent exit for errors is column one. It survived. What didn’t survive was the handler: RTB has no error-handler object at all, because Rust’s own return-from-main convention plus a report hook does the job the handler was built to do. That one has its own post too.

What the exercise was actually worth

Every mechanism told the same story. The container, the registration, the config access, the error path, the cancellation signal that go-tool-base carries on a context.Context and RTB carries on a CancellationToken. In every case the thing it achieved walked across to Rust untouched, and the Go code that achieved it was left behind.

That’s the useful result. Before this port I couldn’t have told you, for any given pattern in go-tool-base, whether it was load-bearing design or just the idiomatic Go way to write it that day. Now I can, because each one was forced to prove itself by being rebuilt from nothing in a language that flatly wouldn’t accept the original. Whatever survived was real. Whatever I had to replace was always replaceable, which means it was never really the point.

The upshot

Porting a framework into a language with different idioms separates design from habit for free. The outcome a pattern produces is design, and it survives the move. The mechanism that produced it is idiom, and it gets left behind for the new language’s equivalent.

go-tool-base’s Props bag, its init() registration, its key-based config access and its error handler were all idiom. The single context object, self-registration, layered precedence and a consistent error exit were all design, and all four came through to RTB intact. The next three posts take the most interesting replacements one at a time, starting with how a Rust command registers itself when the language won’t run anything before main.

rust-tool-base: the same idea, in a language that argues back

Wed, 22 Apr 2026 00:00:00 +0000

I built go-tool-base because I was sick of rebuilding the same CLI scaffolding every time I started a new Go tool. You’d think that would have taught me a lesson about doing things more than once. Apparently not, because I’ve now started building rust-tool-base: the same idea, the same itch, for Rust.

In my defence, there’s method in it.

The same itch, a different language

go-tool-base exists because I kept writing the same couple of hundred lines of wiring every time I started a new Go CLI. Config loading, logging setup, an update check, an error path, a help system. None of it was the tool. All of it had to be there before the tool could be.

Lately I’ve been learning Rust, and two things collided. The first is how I tend to learn a language. I’ve always picked them up reasonably quickly, and the way I do it isn’t with a tutorial that builds a toy, it’s by rebuilding something whose shape I already know cold, so that every decision is about the language rather than the problem. The second is that every time I started a Rust CLI of any size, I hit the very same gap I’d already filled once in Go.

So rather than learn Rust on a throwaway, I decided to learn it by building rust-tool-base: the same idea, the same niche, for Rust.

The gap in Rust

The Rust ecosystem has a well-earned reputation for sharp, focused crates and a deliberate shortage of big opinionated frameworks. clap for argument parsing, figment for layered config, tracing for logging, miette for errors, ratatui for terminal UI, reqwest and tokio underneath. Each of them is genuinely best-in-class.

What nobody hands you is the assembly. Wiring those into one coherent product, and then adding self-update, AI integration, an MCP server, embedded documentation, credential handling, telemetry and a scaffolder, is real work, and it’s the same work on every project.

The closest existing neighbours stop short of it. cli-batteries is a thin preamble: argument parsing plus a logging subscriber plus panic and signal handling. starbase has a proper session and lifecycle model but is CLI-agnostic and shaped around the moonrepo tooling it came from. cargo-dist and cargo-release are about release packaging, not the runtime. Good tools, all of them, but none is the opinionated, full-lifecycle, scaffolded base that go-tool-base is in the Go world. That space is empty, and rust-tool-base is built to fill it.

Why it is not a port

The obvious way to build this would be to open go-tool-base and translate it file by file. I’m not doing that, and the reason matters enough that it’s the rule the whole project is built around.

go-tool-base is full of Go. It leans on a Props struct that carries the framework’s services in loosely-typed fields. It configures things with functional options. It registers commands using package-level init(). It threads a context.Context through every call. Those are all good, idiomatic Go. Transliterated into Rust they’d become code that argues with the compiler on every single line, because Rust has its own answers to every one of those problems and they are emphatically not the Go answers.

So rust-tool-base reaches the same outcomes by Rust’s means. Commands still self-register, but through link-time machinery instead of init(). There’s still one context object per command, but it’s strongly typed rather than a loosely-keyed bag. Configuration is still layered, but it lands in your own typed struct instead of a string-keyed lookup. Same philosophy, same shape of product, an entirely different ecosystem underneath. The README says it plainly: it’s a sibling, not a port.

Why do it twice at all

Three reasons, and they reinforce each other.

The first is plain usefulness. The next time I want a Rust CLI tool, I want the same head start go-tool-base already gives me in Go.

The second is the learning. Rebuilding a system I understand forces me to meet Rust’s idioms where they actually bite, not where a tutorial gently stages them. You learn ownership properly when a real design is pushing back at you.

The third is the one I didn’t expect, and it’s the subject of the next post. Building the same framework twice, in two languages, turns out to be the cleanest way to find out which of your original decisions were genuine design and which were merely idiom. The design survives the move. The idiom does not. Sorting one from the other has been the most interesting part so far.

Boiling it down

rust-tool-base is the Rust sibling of go-tool-base: the same batteries-included, scaffolded, opinionated CLI framework, aimed at the same gap, which in Rust is the gap between a pile of excellent crates and a coherent product.

It’s not a port. Transliterating Go idioms into Rust produces code that fights the language, so RTB reaches the same outcomes through Rust’s own mechanisms instead. The posts after this one walk through the specific cases: how commands register, how the builder works, how errors are reported, and a few things RTB can do that the Go version structurally can’t. First, though, the thing the exercise taught me about my own design.

The blank import that keeps a dependency out of your binary

Wed, 22 Apr 2026 00:00:00 +0000

go-tool-base can stash your credentials in the OS keychain, which most people building on it are perfectly happy about. But some of them ship into regulated and air-gapped environments where the binary isn’t permitted to contain keychain or session-bus code at all… not dormant, not unused, simply not there.

So I had a feature most users want and a minority must be able to provably not have. The way I ended up solving it is one of my favourite little bits of honest Go.

A feature some users have to be able to not have

go-tool-base needs somewhere to keep secrets: AI provider keys, VCS tokens, the occasional app password. The best home for those on a developer’s machine is the operating system’s own keychain. macOS Keychain, GNOME Keyring or KWallet on Linux via the Secret Service, Windows Credential Manager. So I wanted go-tool-base to support all three. (This is the keychain mode I mentioned back in the credentials post, finally getting the explanation I promised it.)

The Go library for that is go-keyring, and it’s good. The catch is what it drags in behind it. On Linux it talks to the Secret Service over D-Bus, which means godbus. On Windows it pulls wincred. Perfectly reasonable dependencies for a desktop tool.

Now here’s the constraint that made this interesting. Some of the people building tools on go-tool-base don’t ship to developer laptops. They ship into regulated sectors and air-gapped deployments where a security review will scan the binary, enumerate every dependency, and ask pointed questions about anything that does inter-process communication. For those builds, “the keychain code is there but we never call it” is not an acceptable answer. The reviewer’s position, and it’s a fair one, is that code which isn’t in the binary cannot be a finding.

So I had a feature that most users want, and a minority of users must be able to provably not have. Same framework, same release.

Why I didn’t reach for a build tag

The obvious Go answer is a build tag. Compile with -tags keychain to get it, leave the tag off to not. I started down that road. I even spent a while on an inverted version, a nokeychain tag, on the theory that the regulated build should be the one that has to ask, so a forgotten flag fails safe.

It works. It also isn’t very nice. Build tags are invisible at the call site. Nothing in the source tells you that a file only exists in some builds. The two worlds drift, because the tagged-out path isn’t compiled in your normal editor session and quietly rots. And the ergonomics for a downstream consumer are poor: every tool built on go-tool-base would have to know the right magic incantation and thread it through their own release pipeline correctly, forever.

I tried a second approach too: pull the keychain backend out into a completely separate Go module. That genuinely solves the dependency question (a module you don’t require can’t contribute to your go.sum). But a separate module for one backend is clunky. Separate versioning, separate release, separate repo, all for a single file’s worth of behaviour. It felt like using a shipping container to post a letter.

The shape that actually fits: a registry and an `init()`

The version I’m happy with leans on two boring, well-worn Go mechanisms and lets them do something quietly clever together.

First, pkg/credentials defines a Backend interface and a registry. By default the registry holds a stub backend that politely returns “unsupported” for everything. The framework only ever talks to the registered backend, whatever that happens to be.

Second, the keychain implementation lives in its own package, pkg/credentials/keychain, still inside the same module, no separate release to manage. That package has an init() that registers its go-keyring-backed backend:

//nolint:gochecknoinits // registration via import is the whole point
func init() {
 credentials.RegisterBackend(Backend{})
}

And go-keyring, godbus, wincred, the whole IPC dependency chain, are only imported by that package.

Now the trick. To switch keychain support on, you import the package. You don’t have to use anything from it. A blank import is enough, because a blank import still runs the package’s init():

// cmd/gtb/keychain.go - the entire file.
package main

import _ "gitlab.com/phpboyscout/go-tool-base/pkg/credentials/keychain"

That single line is the on/off switch for the shipped gtb binary. The blank import means init() runs, the keychain backend registers itself, and credential operations start routing through the OS keychain. No flag, no tag, no config.

The part that makes it provable

Here’s why this beats the build tag, and it comes down to one guarantee in the Go toolchain: the linker only includes packages that are actually imported.

If cmd/gtb/keychain.go exists, the keychain package is in the import graph, so go-keyring, godbus and wincred are linked in. Delete that one file and rebuild, and the keychain package is no longer reachable from main. The linker performs dead-code elimination, and the entire go-keyring chain is gone. Not dormant. Not present-but-unused. Absent from the binary.

That’s the bit a regulated build needs. It isn’t a promise that the code won’t run. It’s a structural fact that the code isn’t there, and you can hand a security reviewer an SBOM that proves it. go-keyring won’t appear, because it genuinely isn’t linked.

For a downstream tool built on go-tool-base the story is the same, and just as cheap. Want keychain support? Add the one-line blank import to your own cmd package. Must ship keychain-free? Don’t. Your binary’s dependency graph follows your import graph, exactly as Go always promised it would. The default (no import) is the locked-down one, which is the right way round for a safety property.

Why I like this more than I expected to

Build tags hide a decision in the compiler invocation. This pattern puts the decision in the source, as an import, where it’s greppable, obvious in code review, and impossible to get subtly wrong. There’s a real file called keychain.go whose entire content is one import, and it reads as exactly what it is: a switch.

It’s also just honest Go. No reflection, no plugin loader, no clever runtime. A registry, an init(), and the linker doing the one job it’s always done. The cleverness, such as it is, is in the arrangement, not in any individual piece.

Stepping back

go-tool-base needed OS keychain support for the many, and a way to provably exclude it for the few. Build tags could express the toggle but hid it in the build invocation and rotted in the dark. A separate module solved the dependency question but was far too much machinery for one backend.

Putting the keychain backend in its own package, activated by a blank import _ that fires its init(), gets you both: a one-line, in-source, code-reviewable switch, and, because the linker only links what’s imported, a build with the import omitted that contains none of the keychain dependency chain. Provable absence, not promised disuse.

If you’re carrying an optional dependency that some of your users need gone rather than merely idle, this is the pattern. Let the import graph be the feature flag.

Where should a CLI keep your API keys?

Mon, 20 Apr 2026 00:00:00 +0000

Your CLI tool needs the user’s API key. It has to come from somewhere, and it has to survive between runs, so the obvious move is to ask once and write it into the config file. One tidy api_key: line. Job done.

It works beautifully on the first afternoon. And then, months later, it’s quietly become a liability nobody actually decided to create.

The config file that quietly becomes a liability

Your CLI tool needs the user’s API key. It has to come from somewhere, and it has to survive between invocations, so the obvious move is to ask once and write it into the tool’s config file. ~/.config/yourtool/config.yaml, a nice api_key: line, done.

It works on the first afternoon. It keeps working. And then, slowly, it becomes a problem nobody decided to create.

The config file gets committed to a dotfiles repo. It gets caught in a tar of someone’s home directory that lands in a backup bucket. It scrolls past in a screen share. It sits, world-readable, on a shared build box. None of these are exotic. They’re just a Tuesday. The plaintext key was fine right up until the file went somewhere the key shouldn’t, and config files go places.

I didn’t want go-tool-base handing every tool built on it that same slow-motion liability by default. So credential handling got rebuilt around a simple idea: the config file should usually hold a reference to the secret, not the secret itself.

Three modes, and which one you get

go-tool-base supports three ways to store a credential.

Environment-variable reference, the default. The config records the name of an environment variable, not its value:

anthropic:
 api:
 env: ANTHROPIC_API_KEY

The secret itself lives in your shell profile, your direnv setup, or your CI platform’s secret store, wherever you already keep that sort of thing. The config file now contains nothing sensitive at all. You can commit it, back it up, paste it into a bug report. The reference is inert on its own.

OS keychain, opt-in. The config holds a <service>/<account> reference and the actual secret goes into the operating system’s keychain: macOS Keychain, GNOME Keyring or KWallet via the Secret Service, Windows Credential Manager.

anthropic:
 api:
 keychain: mytool/anthropic.api

This one is opt-in by design, because the keychain backend carries dependencies that some deployments simply aren’t allowed to ship. (That opt-in mechanism turned out to be an interesting little problem all of its own, and it gets its own post in a couple of days.)

Literal value, legacy and grudging. The old behaviour. The secret sits in the config in plaintext:

anthropic:
 api:
 key: sk-ant-...

It still works, because breaking every existing tool’s config on an upgrade would be its own kind of vandalism. But it’s the last resort, it’s documented as the last resort, and the setup wizard puts a warning in front of you when you pick it.

The one place literal mode is not allowed

There’s a single hard “no” in all of this. If go-tool-base detects it’s running in CI (CI=true, which every major CI platform sets) the setup flow will refuse to write a literal credential, and exits non-zero.

The reasoning is that a plaintext secret written during a CI run is a plaintext secret written onto an ephemeral, often shared, frequently-logged machine, by an automated process that no human is watching. That’s the exact situation where the slow-motion liability becomes a fast one. CI environments inject secrets as environment variables already; there’s no good reason for a tool to be writing one to disk there, so go-tool-base simply won’t.

How it decides at runtime

A credential can be configured more than one way at once. You might have an env reference and an old literal key still lurking. So resolution follows a fixed precedence, highest to lowest:

The *.env reference. If that env var is set, use it.
Otherwise the *.keychain reference. If a keychain entry resolves, use it.
Otherwise the literal *.key / *.value, the legacy path.
Otherwise a well-known fallback env var (ANTHROPIC_API_KEY and friends), so a tool still picks up the ecosystem-standard variable with no config at all.

The useful property here is that adding a more secure mode transparently wins. Drop an env reference next to an old literal key and the next run uses the env var. You can migrate a credential to a better home without first removing it from its worse one, which makes the migration safe to do incrementally instead of as one nervous big-bang edit.

The tool tells on itself

A precedence rule is no use if nobody knows their config still has a plaintext key three layers down. So the built-in doctor command grew a check for exactly that. Run doctor, and if any literal credential is sitting in your config it reports a warning, names the offending keys (the key names, never the values) and points you at how to migrate.

It’s not an error. Literal mode is still legal. But the tool will quietly keep reminding you that you left the campsite messier than you could have, until you go and tidy it. (Old Scout habits die hard, and they’ve leaked all the way into the framework.)

The gist

A CLI tool that writes your API key into a plaintext config file isn’t doing anything wrong, exactly. It’s just handing you a liability that activates later, when the file travels somewhere the key shouldn’t. go-tool-base’s answer is three storage modes: an env-var reference by default, the OS keychain on request, and a plaintext literal only as a documented last resort that CI environments can’t use at all. Runtime resolution runs in a fixed precedence so a more secure mode always wins, which makes migrating a credential safe to do gradually. And doctor keeps an eye on the config so a stray plaintext secret doesn’t get to hide forever.

The secret should live in a secret store. The config file should just know its name.

I had the framework audited: every finding was the same shape

Fri, 17 Apr 2026 00:00:00 +0000

When a real security audit lands back in your inbox, the temptation is to read it as a shopping list of unrelated mistakes. Fix one, fix the next, tick them off, move on. I did exactly that the first time. The second time, I noticed something far more useful: the findings weren’t scattered at all. They clustered. Almost every one was the same sentence with the nouns swapped out.

Findings cluster, they don’t scatter

When you get a real security audit back, the instinct is to read it as a list of unrelated mistakes. Finding 1, unrelated to Finding 2, unrelated to Finding 3. Triage each, fix each, move on.

That’s not what the go-tool-base audits looked like once I stopped reading them as a list. The findings clustered. Strip away the specifics and almost every one was the same sentence with the nouns swapped: untrusted input reaches a powerful operation, and nothing checks it in between.

That reframe is worth more than any individual fix, because it turns “we patched some bugs” into “we know where to look next time”. A framework’s attack surface isn’t spread evenly. It’s concentrated at the boundaries: the handful of points where data from outside (a config file, a command-line flag, something typed into a TUI, an HTTP response) flows into machinery that can be made to misbehave. Audit the boundaries and you’ve audited most of the risk. Three examples make the pattern obvious.

Boundary one: a regex compiler

Somewhere in the tool, a user-supplied string gets compiled into a regular expression. A search pattern typed into the docs browser, a filter from a config file. Feeding user input to regexp.Compile feels harmless. It’s just pattern matching, after all.

It isn’t quite harmless. A regular expression is a tiny program, and some tiny programs are catastrophically slow. A pattern with the wrong kind of nested repetition can take exponential time to evaluate against a modestly hostile input. That’s the class of bug known as ReDoS. A user, or something feeding the user’s config, hands you a pathological pattern and your tool wedges, burning a whole core, on what looked for all the world like a search box.

The fix isn’t to ban user-supplied regexes. It’s to stop treating “compile this string” as free. go-tool-base routes any regex whose pattern came from outside the binary through a regexutil.CompileBounded helper. It caps the pattern length and puts a hard timeout on compilation. A pattern known at build time can still use plain regexp.MustCompile, because that isn’t a boundary, it’s a constant. The discipline only applies where the input genuinely crosses in.

Boundary two: a URL opener

The tool needs to open a URL in the user’s browser, a docs link or an OAuth flow. Under the hood that’s the OS handler: xdg-open, or open, or rundll32.

Now ask where the URL came from. If any part of it is influenced by config, by a server response, by user input, then “open this URL” has quietly become “ask the operating system to do something with an attacker-influenced string”. A file:// URL. A javascript: URL. Something with control characters smuggled into it. The browser-open was never the dangerous part. The unvalidated string was.

So go-tool-base funnels every URL-open through one package, pkg/browser, and that package is a gate. It enforces an allowlist of schemes (https, http, mailto, and nothing else), bounds the length, and rejects control characters before the OS ever sees the string. The rule that makes it stick is that nothing else is allowed to call the OS handler directly. One door, and the door has a lock. A scattered capability with no chokepoint can’t be secured; a capability that has a chokepoint can. (You’ll have spotted the “one door out” idea by now… it’s the same instinct as the single error handler, pointed at security instead of consistency.)

Boundary three: a log sink

This one’s the sneakiest, because it runs the wrong way round. The first two boundaries are about dangerous input coming in. This one is about sensitive data leaking out.

The tool handles credentials. It also logs, emits telemetry, and reports errors, and all three of those are exit boundaries: places where strings leave the process for somewhere more persistent and more public, like a log aggregator, an analytics backend, an error tracker. If a token ever ends up in a string that flows to one of those, you haven’t logged an event, you’ve published a secret.

The defence is pkg/redact. Any free-form string heading for an observability surface goes through it first, and it strips the usual suspects: credentials in URL userinfo, sensitive query parameters, Authorization headers, the well-known provider key prefixes (sk-, ghp_, AIza and friends), long opaque tokens. The places most likely to leak, command arguments and error messages in telemetry, get it applied automatically rather than relying on every caller to remember.

Same pattern as the other two. A boundary, and something standing on it checking what goes through.

The unglamorous part

None of these fixes is clever. There’s no exploit demo, no neat trick to show off. Bound a length. Check a scheme against an allowlist. Run a string through a redactor. The work was almost entirely in noticing the boundary existed, and then making sure everything routes through the one checked path instead of dotting raw calls all over the codebase.

That’s the actual lesson of a security audit, and it’s why the cluster reframe matters. The value wasn’t the dozen-or-so individual fixes. It was learning that the next risk will be at a boundary too, the next place untrusted input meets a powerful operation with nothing in between, and that the job is to find those points and put a single, mandatory, checked door on each.

To sum up

A security audit of a CLI framework reads like a list of unrelated bugs and isn’t one. go-tool-base’s findings nearly all reduced to the same shape: untrusted input reaching a powerful operation unchecked. A regex compiler that needed a length and time bound (regexutil.CompileBounded). A URL opener that needed a scheme allowlist and a single chokepoint (pkg/browser). Log and telemetry sinks that needed credentials redacted on the way out (pkg/redact).

The fixes were structural and dull, which is exactly right. Find your boundaries (config, flags, TUI input, network responses, log and telemetry sinks), give each one a single mandatory checked path, and you’ve spent your audit effort where the risk actually lives.

The test-mocking pattern that races

Thu, 16 Apr 2026 00:00:00 +0000

I’m going to tell you about a bug go-tool-base shipped, because it’s one of those bugs that’s so reasonable-looking you’ll find it in textbooks, conference talks, and an awful lot of otherwise excellent Go code. We had it too. It passed every test on my laptop, every single time, and then quietly fell over on CI while blaming an innocent bystander.

It’s the classic Go trick for mocking a dependency, and it races.

A pattern that looks completely reasonable

Here’s a thing you need to do constantly in Go tests: stop a function from really shelling out. It calls exec.LookPath to find a binary, or exec.Command to run one, and your test very much does not want it touching the real $PATH or spawning a real process.

The Go community has a well-worn answer. Hoist the function into a package-level variable, call that, and let tests reassign it:

// production code
var execLookPath = exec.LookPath

func findTool() (string, error) {
 return execLookPath("sometool")
}

// test
func TestFindTool(t *testing.T) {
 old := execLookPath
 defer func() { execLookPath = old }()
 execLookPath = func(string) (string, error) {
 return "/fake/path", nil
 }
 // ...assert...
}

It’s tidy. No interface to thread through, no constructor to change. You’ll find it in a great deal of Go code, including some very respectable Go code indeed. go-tool-base had it too.

And it works. It works on your machine, it works in code review, it works the first hundred times CI runs it. Which is precisely what makes it dangerous, because it’s wrong, and it’s just been biding its time.

Add one line and it detonates

Go’s t.Parallel() is more or less free performance. Mark your tests with it and the runner overlaps them instead of plodding through one at a time. On a package with a few hundred tests it’s a real, worthwhile speed-up, so naturally you reach for it.

Now picture two tests, both using the pattern above, both marked t.Parallel(). They run concurrently. Test A assigns its fake to execLookPath. Test B assigns its fake to execLookPath. Test A reads execLookPath expecting its own fake. Two goroutines, one variable, writes and reads with nothing synchronising them. That’s a textbook data race, and the textbook is right: the behaviour is undefined. Test A might see B’s fake. The deferred restore might land in the wrong order and leave the variable pointing at a fake after both tests have finished, poisoning a third one for good measure.

The truly nasty part is the intermittency. Whether the race actually bites depends on goroutine scheduling, which depends on machine load and core count. Your laptop running eight tests at once might never lose the coin-toss. A CI runner under load, scheduling differently, loses it and fails a test that has nothing obviously to do with the change in the commit. You re-run the pipeline, it passes, everyone shrugs and moves on. A test suite that fails one run in twenty trains your team to ignore it, and an ignored CI failure is worse than no CI at all.

I can tell you this one from direct, slightly embarrassed experience, because go-tool-base shipped exactly this bug and CI caught it the honest way: green on the laptop, red on the runner, with the failure cheerfully pointing at innocent bystander tests rather than the global that was actually the culprit. go test -race will name it for you if you crank the parallelism up high enough to lose the toss reliably… but you have to go looking, and you only go looking once it’s already ruined an afternoon.

The fix isn’t synchronisation, it’s structure

The instinct is to slap a mutex around the variable. Resist it. A mutex makes the race defined, but it doesn’t make the design any good. You’ve still got global mutable state, you’ve just queued the fight instead of cancelling it. And tests that serialise on a shared lock aren’t really parallel any more, so you’ve also handed back the speed-up you came for in the first place.

The real fix is to not have a shared variable at all. The dependency was always an input to the code; the package-level var was just a way of avoiding saying so out loud. So say it. Inject it.

A struct field:

type Finder struct {
 lookPath func(string) (string, error) // defaults to exec.LookPath
}

func (f *Finder) find() (string, error) {
 return f.lookPath("sometool")
}

Or a functional option, if you’d rather keep the zero value clean. Either way, each test constructs its own Finder with its own fake. There’s no shared variable, so there’s no race, and t.Parallel() is free again because the tests genuinely don’t touch each other.

go-tool-base wrote this straight into its standing rules: no package-level mocking hooks, full stop. Dependencies come in through struct fields, functional options, or config fields. (The same injection discipline that makes Props so testable, applied one rung further down.) And to stop everyone hand-rolling the same exec fakes, there’s a small internal package, internal/exectest, with ready-made LookPath and CommandContext doubles you construct per-test. The pattern is gone, and the door it came in through is shut.

The rule worth taking away

A package-level variable that tests reassign is shared mutable state. It reads as a harmless convenience because in a single-threaded test run it behaves like one. t.Parallel() is the thing that reveals it was never harmless, only unobserved.

The general lesson is older than Go: if a value is an input to your code, make it an input. Smuggling it in as a global is borrowing test-time convenience against a debt that comes due, with interest, the day someone wants their tests to run in parallel. Pay cash. Inject the dependency.

Worth remembering

Mocking via a reassignable package-level variable is a beloved Go shortcut and a latent data race. It survives because single-threaded test runs hide it; t.Parallel() exposes it as intermittent, bystander-blaming CI flake that’s miserable to trace. A mutex only makes the bad design defined. The fix is structural: inject the dependency as a struct field or functional option, so each test owns its own double and there’s no shared state to race over. go-tool-base banned the global-hook pattern outright and ships internal/exectest so nobody’s tempted back to it.

If a piece of code depends on something, let it say so in its signature. Your future self, staring at a CI failure that flatly refuses to reproduce, will thank you.

Testing code that calls an LLM: yes, you actually can

Wed, 08 Apr 2026 00:00:00 +0000

“You can’t test code that calls an AI.” I’ve heard it said with great confidence, and it’s half right, which is the most dangerous kind of right. You genuinely can’t assert on what a non-deterministic model says. But the model isn’t your code, and the bits sitting either side of it most certainly are.

“You can’t test AI code”

It’s a fair worry. Your command calls an LLM. The LLM returns something slightly different every run. A test that asserts response == "..." is broken before you’ve finished typing it. So the conclusion arrives quickly: the AI path can’t be tested, leave it uncovered.

Which is a shame, because the AI call is usually the riskiest line in the whole command.

The conclusion is also wrong. It mistakes “I can’t test the model” for “I can’t test my code”. The model is not your code. Your code is the two pieces sitting on either side of it.

Your code is a prompt and a handler

Strip the command down to what it actually does:

It builds a prompt. It assembles a system prompt, the user’s input, perhaps some context, and sends it.
The model does something. This is not your code.
It takes the response and does something with it. It parses it, branches on it, prints it, stores it.

Steps one and three are entirely yours, and entirely deterministic. The same inputs build the same prompt and handle the same response the same way, every single time. That’s testable. Step two is the only part that isn’t, and step two was never yours to test in the first place.

So the job is to pin step two to a known value, and then test one and three properly.

Test the prompt: snapshot it

Step one produces a prompt, and a prompt is just a string, which means you can pin it.

Both frameworks lean on snapshot testing here. go-tool-base uses a golden-file approach: the prompt your code generates is recorded to a file, and the test re-generates it and compares against that file. rust-tool-base does the same with insta, snapshotting the request body the client would send.

The reason this matters is that the prompt is load-bearing and quietly easy to break. You refactor how context gets assembled. Without noticing, you’ve changed the wording, or the ordering, or dropped a line the model was leaning on. Nothing fails to compile. The behaviour just drifts, silently.

A snapshot test catches exactly that. It fails, it shows you the diff between the old prompt and the new one, and it makes you stop and make a decision. Was this change intended? If yes, you accept the new snapshot and move on. If no, you’ve just caught a bug before it shipped. Either way the prompt never changes by accident, which for AI code is most of the battle.

Test the handler: mock the response

Step three needs a response to handle, and in a unit test you don’t get that response from the real model. You supply it.

go-tool-base ships generated mocks for the ChatClient interface. A test builds a mock client, tells it “when Ask is called, return this canned value”, and runs the command against it:

mockClient := mock_chat.NewMockChatClient(t)
mockClient.EXPECT().
 Ask(mock.Anything, mock.Anything, mock.AnythingOfType("*main.Analysis")).
 RunAndReturn(func(_ context.Context, _ string, target any) error {
 *(target.(*Analysis)) = Analysis{Severity: "critical"}
 return nil
 })

Because the interface is only four methods, that mock is trivial to set up and complete by construction. rust-tool-base takes the same idea one layer down: HTTP-bound tests use wiremock, which stands up a fake server returning a canned response body. The client makes a real HTTP request; it just goes to a fake endpoint the test controls.

Either way, step two is now fixed to a value you chose, which makes step three deterministic. And that unlocks the tests that actually matter: given a malformed response, does the command fail gracefully? Given a rate-limit error, an empty answer, a field missing? Those are the cases a live model almost never hands you on demand, and a mock hands you every time, on the first run.

This is, incidentally, the same discipline as the test-mocking work elsewhere in the framework: the dependency is injected, so the test gets to decide what it does.

What you deliberately don’t test

One honest boundary. None of this tests whether the model gives good answers. That question is real, but it’s a different activity (evaluations, run as their own suite) and not something to mix into the unit tests.

The unit suite’s job is your code: that it builds a sound prompt, and that it handles every shape of response correctly, including the ugly ones. Keep that well away from “is the model clever today”. A unit test that depends on the model being clever is a unit test that fails when the weather changes, and a flaky test just teaches people to ignore the whole suite.

What it comes down to

Code that calls an LLM is testable; the model is not, and those are different statements. Your code is a prompt builder and a response handler, both deterministic, with the model sat in between.

go-tool-base and rust-tool-base converge on the same approach. Snapshot the prompt, with golden files or insta, so a refactor can’t change what you send without a test noticing. Mock the response, with generated ChatClient mocks or a wiremock server, so tests run with no network and you can feed in the malformed and error cases a real model won’t reliably produce. Leave “are the answers any good” to a separate evaluation suite. Test the two halves you own, and the non-determinism in the middle stops being an excuse to leave the riskiest line uncovered.

The AI provider that isn't an API

Mon, 06 Apr 2026 00:00:00 +0000

go-tool-base’s chat package puts five AI providers behind one interface. Four of them are exactly what you’d guess: HTTP calls to OpenAI, Claude, Gemini, and anything OpenAI-compatible. The fifth one isn’t an API at all. It shells out to a binary.

That sounds like a slightly mad thing to want, right up until you’ve worked somewhere the network says no.

The fifth provider shells out

The chat package speaks to five providers through one ChatClient interface. Four of them are what you’d expect: HTTP requests to OpenAI, to Claude, to Gemini, to any OpenAI-compatible endpoint. The tool author picks one in config, and the rest of the code never knows the difference.

The fifth, ProviderClaudeLocal, is different in kind. It doesn’t make an HTTP request at all. It shells out. It runs the claude CLI binary as a child process, passes the prompt in, and reads the answer back from the binary’s output.

That sounds like an odd thing to want until you’ve been stuck in the environment it was built for.

Why you’d want that

Picture a corporate network with its egress locked right down. Outbound HTTPS to api.anthropic.com is blocked by policy. A tool built on go-tool-base that uses AI would simply fall over there. It tries to reach the API, there’s no route, and that’s the end of the feature.

But the developer at that machine has the claude CLI installed, and has run claude login. That binary is permitted. It’s an approved, managed tool, and it has its own sanctioned path out. The direct API call is blocked; the claude command is not.

ProviderClaudeLocal is what bridges those two facts. If your tool’s AI calls go through that already-blessed binary instead of straight at the API, they work, in an environment where the direct call cannot. That’s the whole reason the provider exists. It isn’t faster (a real API call has lower latency) and it isn’t more capable. It’s for the place where the API call simply isn’t an option, and “isn’t an option” is a surprisingly common place to find yourself inside a large organisation.

What it costs, honestly

It’s worth being straight about the trade, because ProviderClaudeLocal is the reduced-capability provider.

It doesn’t do tool calling. It doesn’t do parallel tools. It doesn’t stream. Those need a live, structured connection to the model’s API, and a subprocess that runs once and prints an answer is not that. What it does support is plain chat and structured output, the latter through the binary’s own --json-schema flag.

So the honest positioning, and the package’s documentation says exactly this, is: prefer the API providers when you can reach them, because they’re lower latency and feature-complete. Reach for ProviderClaudeLocal when API access is restricted. You accept the narrower capability set as the price of working at all. For a tool whose AI feature is “answer a question” or “return a structured analysis”, that price is often nothing you’d even notice. For one built on an agentic tool-calling loop, it’s a real limitation, and you’d know to expect it.

How it stays behind the same interface

Here’s the part that makes it pleasant rather than a special case to maintain. Despite being a subprocess and not an API, ProviderClaudeLocal is still a ChatClient. Your feature code calls Chat and Ask exactly the way it would for any other provider.

Everything that makes a subprocess provider awkward stays inside the provider. Spawning the binary, feeding it the prompt, parsing its output, capturing stderr and surfacing it when the binary exits non-zero, and threading multi-turn continuity through session identifiers passed back on the next call with --resume: all of that is the provider’s problem, and all of it sits behind the interface. The code in your tool that uses AI doesn’t know, and has no way to find out, that this particular provider is a child process rather than an HTTPS call.

That’s a unified interface genuinely earning its place. It’s easy to put a uniform face on four things that already work the same way underneath. The real test of the abstraction is whether something that works in a completely different way, a subprocess instead of a socket, can still slot in without the caller changing a line. Here it can. You swap one config value, and a tool that talked to an API now talks through a binary, and nothing downstream so much as blinks.

The bottom line

go-tool-base’s chat package puts five providers behind one ChatClient interface, and ProviderClaudeLocal is the one that isn’t an API. It runs the locally installed, pre-authenticated claude CLI as a subprocess.

It exists for the locked-down environment where outbound HTTPS to the AI API is blocked but the claude binary is allowed: there, AI features keep working where a direct call would fail. The trade is a narrower capability set (no tool calling, no streaming, plain chat and structured output only) so you prefer the API providers when you can reach them and fall back to this when you can’t. And because it’s still a ChatClient, all the subprocess machinery stays hidden, and your code uses it without knowing it’s there. That last part is the real test of an abstraction: a provider that works in an entirely different way still slots in unchanged.

AI conversations you can resume

Sat, 04 Apr 2026 00:00:00 +0000

An AI conversation is, fundamentally, its own history. The model’s next answer depends on everything said so far. And a CLI tool, by its very nature, forgets everything the moment it exits. Put those two facts together and you get the problem: run an AI command, exit, run it again, and you’re talking to someone who’s never met you.

A CLI forgets everything

A long-running service keeps its state in memory for as long as it runs. A CLI tool doesn’t get that luxury. It starts, does one thing, exits. The next invocation is a brand-new process with no memory of the last one.

For most commands that’s exactly right, and you wouldn’t want it any other way. But an AI conversation is a different kind of beast, because a conversation is its history. The model’s next answer depends on everything said so far. Run an AI command, exit, run it again, and you’ve started a fresh conversation with someone who’s never met you. For an interactive assistant, or any AI workflow that unfolds across several invocations, that’s plainly the wrong behaviour. The user expects to pick up where they left off.

Save and restore

The chat package handles this through a PersistentChatClient interface. Like streaming, it’s an optional capability discovered with a type assertion, sitting beside the four-method core rather than bloating it. A client that supports persistence also satisfies this interface:

if pc, ok := client.(chat.PersistentChatClient); ok {
 snapshot, err := pc.Save()
 // store the snapshot somewhere
}

A snapshot is a serialisable value that captures the conversation. You store it. Next run, you load it, Restore it onto a fresh client, re-register your tools, and call Chat again. “Where were we?” works, because the model is handed back the whole history.

A snapshot is opinionated about what it carries

The interesting part is what a snapshot does and doesn’t contain, because that’s a series of deliberate decisions.

It carries the messages, the system prompt, the model name, and tool metadata: the names, descriptions and parameter schemas of the tools that were registered.

It does not carry tool handlers. Handlers are code, not data; you can’t serialise a function meaningfully, so after a restore you re-register them with SetTools. The snapshot remembers that a tool called read_file existed and what its shape was; it doesn’t try to remember the Go function behind it.

And it does not carry API tokens. This is the one to dwell on. A snapshot is a file. A file gets synced, backed up, copied between machines, attached to a support ticket by a user trying to be helpful. A snapshot that carried the API key would be a credential leak the moment it left the laptop it was made on. So the snapshot never contains a token, at all. On restore, the client picks the credential up again the ordinary way, from the environment or the keychain. The conversation and the secret are kept in separate places on purpose, and only one of them is ever in the file.

Encrypted at rest, if you want it

The package ships a FileStore that writes snapshots as JSON files, with 0600 permissions in a 0700 directory, and it can encrypt them. Pass WithEncryption a 32-byte key and snapshots are written with AES-256-GCM.

That option exists because a conversation can hold sensitive content even when it holds no credential. The log a user pasted in for analysis, the source file they asked the model to review, the internal details tucked into their questions: none of that is an API key, and all of it might be something you’d rather not have sitting in plain JSON in a backup somewhere. Encryption at rest covers it.

The FileStore is also careful about the snapshot identifiers it’s handed. An ID has to be a canonical UUID, and the resolved file path is checked to lie inside the store directory, so a snapshot ID arriving from an untrusted source (a CLI flag, a request payload) can’t be bent into a path-traversal that reads or writes somewhere it shouldn’t. Persisting conversations adds a small filesystem surface, and the store treats it as exactly that.

The short version

A CLI tool forgets everything between invocations, which is correct for most commands and wrong for an AI conversation, because a conversation is its history.

go-tool-base’s chat package lets you persist one. PersistentChatClient saves a snapshot you can store and restore later, picking the conversation back up where it ended. The snapshot is deliberate about its contents: messages, system prompt and tool metadata yes; tool handlers no, because they’re code you re-register; API tokens never, because a snapshot is a file and a file travels. The built-in FileStore can encrypt snapshots at rest with AES-256-GCM and validates snapshot IDs against path traversal. Resumable conversations, without the conversation file turning into a place secrets leak from.

An AI agent that has to make the build pass

Thu, 02 Apr 2026 00:00:00 +0000

Most AI code generation works on a charming little principle I’ll call generate-and-hope. The model writes the code, the model stops at the closing brace, and whether the thing actually compiles is left as an exercise for you. For a snippet you paste into an editor, fine. For a whole generated command, that’s just outsourcing the disappointment.

go-tool-base does something I’m rather happier with: the AI has to make the build pass before it’s allowed to claim it’s done.

Generate and hope

The usual shape of AI code generation is this. You ask for code, the model produces it, and the model’s job ends at the closing brace. Whether it compiles, whether the tests pass, whether the imports even resolve, none of that has been checked. The model produced something that looks right. You find out whether it is right when you build it.

For a snippet you paste into an editor, that’s perfectly fine. The compiler tells you in a second. But go-tool-base’s generator, driven by gtb generate command --script or --prompt, produces a whole command: the implementation, its tests, the lot. “Generate and hope” at that scale means handing the user a project that may or may not build, and quietly making them the one who finds out which.

Drafting is only step one

So the generator doesn’t stop at drafting. Writing the first version of the implementation and its tests is step one of two. Step two is an autonomous repair agent.

Once the draft is on the filesystem, a separate agent takes over. It’s an LLM running in a loop, but a loop aimed at one narrow, checkable job: make this project build and pass its tests. It isn’t asked to be creative. It’s asked to get to green.

A fixed set of tools, and no shell

The agent is not handed a shell. It’s given a fixed, defined set of tools and nothing else. Three of them let it explore and edit the project: list_dir, read_file, write_file. Four of them let it verify the project:

go_build runs the build and captures the compiler errors.
go_test runs the tests and captures the failures.
go_get resolves a missing dependency.
golangci_lint runs the project’s linter.

That restriction is the design, not a limitation of it. The agent can’t delete arbitrary files, can’t reach the network, can’t run anything that isn’t on the list. It has exactly what it needs to make code compile and nothing it would need to do damage. Its file writes are confined to the project directory by an explicit path check, so even write_file can’t go wandering up into /etc. A coding agent you’d actually let near a filesystem is one whose abilities are an allowlist, not a denylist. (I keep coming back to that principle through this series… safety as a boundary you draw, not a behaviour you hope for.)

The loop

The repair loop is a ReAct loop, the same reason-act-observe shape as the tool-calling loop, only this time pointed at a goal:

The draft is on disk.
Verify: run go_build and go_test.
If verification failed, read the error logs, the compiler error or the failing test.
Reason about the cause: an undefined variable, a missing import, a wrong signature.
Act: call write_file to patch the code, or go_get to add the dependency.
Loop. Steps two to five repeat until the project is green, or the agent hits its step limit, which defaults to 15.

What makes this work is treating the error output as feedback rather than as a failure to log and walk away from. A compiler error is the single most useful sentence you can hand a model that’s trying to fix code. It says what’s wrong, and usually where. The loop feeds it straight back in, and the model fixes against it.

Verification changes what “done” means

Here’s the real shift, and the agent’s own documentation puts it well: the agent “doesn’t just say it fixed a bug; it uses a Test tool to verify the fix before reporting success.”

A generate-and-hope model reports success when it finishes writing. It has no idea whether the code works, and it isn’t really claiming otherwise. “Done” means “I produced text”. The repair agent reports success when go_build and go_test actually pass. “Done” means “the build is green”. Those are two completely different claims, and only the second is worth anything to the person who asked for the command.

That’s the line between an AI that’s a creative writer and an AI that’s a collaborator you can hand a task to. And when the agent can’t reach green, when it spends its whole step budget and the project is still broken, the generator fails safely: it leaves the best-attempt code in place, commented out so the project still compiles, and tells the user what to finish by hand. There’s also an --agentless flag for anyone who’d rather have a plain single-shot retry than the multi-step agent. The default, though, is the agent, because the default should be code that’s been checked.

Where this leaves us

Most AI code generation generates and hopes: the model writes code and the user discovers whether it works. For a whole generated command, that pushes a may-or-may-not-build project onto the user.

go-tool-base’s generator drafts the command and then hands it to an autonomous repair agent. The agent has a fixed set of tools (explore and edit the project, build it, test it, lint it, fetch dependencies) and no shell at all, with file writes confined to the project directory. It runs a ReAct loop, reading each error and patching against it, until the build is green or it exhausts its steps. The point is what “done” comes to mean: not “the model finished writing”, but “the build passes”. Only one of those is a claim worth trusting.

Stop regex-ing the LLM's prose

Tue, 31 Mar 2026 00:00:00 +0000

Ask an LLM a question and it hands you back prose. Lovely to read, miserable to program against. You wanted the one number buried in the middle of it, and now you’re writing a regular expression to fish a word out of three well-written paragraphs that phrase themselves slightly differently every single time you run them.

There’s a much better way, and it’s the difference between forever interpreting an LLM and actually building on one.

The problem with a paragraph

You ask an LLM to analyse a log file and tell you the severity of what it found and a suggested fix. It comes back with three well-written paragraphs. Somewhere in there is the word “critical”, and somewhere is the fix.

Your program now has to extract those two facts from prose, and prose has no contract. The next run, the model phrases it differently. It leads with a caveat. It says “severe” where last time it said “critical”. It puts the fix first. Anything that worked by finding “critical” in the text is now quietly wrong, and you didn’t change a line. Parsing free text for structured facts is a game you lose slowly.

What you actually wanted was never a paragraph. It was a value: a thing with a severity field and a fix field, that you can branch on and store and pass around like any other.

Ask for the struct, not the prose

go-tool-base’s chat package draws the line with two methods. Chat gives you text. Ask gives you a struct.

You define the Go type you want back:

type Analysis struct {
 Severity string `json:"severity"`
 Fix string `json:"fix"`
}

var result Analysis
err := client.Ask(ctx, "Analyse this log file: "+logText, &result)

The framework generates a JSON Schema from that struct, sends it to the model as the required response format, and unmarshals the reply straight into result. You never lay a finger on the prose. You get result.Severity and result.Fix, typed, ready to use. If you want the model’s answer to drive a switch statement, this is the method that lets it.

The struct is the schema is the contract

The detail that makes this hold up over time: you don’t write the schema. The struct is the schema.

The framework derives the JSON Schema from your type. In go-tool-base that’s GenerateSchema[T](); in rust-tool-base the schema comes from your Rust type through schemars. (Yes, there’s a Rust sibling now. I’ll introduce it properly in a few weeks, but it keeps gatecrashing these posts because the two frameworks deliberately share ideas.) Either way there’s one definition, your type, and the schema is just a projection of it.

That matters, because otherwise two things have to agree. There’s the schema you tell the model to obey, and there’s the type you unmarshal the answer into. Hand-write the schema and those two can drift: add a field to the struct, forget to add it to the schema, and the model is never told to produce it, so it silently never appears. Deriving the schema from the type collapses the two into one. They can’t disagree, because there’s only one of them.

Both frameworks, with one extra step in Rust

go-tool-base does this with Ask and a ResponseSchema set on the client config. rust-tool-base does it with chat_structured::<T>, where T is any type that’s both deserialisable and JsonSchema.

rust-tool-base adds one step worth calling out. Before it deserialises the model’s reply into your T, it validates the raw response against the schema with a JSON Schema validator. That splits the failure into two distinct, named cases: the response didn’t match the schema, or it matched the schema but still wouldn’t deserialise. A model that returns subtly wrong JSON fails loudly and specifically, with an error that tells you which of those happened, instead of quietly handing you a zero-valued struct that you end up debugging an hour later.

When you’d reach for it

The line is simple, and it’s about who reads the answer.

If a human reads the answer, prose is right. Chat, free text, let the model write well. A summary, an explanation, an interactive reply: leave all of those as prose.

If a program consumes the answer, you want a value. Classification, extraction, a code review scored out of a hundred with a list of issues, a yes-or-no with reasons: anything where the next thing that happens is your code branching on the result. There, Ask and chat_structured turn the LLM from something you have to interpret into something that returns a value, and a typed value is a thing you can actually build on.

To sum up

An LLM returns prose by default, and prose has no contract, so a program that picks structured facts out of it breaks the moment the model rephrases.

Structured output asks for the value instead. You define a struct, the framework derives a JSON Schema from it, the model is constrained to that shape, and you get a typed result. go-tool-base’s Ask and rust-tool-base’s chat_structured both work this way, with the schema derived from your type so the schema and the type can’t drift; rust-tool-base additionally validates the response against the schema before deserialising. Use it whenever the answer feeds code rather than a human. It’s one of the four methods that make up go-tool-base’s small chat interface, and it’s the one that makes an LLM safe to program against.

Telemetry that asks first

Mon, 30 Mar 2026 00:00:00 +0000

Usage telemetry is genuinely useful. Knowing which commands people actually run, where the errors cluster, whether anyone ever touched the feature you spent a fortnight on… that’s the stuff that makes you a better maintainer. Wanting it is completely legitimate.

The trouble is that the usual way of getting it, on by default and quietly hoovering up everything, is a small betrayal of the people who installed your tool to get a job done. I wasn’t willing to build that, so go-tool-base’s telemetry starts from a different question.

The data you want, and the line you shouldn’t cross

If you maintain a tool, you want to know how it’s actually used. Which commands matter and which are dead weight. Where the error rate spikes. Whether anyone touched the feature you spent that fortnight on. That information makes you a better maintainer, and, to say it again, wanting it is completely legitimate.

The trouble is the standard way of getting it. Telemetry on by default. An opt-out buried three levels down in a settings file nobody reads. And once it’s running, it quietly collects far more than it ever admitted to: the arguments people passed, the paths they were working in, an IP address for good measure.

Every one of those is a small betrayal of someone who installed your tool to get a job done, not to become a data point. And the cost when users notice isn’t a slap on the wrist. It’s trust, and trust in a developer tool does not grow back quickly. A tool that surprises you once with what it was quietly collecting is a tool you uninstall and warn your colleagues about.

So go-tool-base’s telemetry started from a different question. Not “how do we collect the most data” but “how do we collect useful data without ever putting the user in a position they didn’t choose”.

Rule one: it is off until you say otherwise

The foundation is the simplest possible rule, and it’s absolute. Telemetry is never enabled by default. A freshly installed tool built on go-tool-base sends nothing. Not a heartbeat, not a ping, nothing at all.

It only starts collecting when the user makes an explicit, visible choice to let it. Three honest doors: they run telemetry enable, they say yes to a clear prompt during init, or they set TELEMETRY_ENABLED themselves. All three are deliberate acts. None of them is a pre-ticked box or a default they have to discover and then undo.

This is opt-in, and the distinction from a well-hidden opt-out is the entire point. Opt-out telemetry treats consent as something to be assumed and grudgingly reversed. Opt-in treats it as something that has to be given. Only one of those is actually consent.

Rule two: no personally identifiable information, full stop

Consent to “some telemetry” is not consent to “any telemetry”, so the second rule constrains what can ever be collected, even from a user who’s opted in.

No personally identifiable information. The framework does not record command arguments (they routinely contain paths, hostnames, the occasional secret someone’s pasted in). It does not record file contents. It does not record IP addresses.

It does need some notion of “distinct installations” for the numbers to mean anything, so it derives a machine ID from a handful of system signals and runs it through SHA-256. What leaves the machine is a hash. It tells you “this is the same install as last week” and tells you precisely nothing about whose install it is, and the hash can’t be walked backwards into the signals it came from.

The events themselves are deliberately thin. Which command ran, roughly how long it took, whether it errored. The shape of usage, not a transcript of it.

Rule three: the author picks the destination

Even with consent given and PII excluded, there’s a third question: where does the data actually go? go-tool-base doesn’t answer that for you, because it can’t. A corporate internal tool, an open-source CLI and an air-gapped utility have completely different right answers.

So the backend is the tool author’s choice. The framework ships several (a noop backend, stdout, a file, plain HTTP, and OpenTelemetry over OTLP) and supports custom ones. The noop backend matters more than it looks: it lets a tool wire up the whole telemetry surface, commands and all, while sending data precisely nowhere. A perfectly reasonable, fully supported configuration.

Pluggable backends also mean the data never has to touch any infrastructure I run. It goes where the tool’s author decides, on their terms. The framework provides the plumbing and stays well out of the destination.

And a way back out

One last thing, because it’s the part that makes the opt-in real rather than decorative. A user who opted in can opt straight back out, and the package includes a GDPR-aligned deletion path, so “stop, and remove what you have” is an actual supported request rather than a polite fiction.

Consent you can’t withdraw isn’t consent. It’s a one-way door with a friendly sign on it. The deletion path is what keeps the front door an actual door.

The bottom line

Telemetry is genuinely useful to a maintainer and genuinely dangerous to the trust of the people running the tool, and the usual implementation (on by default, opt-out buried, collecting everything) spends that trust recklessly. go-tool-base’s telemetry holds three lines: never enabled without an explicit user action, never collecting personally identifiable information even once enabled, and always sending data to a destination the tool’s author chose, up to and including nowhere. A real deletion path makes the opt-in something you can take back.

You can have your usage numbers. You just have to ask for them, the way you would for anything else that wasn’t yours to begin with.

Letting the AI call your Go functions

Sun, 29 Mar 2026 00:00:00 +0000

An AI that can only produce text can describe your system. An AI that can call your Go functions can actually operate it. That gap, between describing and doing, is the difference between a chatbot and something genuinely useful, and crossing it comes down to one fiddly mechanism: tool-calling, and the loop that drives it.

Talking about the system versus operating it

Wire an AI provider into a CLI command and you get something that can talk. Ask it a question, get a paragraph back. Useful, up to a point.

But notice the ceiling. An AI that can only generate text can describe things. It can tell you what it would do. What it can’t do is look at the actual current state of your system, or take a real action, because it has no hands. It’s reasoning in a vacuum about a world it can’t reach out and touch.

The thing that gives it hands is tool-calling. You hand the AI a set of functions it’s allowed to call. Now, mid-conversation, it can decide it needs to read that file before it can answer, or run that query, or check that status, and actually go and do it, and then reason about the real result. The AI stops describing your system and starts operating it.

The loop is the hard part

Tool-calling has a shape, and the shape is a loop. The literature calls it ReAct: Reason, Act, Observe.

The AI reasons about the prompt and decides whether it needs a tool.
If it does, it acts, asking for a specific tool with specific arguments.
Your code runs the tool and feeds the result back. The AI observes that result.
Round again. Reason about the new information, maybe call another tool, maybe several. Keep going until the AI has what it needs and produces a final text answer with no more tool calls.

Conceptually simple. Tedious and error-prone to implement by hand every single time: parsing the model’s tool-call requests, dispatching to the right function, marshalling arguments in and results out, feeding observations back in the exact format the provider expects, knowing when to stop, and not looping forever if the model gets itself stuck.

That orchestration is pure plumbing, and it’s identical for every tool and every command. So you can probably guess what’s coming: go-tool-base’s chat package owns it. You don’t write the loop. You write the tools.

Defining a tool

A chat.Tool is four things: a name, a description, a parameter schema, and a handler. The description is what the AI reads to decide whether to use the tool, so it’s worth writing well. The schema describes the arguments, and you don’t hand-write it. You write a tagged Go struct and let it generate:

type ReadFileParams struct {
 Path string `json:"path" jsonschema_description:"Relative path to the file"`
}

The struct is the contract. The framework derives the JSON Schema the AI is given straight from those tags, so the schema and the Go type the handler receives can’t drift apart, because they share a single source. The handler is then just an ordinary Go function that takes those parameters and returns a result.

You register your tools with SetTools, call Chat, and that’s the whole of your involvement. The framework runs the ReAct loop and Chat returns the AI’s final text answer once the loop settles.

Two details that show it was built for real use

A couple of decisions in the loop tell you it’s meant for production, not a demo.

Tool errors don’t abort the conversation. When a handler returns an error, the framework doesn’t crash the loop. It hands the error back to the AI as a string, as just another observation. That’s deliberate, and it’s right. A real agent should be able to call a tool, watch it fail, and react: try different arguments, take a different route, or tell the user it couldn’t manage it. A loop that aborted on the first tool error would be far more brittle than the model driving it.

The loop is bounded. There’s a MaxSteps limit, default 20. An AI that gets confused could otherwise call tools forever, and a CLI command that never returns is a worse failure than a wrong answer. The cap guarantees the command terminates. The agent gets room to genuinely work a problem across many steps, but not infinite room to flail about in.

There’s also parallel tool execution: when the model asks for several tools in a single step (three independent file reads, say) the framework runs them concurrently rather than one after another, because there’s no reason to make the AI sit and wait out a sequence of things that don’t depend on each other.

Boiling it down

A text-only AI can describe your system; an AI that can call your functions can operate it. Bridging that gap means tool-calling, and tool-calling means the ReAct loop (reason, act, observe, repeat) whose orchestration is fiddly, identical every time, and not a problem worth solving twice.

go-tool-base’s chat package runs the loop for you. You define chat.Tool values (name, description, a tagged parameter struct that generates its own schema, a handler), call SetTools and Chat, and get the final answer. Tool errors go back to the AI as observations so it can recover, and a MaxSteps cap guarantees the command always terminates. You write Go functions. The framework turns them into things an agent can reach for.

Nobody reads the manual

Sun, 29 Mar 2026 00:00:00 +0000

Let me describe the actual lifecycle of a user meeting your CLI tool, because it’s a bit humbling. They run it. It doesn’t quite do what they expected. They run it again with --help. They get a wall of monospaced flag descriptions, skim it, don’t find the thing they wanted, and either give up or go and ask a human who already knows.

Your documentation might be magnificent. It doesn’t matter, because the user never reached it.

The manual loses on location, not quality

That’s the lifecycle, and notice exactly where it breaks. The documentation might be excellent. It might answer their precise question in full. It doesn’t matter, because it’s on a website, in another window, behind a search box, and the user is here, in the terminal, mid-task. The docs lost not on quality but on location. They simply weren’t where the work was.

go-tool-base’s answer starts with a decision about location: the documentation gets embedded into the binary itself. Your docs/ folder ships inside the tool, the same way its default config does. Wherever the tool is installed, the docs are right there alongside it, no network, no browser. That embedding is what makes everything else possible, and there are two things built on top of it.

A browser, in the terminal

The first is the docs command, and it’s not --help with extra steps. It launches a proper Terminal User Interface, built on Bubble Tea.

It has a sidebar, structured from the project’s own zensical.toml or mkdocs.yml, so the docs are a navigable tree rather than one flat scroll. Markdown renders with real formatting through Glamour (colour, tables, lists, headings) instead of collapsing into monospaced soup. There’s live search across every page, regex included.

Compared with man and --help, the difference isn’t a nicer coat of paint. man gives you linear scrolling and grep; this gives you a structured tree, rich rendering and real search. It’s the documentation experience a modern developer expects, except it followed the tool into the terminal instead of demanding the user leave it.

A documentation assistant that won’t make things up

The second thing built on the embedded docs is the one I find genuinely transformative: docs ask.

The user doesn’t navigate anything. They just ask:

mytool docs ask "how do I point this at a self-hosted server?"

and get a direct, specific answer. Under the hood, the framework collates the tool’s embedded markdown and hands it to the configured AI provider (Claude, OpenAI, Gemini, Claude Local, any OpenAI-compatible endpoint) as the context for the question.

Now, “an AI answers questions about my tool” should immediately make you nervous, and the correct thing to be nervous about is hallucination. An AI that confidently invents a flag that doesn’t exist, or describes behaviour the tool simply doesn’t have, is worse than no assistant at all, because the user trusts it.

This is where embedding the docs pays off a second time, and it’s why I keep stressing that the corpus is closed. The model is instructed to answer only from the tool’s actual documentation, and the context it’s handed is exactly that documentation and nothing else. It isn’t drawing on a vague memory of similar tools from its training data. It’s answering from this tool’s real, shipped, version-matched docs. The corpus is small, closed and authoritative, which is the combination that keeps the answers honest. “Zero hallucination by design” isn’t a slogan about the model. It’s a property of bounding what the model is allowed to look at, which is the same instinct I leaned on with the mcp command: the safety comes from the boundary you drew, not from trusting the AI to behave itself.

There’s a nice second-order effect, too. The answer is always about the version of the tool the user actually has, because the docs were embedded into that build. No mismatch between a website documenting the latest release and the slightly older binary sitting on the user’s machine.

The upshot

Documentation usually loses to --help not on quality but on location: it’s in a browser, and the user is in the terminal. go-tool-base embeds the docs into the binary and surfaces them two ways: a docs command that’s a real TUI browser with a sidebar, rich markdown and search, and docs ask, which answers natural-language questions using the embedded docs as context.

Because that context is the tool’s own closed, shipped documentation and the model is told to use nothing else, the assistant stays grounded, and it’s always describing the exact version the user is holding. The fix for unread documentation was never to write more of it. It was to put it where the work happens and let it answer back.

BDD where it earns its place, and nowhere else

Sat, 28 Mar 2026 00:00:00 +0000

I have a slightly complicated relationship with BDD. I’ve watched it turn a tangled test suite into something the whole team could read and reason about, and I’ve watched it turn a perfectly good unit test into a paragraph of ceremonial English that nobody benefits from. So when go-tool-base brought in Cucumber-style BDD, the interesting decision wasn’t adopting it. It was being ruthless about where not to.

Two tests that hurt for different reasons

Most of go-tool-base’s tests are ordinary table-driven Go tests, and they’re absolutely fine. A function, a slice of input/expected pairs, a loop. Nobody needs Gherkin to understand a parser test.

But two areas were genuinely painful, and they were painful in the same way: the test had become harder to understand than the thing it was testing.

The first was pkg/controls, the service-lifecycle package. It runs a small state machine (Unknown, Running, Stopping, Stopped) with signal handling, health monitoring, restart policies and graceful shutdown all woven through it. The integration tests for graceful shutdown had grown to over three hundred lines of imperative goroutine and channel coordination. They worked. But reviewing them was a slog, and a test you can’t review with confidence is a test you can’t trust when it fails. The behaviour being checked, “when a shutdown signal arrives mid-startup, the controller stops cleanly”, was a simple sentence buried under a heap of synchronisation scaffolding.

The second was the CLI itself. init, update, doctor are user workflows. “Given a config file with a custom value, when I run init, then the custom value survives the merge.” That’s already a Given/When/Then; it just happened to be written out as Go.

Godog, and the line I drew

Godog is the official Go implementation of Cucumber. You write .feature files in plain Gherkin and bind each step to a Go function. The shutdown scenario stops being three hundred lines of channels and becomes this:

Scenario: graceful shutdown completes within the deadline
 Given a controller with two registered services
 When a shutdown signal is received
 Then both services stop in registration order
 And the controller reports a clean shutdown

The goroutine choreography doesn’t vanish, of course. It moves into the step definitions, written once and reused. What changes is that the scenario is now readable by someone who’s never opened the file before, including someone from an ops team who’ll never write a line of Go but absolutely has opinions about how shutdown should behave.

Here’s the part I want to dwell on, because it’s the part most BDD adoptions get wrong. The first design decision written down for this work was: strategic, not universal. Use Godog only where BDD adds clarity. Keep table-driven Go tests as the baseline everywhere else.

That sounds obvious written down. It is not obvious in practice, because BDD has a gravitational pull. Once a team has feature files, there’s a powerful urge to express everything as feature files, for consistency. And that’s how you end up with Gherkin scenarios for a pure function (Given the number 2, When I double it, Then I get 4) which is pure ceremony. You’ve wrapped a one-line table test in a paragraph of English and a step-definition indirection, and made it actively worse.

The honest test for whether BDD belongs is this: is this test a narrative, or is it a matrix?

A matrix is the same logic with many input/output pairs. That’s a table-driven test, that’s most unit tests, and Gherkin actively harms them. A narrative is a sequence of steps where the ordering and the state between steps is the thing under test, and that’s where Gherkin pays for itself. Lifecycle transitions are narratives. A user running three commands in sequence is a narrative. Doubling a number is not.

go-tool-base drew that line and stuck to it. Feature files live in features/ at the project root, where a non-Go developer can find and read them. Step definitions live in test/e2e/, kept well away from the unit tests. And the unit tests stayed exactly what they were, because they were already the right tool.

Made to fit, not bolted on

A couple of smaller decisions kept the BDD layer from feeling like a foreign object.

It runs under go test. There’s no separate Cucumber runner to install or remember. A godog.TestSuite is invoked from an ordinary TestFeatures(t *testing.T), so the BDD scenarios run in the same go test ./... as everything else. CI didn’t need a new concept bolted onto it.

And the CLI end-to-end tests build the gtb binary once and reuse it across every scenario. Compiling a binary per scenario would make the suite slow enough that people would quietly start skipping it, and a test suite people skip is just decoration. Build once, test many.

Stepping back

go-tool-base brought in Godog for BDD, but the decision worth writing about is the restraint. BDD was applied to exactly two things: the service-lifecycle state machine, where a 300-line goroutine tangle became a four-line scenario anyone can review, and CLI workflows, which are Given/When/Then by their very nature. Everywhere else, table-driven Go tests remained the baseline, because wrapping a matrix test in Gherkin makes it worse, not better.

The useful rule: BDD fits a narrative, ordered steps with meaningful state in between, and fights a matrix. Adopt it as a scalpel for the narratives. Resist the pull to turn it into a religion.

An AI interface that fits on one screen

Fri, 27 Mar 2026 00:00:00 +0000

The moment you decide a CLI tool should talk to an LLM, there’s a strong gravitational pull towards reaching for LangChain, or one of its many relatives. It’s the obvious move. It’s also, for most CLI work, a bit like hiring a removals firm to carry a single box up the stairs.

Let me explain why go-tool-base went the other way, and what “the other way” actually looks like.

The instinct, and why it overshoots

When you add AI to a tool, the instinct is to reach for the big general-purpose framework. LangChain and its relatives are capable, and they exist for a real need: orchestrating complex multi-step AI applications, with retrieval pipelines, memory stores, chains of calls, whole fleets of agents.

Now look at what a CLI tool actually needs from an LLM. It needs to send a prompt and get text back. Sometimes it wants structured data back instead of prose. Sometimes it wants to let the model call a few of the tool’s own functions. That’s pretty much the whole list.

Pulling in a framework built to orchestrate retrieval and agent swarms in order to do that is a poor trade. You take on a large new vocabulary of concepts, a wide dependency surface, and a great deal of abstraction you’ll never touch, all to perform three or four operations. The framework isn’t wrong. It’s just answering a far bigger question than the one a CLI tool is asking.

What go-tool-base chose instead

go-tool-base didn’t reach for a framework. The decision is on the record in its own design notes: before a single line was written, LangChain Go, go-openai, Vercel’s AI SDK and around ten other options were evaluated, and not one of them matched what a CLI framework actually needs. So the chat package was built deliberately small.

How small? The entire core ChatClient interface is four methods:

type ChatClient interface {
 Add(prompt string) error
 Chat(ctx context.Context, prompt string) (string, error)
 Ask(question string, target any) error
 SetTools(tools []Tool) error
}

Add appends a message to the conversation. Chat sends a prompt and returns text. Ask sends a prompt and returns a typed Go struct, the model’s answer unmarshalled straight into a value you defined. SetTools hands the model a set of your own functions it’s allowed to call. That’s the whole surface. Downstream code that uses AI never holds anything larger than this, and never has to know which provider is behind it.

The package’s own documentation has a word for this: right-sized. Large enough to solve genuine provider-abstraction complexity, small enough that the full interface fits on a single screen.

“Thin” is not the same as “does little”

This is the part worth being precise about, because “four methods” can sound like “barely does anything”, and that’s the wrong read entirely.

Behind those four methods sits genuinely awkward work. Five providers (OpenAI, Claude, Gemini, a locally installed claude binary, and any OpenAI-compatible endpoint) each with a different wire API, all normalised behind the one interface. A tool-calling loop. Structured output via JSON Schema, made to behave consistently across providers that each express it differently. Error normalisation. Token chunking.

The point of a thin abstraction is not that there’s little underneath it. It’s that the interface stays small while the implementation quietly absorbs the complexity. Four methods on the surface; five provider integrations and a tool-calling loop below the waterline. The thinness is a property of what the caller sees, not of what the package does. A reach-for-LangChain decision gets that backwards: it exposes the caller to all the machinery, whether or not the caller will ever need it.

The core stays small even as features grow

There’s a neat detail in how chat keeps the interface from creeping. The package also supports streaming responses and conversation persistence, both of which are real features with real surface area. Neither of them is in the four-method core.

Instead they’re separate, optional interfaces. A streaming-capable client also satisfies StreamingChatClient; a persistable one also satisfies PersistentChatClient. Code that wants those capabilities does a type assertion to ask for them, and code that doesn’t simply never sees them. So the common path stays four methods forever. New capabilities arrive as opt-in interfaces alongside the core, not as new methods bolted onto it. The thing that fits on one screen keeps fitting on one screen.

Extensible without forking, testable without a network

Two more properties keep the package small without making it limiting.

It’s extensible. The provider list isn’t closed. A RegisterProvider call lets any package contribute a new provider, and chat.New will route to it. You add a backend without forking pkg/chat or sending a patch upstream.

And it’s testable. The package ships generated mocks. A downstream tool’s AI features can be tested against a mock ChatClient returning canned responses, with no network, no API key, and no flakiness. Because the interface is four methods, that mock is trivial to set up and complete by construction. A sprawling framework interface is a sprawling thing to fake; a four-method one is not. (I’ll come back to testing AI code properly in a later post, because it deserves a whole article of its own.)

The right size

When a CLI tool needs AI, the instinct is a large framework like LangChain. For orchestrating retrieval pipelines and agent swarms, that’s exactly the right tool. For sending a prompt, getting a struct back, and letting the model call a few functions, it’s enormous overkill.

go-tool-base’s chat package is the deliberate alternative, chosen only after LangChain Go and a dozen others were weighed up and rejected. Its core ChatClient interface is four methods. Underneath sit five normalised providers, a tool-calling loop, structured output and error handling, but the caller sees four methods and never learns which provider is active. Streaming and persistence are opt-in interfaces beside the core, not additions to it. It extends without forking and tests without a network. Right-sized: the complexity is real, but it lives under the interface rather than in it.

Half your users don't have eyes

Wed, 25 Mar 2026 00:00:00 +0000

Run a command in your favourite CLI tool and look at what comes back. Colour. Neatly aligned columns. A friendly little summary sentence. Lovely… if you happen to be a human with eyes.

But a good half of any tool’s users aren’t people at all. They’re scripts, CI pipelines, bits of automation. And that pretty output you’re so proud of is, to them, actively hostile.

Your tool has two audiences and only serves one

I made more or less this same point about AI assistants when I argued that your CLI is already an AI tool. The machines are users too. Here it isn’t an AI doing the calling, it’s a humble shell script, but the principle is identical.

Run a CLI command and look at what comes back. Colour. Aligned columns. A friendly summary sentence. It’s designed for a person reading a terminal, and for a person reading a terminal it’s great.

Now picture the other half of your users. A deploy script that needs to know which version is installed. A CI job that runs doctor and wants to fail the build on one specific check. A bit of automation gluing your tool to three others. None of them have eyes. They have parsers.

So what do they do with your beautiful human output? They butcher it. They grep for a keyword, awk out the third field, sed off a prefix. It works in the demo. Then someone rewords a status line, or adds a column, or the colour codes shift, and every script downstream breaks at once. Silently, too, because a broken grep returns nothing rather than an error. You changed a sentence and quietly took out somebody’s pipeline without ever knowing.

The human-readable output was never the contract. It just got used as one, because it was the only output there was.

Give the machines their own channel

The fix is not to make the human output more parseable. That’s a trap. You’d be constraining prose meant for people in order to satisfy programs, and end up serving neither of them well. The fix is to give programs their own output format, declared and stable, kept well away from the prose.

So every command built with go-tool-base gets a --output flag. Leave it alone and you get the friendly human rendering. Pass --output json and you get something a parser can actually rely on.

And not just some JSON. JSON with a fixed shape.

One envelope, every command

The temptation with JSON output is to let each command emit whatever structure happens to suit it. Don’t. A consumer scripting against five of your commands then has to learn five shapes, and “where’s the actual payload?” has a different answer every single time.

go-tool-base wraps every command’s JSON in one standard Response envelope:

{
 "status": "success",
 "command": "deploy",
 "data": {
 "environment": "production",
 "version": "1.4.0",
 "replicas": 3
 }
}

status says how it went. command says what produced it. data holds the command-specific payload, and only the payload. Every built-in command (version, doctor, update, init) emits exactly this shape. So does every command you write, because pkg/output hands you the envelope rather than letting you freelance:

format, _ := cmd.Flags().GetString("output")
w := output.NewWriter(os.Stdout, output.Format(format))

return w.Write(output.Response{
 Status: output.StatusSuccess,
 Command: "deploy",
 Data: result,
})

The consumer-side payoff is the whole point. A script can check .status without ever touching .data. It can pull .data.version and know the field is there because it’s typed, not scraped. It learns the envelope once, and every command in your tool, and every tool built on the framework, honours it. The contract is explicit, versioned, and the same everywhere, which is precisely what the abused human output never was.

The human output gets to relax

There’s a quiet second benefit, and it’s my favourite kind: the sort you get for free. Once programs have their own reliable channel, the human output is freed. It no longer has to stay accidentally parseable. You can reword a status line, add colour, restructure a table, make it genuinely nicer to read, and not break a single script, because no script is reading it any more. They’re all over on --output json, where the real contract lives.

Two audiences, two formats, each one actually suited to its reader. That’s the deal a CLI tool ought to be offering, and most of them don’t.

In short

A CLI tool that only emits human-readable output is only half-built, because half its users are programs that end up grep-ing prose and shattering the moment that prose changes. go-tool-base gives every command a --output json flag and one standard Response envelope (status, command, data) used identically by every built-in command and by anything you write through pkg/output. Machines get a stable, explicit, learn-it-once contract; humans get output that’s now free to be properly readable, because nothing fragile depends on its wording any more.

If your tool will ever be called by another program (and it will), give that program a front door. Don’t make it climb in through the window.

Lifecycle management for when your CLI grows up into a service

Tue, 24 Mar 2026 00:00:00 +0000

There’s a moment in the life of a lot of CLI tools where they stop being a CLI tool. Nobody quite decides it. It just happens. Someone needs the thing to also expose a little HTTP endpoint, or poll a queue, or run a scheduler, so it grows a serve command… and the honest command-line utility you wrote is suddenly a long-running service wearing a CLI as a hat.

And a service needs a whole pile of production plumbing that a one-shot command never did.

The command that stops being a command

go-tool-base is CLI-first. It is not CLI-only, and the reason is a pattern I’ve watched play out more times than I can count.

A tool starts its life as an honest command-line utility. It runs, it does its thing, it exits. Then someone needs it to expose a small HTTP endpoint. Or poll a queue. Or run a scheduler. So it grows a serve command, or a run command, and the moment it does, the thing that was a CLI tool is now a long-running service that happens to have a CLI bolted on the front.

And a long-running service needs a whole category of plumbing a one-shot command never did. It has to start things up in a sensible order. It has to shut them down gracefully when someone sends a SIGTERM, finishing in-flight work rather than dropping it on the floor. It has to tell an orchestrator whether it’s alive, and whether it’s ready. It has to do something sensible when one of its internal services quietly falls over at 3am.

Hand-rolled, that’s a few hundred lines of goroutine choreography, channel-wrangling and signal handling that every such tool reinvents, slightly differently and slightly wrong each time. It’s the first-afternoon problem all over again, just turning up later in the project’s life. So go-tool-base ships it: pkg/controls.

A controller and the things it controls

The model is small. A Controller manages any number of services, each of which satisfies a Controllable interface, which at heart is just a StartFunc and a StopFunc. An HTTP server, a background worker, a scheduler, anything with a “begin” and an “end”.

You register your services with the controller and it owns their collective lifecycle. They share a common set of channels (errors, OS signals, health, control messages) so the whole set can react together. A SIGTERM doesn’t get caught by one service off in a corner; it reaches the controller, and the controller takes everything down in order, each StopFunc handed a context with a deadline so that one sulking service can’t wedge the whole shutdown forever.

That ordering and timeout handling is the bit nobody enjoys writing and everybody needs. Centralising it means a tool that adds a second service later inherits correct coordinated shutdown for free, rather than discovering on its first production SIGTERM that it only half shuts down.

Probes, because something is usually watching

If the service ends up in Kubernetes (and a lot of them do) the orchestrator wants to ask two different questions, and they really are different questions.

Liveness: are you alive, or are you wedged and in need of a kill? Readiness: are you alive and able to take traffic right now? A service can quite easily be live but not ready… still warming a cache, still waiting on a dependency. Conflate the two and you get yourself killed during a slow startup, or sent traffic before you can actually serve it.

controls keeps them separate. You attach a WithLiveness probe and a WithReadiness probe to a service, each just a function returning a health report, and the controller exposes them. The tool answers Kubernetes honestly, in Kubernetes’ own terms, without you hand-wiring two more HTTP handlers.

Self-healing, but only if you ask

The last piece is what happens when a service fails. A worker’s StartFunc returns an error. Health checks start failing. In a hand-rolled setup this is where you either crash the whole process or write yourself a bespoke restart loop.

controls has a supervisor that can restart a failed service for you, and the important word in that sentence is can. It’s off by default. A service is only supervised if you hand it a RestartPolicy at registration:

controls.WithRestartPolicy(controls.RestartPolicy{
 MaxRestarts: 5,
 InitialBackoff: time.Second,
 MaxBackoff: 30 * time.Second,
 HealthFailureThreshold: 3,
})

With a policy in place, the controller restarts the service if its StartFunc errors out, or if it racks up more consecutive health-check failures than the threshold allows. Restarts back off exponentially, from InitialBackoff up to a MaxBackoff ceiling, so a service that’s failing because its database is down doesn’t sit there hammering that database flat with a tight restart loop. MaxRestarts caps the attempts, because a service that’s failed five times in a row is not going to be rescued by a sixth go, and at that point honest failure beats a thrashing pretence of health.

Opt-in matters here. Automatic restarts are exactly right for a resilient daemon and exactly wrong for a tool where a failure should stop the line and get a human’s attention. The framework doesn’t make that call for you. It gives you the supervisor and lets you point it at the services that genuinely want it.

The bottom line

A surprising number of CLI tools become long-running services the day they grow a serve command, and the day they do, they need coordinated startup, graceful ordered shutdown, real liveness and readiness probes, and a considered answer to a service falling over. That’s a few hundred lines of fiddly, easy-to-get-wrong plumbing.

pkg/controls provides it: a Controller over Controllable services with shared channels and deadline-bounded graceful shutdown, separate Kubernetes-style liveness and readiness probes, and an opt-in supervisor that restarts failed services with exponential backoff and a restart ceiling. Your tool can start as a command and grow into a daemon without that growth turning into a rewrite.

CLI-first, but not stuck there.

Middleware for CLI commands, not just web servers

Tue, 24 Mar 2026 00:00:00 +0000

Every CLI tool past a certain size grows a category of logic that doesn’t really belong to any one command, and yet has to happen for loads of them. Timing. An auth check. Panic recovery, so a crash becomes a clean error instead of a stack-trace all over someone’s terminal. A log line saying the command started and how it finished.

Web frameworks sorted this out years ago. CLIs, for some reason, mostly still copy-paste it around.

The logic that belongs to no single command

That category of logic doesn’t belong to any one command, yet needs to happen for many of them. Time how long the command took. Check the user is authenticated before a command that needs it. Recover from a panic so a crash becomes a clean error rather than a stack-trace vomited across the screen. Log that the command started and how it ended.

None of that is the command’s job. The deploy command’s job is to deploy. But timing and recovery and auth still have to happen around it, and around build, and around sync.

Put that logic inside each command’s RunE and you’ve copied the same six lines into thirty functions, which means thirty places to fix when the logging format changes and thirty chances to forget one of them. Cross-cutting concerns copied by hand don’t stay consistent. They drift, every time.

Web frameworks already solved this

This is not a new problem. It’s about the oldest problem in web frameworks, and they settled on an answer a long time ago: middleware. Gin has it, Echo has it, every HTTP stack you’ve ever touched has it. A middleware is a wrapper that sits around a handler, runs its cross-cutting logic, and calls through to the handler in the middle.

A CLI command is, structurally, just a handler too. So go-tool-base brings the same pattern to the Cobra command tree, with the same functional Chain shape:

type Middleware func(
 next func(cmd *cobra.Command, args []string) error,
) func(cmd *cobra.Command, args []string) error

A middleware receives the next handler in the chain and returns a new handler that wraps it. You compose a stack of them, and each command’s real RunE runs in the middle of the onion. Write the timing logic once, as one middleware, and every command in the chain is timed. Change the log format once and all thirty commands change with it, because there was only ever one copy. (The “write it once, in a place where everyone inherits it” drum again, which I will keep banging until the series runs out.)

“But Cobra already has PreRun”

It does, and this is the objection worth answering properly, because Cobra ships PersistentPreRun and PreRun hooks and they look, at a glance, like they cover this.

They don’t, and the reason is structural. A PreRun hook is a thing that happens before the command. That’s all it is. It can’t run anything after. It can’t wrap the command in a defer. It can’t catch a panic the command throws. It can’t measure how long the command took, because measuring a duration needs a start point and an end point, and the hook only owns the start.

A middleware wraps the entire execution. Because it’s a function that calls next() in its own body, it straddles the command:

func TimingMiddleware(next HandlerFunc) HandlerFunc {
 return func(cmd *cobra.Command, args []string) error {
 start := time.Now()
 err := next(cmd, args) // the command runs here
 log.Debug("command finished", "took", time.Since(start))
 return err
 }
}

Before, after, and around. A recovery middleware can put a defer recover() in place that a PreRun hook structurally cannot. An auth middleware can check a condition and return an error instead of calling next() at all, refusing to let the command run in the first place. PreRun can’t veto the command; it runs, and then the command runs regardless.

PreRun is a notification that the command is about to happen. Middleware is control over whether and how it happens. For genuine cross-cutting concerns you need the second thing, not the first.

To sum up

Timing, auth, recovery and logging are cross-cutting concerns: necessary for many commands, owned by none. Hand-copied into every RunE, they drift out of sync. Web frameworks fixed this with middleware years ago, and a CLI command is structurally just another handler.

go-tool-base brings the functional Chain middleware pattern to the Cobra command tree. A middleware wraps a command’s whole execution, so it acts before and after and can decide whether the command runs at all… strictly more than Cobra’s PreRun hooks, which only fire beforehand and can’t wrap, recover, time, or veto. Write the concern once, wrap the chain, and every command inherits it consistently.

A logging interface that doesn't leak its backend

Mon, 23 Mar 2026 00:00:00 +0000

The same tool, in two different lives, wants two completely different kinds of log.

On my laptop I want logs I can actually read: colour, alignment, friendly timestamps. The very same tool running as a daemon in a container wants none of that. It wants structured JSON, one object a line, ready for a log aggregator to swallow. And in a test I want the logger to shut up entirely. The interesting question is what it costs you to move between the three.

The same tool wants different logs

On a developer’s machine the tool is a CLI. You want logs that are pleasant to read in a terminal: colour, alignment, human-friendly timestamps. The charmbracelet logger does that beautifully.

Then the very same tool grows a serve command and gets deployed as a daemon in a container. Now coloured terminal output is worse than useless. The log aggregator wants structured JSON, one object per line, machine-parseable. slog does that.

And in tests you want neither. You want the logger to exist, satisfy the interface, and stay completely silent.

That’s three different logging backends, wanted by one tool across three different lives. So what does switching between them actually cost?

What it costs depends on what your packages imported

If your packages import a concrete logger, if pkg/config and pkg/setup and twenty others each have import "github.com/charmbracelet/log" and take a *log.Logger, then the backend is welded into the entire codebase. Switching to JSON for the container build means editing the import and the parameter type in every single one of those packages. The backend has leaked. A detail that should have been one decision has become a property of a hundred files.

go-tool-base doesn’t let it leak. Every package in the framework accepts a logger.Logger, an interface, and nothing else. No package anywhere imports a concrete logging library. A package states, in its types, “I need something I can log through”, and stops right there. It has no idea, and no way to find out, what’s actually on the other end.

// what every package depends on
type Logger interface {
 Debug(msg string, args ...any)
 Info(msg string, args ...any)
 Warn(msg string, args ...any)
 Error(msg string, args ...any)
 // ...
}

The backend gets chosen once, at the top, when the tool builds its Props. It travels down to every package as the interface, through the Props container. The packages underneath never see the concrete type, so the concrete type can change without a single one of them noticing. (There’s that “decide it once, in one place” theme again. I did warn you it runs through everything.)

Three backends, and the swap is one line

go-tool-base ships three implementations of that interface:

charmbracelet (logger.NewCharm(w, opts...)). Coloured, styled, for humans at a terminal. The CLI default.
slog JSON, a slog-backed backend emitting structured JSON, for daemons and containers feeding a log aggregator.
noop, which does precisely nothing, for tests that want a real Logger and total silence.

Switching the tool from a friendly CLI logger to container-ready JSON is a change to the one line in main() that constructs the logger. That’s the lot. pkg/config doesn’t change. pkg/setup doesn’t change. None of the twenty packages change, because none of them ever knew which backend they had. The decision was always one line; the interface is what kept it one line.

The noop backend deserves its own mention, because it’s the one people underrate. A test for a command shouldn’t be spraying log output all over the test run, but the command still needs a non-nil Logger to function. logger.NewNoop() gives you exactly that: interface satisfied, output binned, test quiet. And because it’s just another implementation of the same interface, no test needs any special logging machinery. It passes a different backend, exactly the way the container build does.

The general shape

There’s nothing exotic going on here. It’s “depend on interfaces, not implementations”, which every Go developer has had drilled into them at some point. The bit worth holding onto is where the rule actually pays out, and it’s at the seams between a stable core and a detail you know full well you’ll want to vary.

A logging backend is exactly such a detail. You will want it different in a terminal, in a container, and in a test. So the thing your code depends on has to be the interface, and the concrete backend has to be chosen at one well-known point and nowhere else. Get that boundary right and “we need JSON logs in production” is a one-line change. Get it wrong and it’s a refactor and a bad afternoon.

What it comes down to

One tool legitimately wants three different logging backends across its life: coloured output in a terminal, structured JSON in a container, silence in a test. The cost of moving between them is decided entirely by whether your packages imported a concrete logger or an interface.

go-tool-base’s packages depend only on logger.Logger, never a backend. Three implementations ship (charmbracelet, slog JSON, noop) and the backend is chosen once, in main(), then carried everywhere as the interface through Props. Switching is one line at the top, because the detail was never allowed to leak into the hundred files below it.

Errors that tell the user what to do next

Sun, 22 Mar 2026 00:00:00 +0000

Here’s an error message I’ve been on the receiving end of more times than I’d care to count:

error: failed to read config file

True. Also completely useless! I now know something is broken and I haven’t the faintest idea what to do about it. Which file? Why couldn’t it be read? Should I create it, run some init command, fix a permission, set an environment variable? The message states the problem and then abandons me at it, rather like a sat-nav cheerfully announcing “you have arrived” in the middle of a motorway.

A message is not a fix

The instinct, the moment you notice this, is to go and write a better message:

error: failed to read config file at ~/.config/mytool/config.yaml.
Run 'mytool init' to create one, or set MYTOOL_CONFIG to point at an existing file.

Better for the human, no question. But look at what you’ve just done to the error as a value. The recovery advice is now welded into the error string. Any code that wants to ask “is this the config-missing error?” is reduced to substring-matching English prose. Reword the advice and you break the check. So you’ve helped the user and quietly sabotaged the program at the same time, because you’ve made one poor little string do two completely incompatible jobs… being a stable identity for code, and being friendly guidance for people.

Why I changed error libraries

go-tool-base started out on github.com/go-errors/errors. It’s a perfectly fine library and it gave us stack traces. What it didn’t give us was any way to attach human guidance to an error without shoving it into the message string. So the codebase did exactly the daft thing I just described: multi-line suggestion text baked straight into errors.Errorf calls, user-facing content and programmatic identity all mashed into one value.

That’s the whole reason for the migration to github.com/cockroachdb/errors. Not novelty, and not because I fancied a weekend of find-and-replace. One specific capability: cockroachdb/errors lets you attach a hint to an error as a separate, structured field.

return errors.WithHint(
 errors.New("failed to read config file"),
 "Run 'mytool init' to create one, or set MYTOOL_CONFIG to point at an existing file.",
)

Now there are two things, cleanly apart. errors.New("failed to read config file") is the identity… stable, matchable, the program’s handle on the error. The hint is the guidance… for the human, and rewordable as much as you like without breaking a single check, because no check ever looks at it. errors.Is and errors.As work properly through every wrapper layer, so code matches on identity and never has to read prose.

The migration brought a few other things worth having. Stack traces print with a plain %+v instead of a type assertion. Errors can carry structured, machine-readable metadata. Multiple errors from concurrent work can be combined as a first-class value. But the hint is the one that actually changed the user’s day, because the hint is the recovery step, stored where it belongs.

One door out, and it knows where the help is

Separating the hint is only half of it. The other half is making sure those hints actually reach the user, every time, and that comes down to having a single way out.

Every go-tool-base command returns its errors the idiomatic Cobra way, through RunE. They all funnel into one Execute() wrapper at the root, which routes every error (runtime failure, flag parse error, pre-run failure) through one ErrorHandler. One door out. So error presentation gets decided in exactly one place, and no command can render an error differently from the command sat next to it.

And because there’s one handler, it can pull off something the individual commands never could. The framework knows your tool’s metadata, including its configured support channel, be it a Slack workspace or a Teams channel. So the error handler can finish a fatal error not just with the what and the recovery hint, but with where to go if the hint didn’t help:

error: failed to read config file
hint: Run 'mytool init' to create one, or set MYTOOL_CONFIG.
 Still stuck? Ask in #mytool-support on Slack.

The user is never left at a dead end. The error tells them what broke, the hint tells them the most likely fix, and if that’s still not enough the handler tells them which door to go and knock on. A failure becomes a signpost instead of a full stop.

The short version

An error that only reports what went wrong leaves the user stranded, and the obvious fix (writing the recovery advice into the message) quietly wrecks the error as a value, because now your code has to substring-match prose just to work out what it’s looking at.

go-tool-base moved from go-errors to cockroachdb/errors to get hints: a structured, separate field for human guidance that leaves the error’s identity clean for errors.Is and errors.As. Every command’s errors leave through one Execute() wrapper and one ErrorHandler, so presentation stays consistent, and because that handler knows the tool’s support channel it can point a stuck user at real help.

State the problem for the program. Give the fix to the human. And for pity’s sake, keep the two in different fields.

Many embedded filesystems, one merged view

Sat, 21 Mar 2026 00:00:00 +0000

Go’s embed package is one of those features that makes you slightly giddy the first time you use it. One //go:embed directive and your default config, your templates, your docs are all baked into the binary. The tool just works the moment it’s installed, with nothing external to lose or forget to ship.

And then you go and build something modular on top of it, and you discover the catch nobody warned you about.

`embed.FS` is an island

An embed.FS has a property that’s easy to miss until it bites: it’s local to the package that declared it. The //go:embed directive can only see files at or below its own source file. So in any project bigger than a toy, you don’t have an embedded filesystem. You have many. The root package embeds one. Each feature, each subcommand that ships its own templates or defaults, embeds another. They’re islands, one per package, and Go gives you no native way to make them behave as a whole.

For most files that’s perfectly fine. A feature’s templates can stay on the feature’s island; nothing else needs them.

It stops being fine the moment features need to contribute to something shared.

The shared-config problem

Here’s the case that forces the issue. A go-tool-base tool has a global config.yaml of defaults, embedded at the root. Now you add a feature, and that feature has its own configuration keys, with their own sensible defaults.

Where do those defaults go?

The naive answer is: edit the root config.yaml and add the feature’s section. And that’s a genuinely bad answer, because it inverts the dependency. The root config now has to know about every feature. Add a feature, edit the centre. Remove one, edit the centre again. The central file becomes a pinch point that every feature has to reach into, and a modular architecture where you can’t add a module without editing the core isn’t really modular at all… it just has more files.

What you actually want is for the feature to ship its own slice of default config, on its own island, and for the global config the tool reads to somehow already contain it. The feature contributes; the centre doesn’t budge.

`props.Assets`: merge the islands

That’s the job of props.Assets. (Yes, it lives on Props, the load-bearing container I keep going on about. Most of the good stuff does.) It’s a layer that implements the standard fs.FS interface, and into it you Register each embed.FS under a name:

// root main.go
Assets: props.NewAssets(props.AssetMap{"root": &assets}),

// a feature's command constructor
//go:embed assets/*
var assets embed.FS

func NewCmdFeature(p *props.Props) *cobra.Command {
 p.Assets.Register("feature", &assets)
 // ...
}

Now Props carries one Assets value that represents all the islands as a single filesystem. The root’s files and every registered feature’s files, addressable through one fs.FS. Each registration is named, so the islands stay individually identifiable, but they read as one.

That alone solves the addressing problem. The genuinely clever part is what happens for structured files.

Opening a file that exists in several places

When you Open a path through props.Assets and that path has a structured extension (.yaml, .yml, .json, .csv) it doesn’t simply return the first match it stumbles across. It does this:

Discovery. It finds every instance of that path, across every registered filesystem.
Parsing. It unmarshals each one.
Merging. It deep-merges the parsed data, using mergo.
Re-serialisation. It hands you back a single fs.File whose contents are the combined, merged result.

So picture the shared-config problem again, only solved this time. The root ships a config.yaml with the base defaults. Each feature ships a config.yaml on its own island carrying only its own keys. Nobody edits anybody else’s file. When the init command opens config.yaml through props.Assets, it doesn’t get the root’s copy. It gets the deep-merge of the root’s copy and every registered feature’s copy: one config.yaml that contains every default in the tool, assembled at runtime from contributions that never knew about each other.

A feature contributes its defaults simply by existing and registering. The centre never changes. That’s the modular property the naive approach couldn’t give you, and it generalises well beyond config… the same merge applies to a shared commands.csv, or any structured file features want to add rows or keys to.

There’s also a Mount method for attaching an arbitrary fs.FS at a virtual path, which is handy for surfacing something external (a temp directory, say) as part of the same tree. But the structured merge is the feature that really earns Assets its place.

Boiling it down

embed.FS is per-package by design, so a modular CLI ends up with many embedded filesystems, one island per feature. Most of the time that’s fine. It fails specifically when features need to contribute to a shared resource like the global config.yaml, because the naive fix forces every feature to reach in and edit a central file.

props.Assets merges all the registered islands into a single fs.FS, and for structured files it goes further: opening a .yaml, .json or .csv discovers every copy across every island, deep-merges them, and returns the combined whole. A feature drops its own defaults onto its own island, registers, and the merged config the tool reads already includes them. Contribution without coupling, which is rather the whole point of being modular in the first place.

Props: the container that does the heavy lifting

Sat, 21 Mar 2026 00:00:00 +0000

I name-dropped Props back in the introduction and then rather glossed over it, which was a bit unfair of me, because it’s the single most important design decision in the whole framework. So let’s give it the attention it actually deserves.

And the best place to start, oddly enough, is the name.

Start with the name

The container at the centre of go-tool-base is called Props, and the name is doing real work, so we’ll start there.

It is not short for “properties”, though it does hold a few. A prop is the heavy timber or steel beam that stops a structure quietly collapsing in on itself. And for anyone who follows the rugby: a prop is the position in the scrum, the broad-shouldered forward whose entire job is to provide structural support so everyone else can get on with the game.

That’s the design brief, in a single word. Props is not where the clever, flashy work happens. It scores no tries. It’s the unglamorous, load-bearing thing that holds the framework up so that your actual command logic gets to be the interesting part. Understand the name and you understand what the struct is for.

What it carries

Props is the single object passed to every command constructor in a go-tool-base tool. It holds the dependencies a command might need:

Tool, metadata about the CLI (name, summary, release source).
Logger, the logging abstraction.
Config, the loaded configuration container.
FS, a filesystem abstraction (afero), so a command never touches the real disk directly.
Assets, the embedded-resource manager.
Version, build information.
ErrorHandler, the centralised error reporter.

A command constructor’s signature is, accordingly, boring on purpose:

func NewCmdExample(p *props.Props) *cobra.Command { ... }

One parameter. Everything the command could possibly need is reachable through it. No globals, no init()-time wiring, no twelve-argument constructor that quietly grows a thirteenth argument next month.

Why a struct, and not `context.Context`

Here’s the design decision I actually want to defend, because it’s the one Go developers tend to raise an eyebrow at. Go already has a well-known way to carry things through a call tree: context.Context. So why not just put the logger and the config in the context and pass that around?

Because context.Context carries its values as interface{}, and that’s the wrong trade for dependencies.

Pull a dependency out of a context and you get this:

l := ctx.Value("logger").(logger.Logger) // a runtime type assertion

That one line has two separate ways to hurt you. The key is a bare string, so a typo compiles perfectly happily and then fails at runtime. The type assertion is unchecked, so if the wrong thing is sitting under that key, your tool panics in front of a user. Neither failure is visible to the compiler. Neither is visible to your IDE. You find out when it breaks, which is to say at the worst possible time.

Pull the same dependency out of Props and you get this:

p.Logger.Info("starting") // a field access

p.Logger is a typed field. If it doesn’t exist, or you’ve used it wrong, the code simply doesn’t compile. Your IDE autocompletes it. Refactor the Logger interface and every misuse lights up at build time. There’s no runtime type assertion, because there’s no interface{} to assert from in the first place.

context.Context is the right tool for what it was designed for: cancellation, deadlines, request-scoped signals that genuinely cross API boundaries. It’s the wrong tool for “here are my program’s services”, because it trades away the compiler’s help for a flexibility you really don’t want here. Dependencies should be declared, somewhere the compiler checks them. Props is that somewhere.

What you get back for it

That one decision pays out in three currencies.

Testability. A command is now a pure function of its Props. To test it, you build a Props with the doubles you want (an in-memory FS instead of the real disk, a no-op Logger, a config you’ve populated by hand) and call the constructor. No global state to reset between tests, no monkey-patching, no init() order to puzzle over. The dependency is an argument, so the test just passes a different one.

Consistency. Cross-cutting changes have exactly one place to happen. When the global --debug flag flips the log level, it does so on the Logger inside Props, and because every command reads its logger from the same Props, every command gets the new level. No command can drift, because none of them owns its own copy.

Extensibility. Adding a new framework-wide service is just adding a field to one struct. Every command can immediately reach it; none of them needed touching to make it reachable.

To sum up

Props is the dependency-injection container at the heart of go-tool-base: one struct, passed to every command, holding the logger, config, filesystem, assets, error handler and tool metadata. It’s a concrete struct rather than a context.Context payload entirely on purpose, because dependencies belong somewhere the compiler can check them, not behind a string key and a hopeful runtime type assertion. That single choice buys you testability, consistency and easy extension.

The name says it best, really. Props doesn’t score the tries. It’s the broad-shouldered thing in the scrum that stops the whole framework folding, so the rest of your code is free to go and play.

Design your whole CLI in one file

Fri, 20 Mar 2026 00:00:00 +0000

Here’s a question that sounds trivial and really isn’t: where, exactly, does a CLI tool’s structure live? Not the logic of each command… the structure. Which commands exist, what they’re called, which flags they take, what’s nested under what.

I’d never properly thought to ask it until go-tool-base forced me to, and the honest answer turned out to be a little bit embarrassing.

Where does a CLI’s structure actually live?

Picture a CLI tool with twenty commands, some nested under others. In a typical project, where does its structure live? The honest answer is “smeared across the codebase”. It’s in twenty cmd.go files. It’s in the AddCommand calls that stitch them together. It’s in the flag registrations. To understand the shape of the tool you have to read all of it and assemble the picture in your head, because the picture exists nowhere as a single thing you can point at.

That’s a strange state of affairs for the single most important design fact about a CLI. The command tree is the tool’s interface, it’s the thing users actually touch, and yet it hasn’t got a home.

The manifest gives it one

go-tool-base’s generator gives that structure a home: .gtb/manifest.yaml. The manifest is a single readable file describing the command tree. Every command, its name, its short description, its flags, its place in the hierarchy, whether it carries assets or an initialiser. The shape of the whole tool, in one place you can open and read top to bottom.

And the manifest isn’t documentation about the project. It’s the thing the project’s wiring is generated from. When you run regenerate project, the generator reads the manifest and rebuilds the boilerplate to match it: the command registration, the AddCommand wiring, the flag definitions. The manifest is the source of truth, and the Go wiring is its output.

Design-first, when you want it

This unlocks a way of working that the smeared-across-the-codebase approach simply can’t offer. You can design the interface first, in the manifest, and let the code follow.

Want to rename a command? Edit one line in the manifest, run regenerate, and the rename propagates through every wiring file that ever mentioned it. Want to move a subcommand under a different parent? Change its place in the manifest hierarchy and regenerate. Want to add a flag to three related commands? Add it in the manifest, in three obvious places, and regenerate, instead of going on a little hunting expedition for three flag-registration blocks scattered across the tree.

You’re editing the tool’s interface as a design, in the file whose entire job is to hold that design, and the generator does the mechanical donkey-work of making the code reflect it. The thing you change is the thing that describes the structure. The code is downstream.

If that shape sounds familiar, it should. It’s the same instinct behind spec-driven and test-driven development: write down what the thing should be before you assemble how it works, and keep that statement of intent as a first-class, living artefact rather than a comment that quietly rots in a corner. The manifest is a spec for your command tree, and regenerate is what keeps the implementation honest to it.

It doesn’t trap you

There’s an obvious worry about any generated-from-a-manifest system: am I now locked into editing the manifest? What if I just want to open a Go file and write some Go like a normal person?

You can. The generator is careful not to own everything. It owns the wiring (the registration and the structural boilerplate) and it leaves your command logic well alone. The RunE function where your command actually does its work is yours; the manifest hasn’t got an opinion about it. And the generator tracks the files it produces by content hash, so if you do hand-edit something it generated, regeneration notices and asks before overwriting rather than steamrolling you. That mechanism turned out interesting enough to get its own post.

So the manifest is an option, not a cage. Design-first via the manifest when that suits the change. Drop into Go directly when that suits it better. The two stay in sync because regeneration reconciles them, not because one of them has been forbidden.

Pulling it together

A CLI’s command tree is its most important design surface, and in most projects it has no single home… it gets reconstructed in your head from twenty scattered files every time you need to reason about it. go-tool-base gives it one: .gtb/manifest.yaml, a readable description of the whole tree that the generator rebuilds the wiring code from. Edit the manifest, run regenerate, and the boilerplate follows.

It makes CLI structure something you design in one place, in the spirit of spec-driven development, while still leaving you free to write Go directly when that’s the better tool for the job. The manifest is the spec for your interface. The generator just keeps the code faithful to it.

Scaffolding that respects your edits

Fri, 20 Mar 2026 00:00:00 +0000

When I introduced go-tool-base I made a passing promise to come back to “the generator that won’t clobber your edits”. This is me keeping it, partly because it’s the feature I’m quietly most proud of, and partly because it took the most head-scratching of anything to get right.

The problem it solves is one that every code generator runs into eventually, usually the hard way and usually at the worst possible moment.

The generator’s awkward second act

A project generator has an easy first act. gtb generate skeleton, and you’ve got a complete, wired, idiomatic Go CLI project. Everyone’s happy, me included.

The second act is the hard one. The framework moves on. A convention changes, a new built-in capability appears, the recommended CI shape shifts. Your project, scaffolded three months ago, is now subtly out of date, and you’d quite like the generator to drag it back up to spec.

Except by now it isn’t a fresh scaffold. It’s your project. You tuned the CI workflow. You rewrote the justfile. You added a stanza to the Dockerfile that took an afternoon and a fair bit of swearing to get right. The generated files and your edited files are one and the same files.

A naive generator handles this with breathtaking confidence: it regenerates everything from the template and overwrites the lot. Run it once, lose your afternoon. You learn that lesson exactly once and then never run regeneration again, which means the upkeep feature you were sold is dead on arrival. A scaffold you can’t safely re-run is just a one-shot cp with extra steps.

What the generator needs to know

The thing standing between “safe to overwrite” and “absolutely do not” is a single fact: has this file changed since the generator last wrote it?

If it hasn’t, the file is still pristine boilerplate and the generator owns it. Overwrite away. If it has, a human has been in there, and the generator must not touch it without asking first.

The generator can’t just eyeball that, of course. It needs a record. So every time gtb generate writes a file, it computes a SHA-256 of the content and stores it in the project’s manifest, .gtb/manifest.yaml, as a Hashes map of relative path to hash. The manifest is the generator’s memory of the exact bytes it last produced.

Regeneration becomes a three-way decision

With that record in hand, regeneration stops being “overwrite everything” and becomes a per-file decision with three branches.

The file doesn’t exist. Easy. Write it, store its hash.

The file exists and its current hash matches the manifest. It’s byte-for-byte what the generator last wrote, so nobody has touched it. The generator owns it outright, regenerates from the template and updates the stored hash. No prompt, no fuss. This is the common case, and it’s silent precisely because it’s safe.

The file exists and its hash does not match. Someone has been in there since generation. The generator stops and asks. It will not silently overwrite your hard-won afternoon. You decide: take the new version, or keep yours.

The detail I’m genuinely fond of is what happens when you decline. Declining is non-fatal. Generation carries on with the rest of the files, and the manifest keeps the file’s stored hash rather than dropping it. That matters more than it looks, because it means the file stays tracked. Next time you regenerate, the generator can still tell that file has been modified, and still asks. Skipping a file once doesn’t quietly evict it from the generator’s awareness forever. It stays a known, watched, customised file across every future run.

When you want it to stop asking

Per-file prompting is the right default, but for files you’ve permanently taken ownership of, being asked on every single regeneration is just noise. If you’ve rewritten the CI workflows wholesale and you are never, ever going back to the generated version, you don’t want a prompt. You want the generator to leave them well alone and not bring it up again.

That’s what .gtb/ignore is for. It sits next to the manifest and takes gitignore-style patterns:

# I own the CI workflows now
.github/workflows/**

# ...except the release workflow, keep that managed
!.github/workflows/release.yml

# and my build config
justfile
Dockerfile

Anything matching is skipped during regeneration with no prompt at all. Patterns evaluate top to bottom and later ones win, so the negation (!) behaves the way you’d expect from .gitignore: exclude a whole directory, then claw one file back.

It’s a deliberate escalation ladder. Unmodified files are handled silently. Modified files get a prompt. Files you’ve formally claimed get total silence. Each rung asks for less of your attention than the last, and you choose how far up to climb, file by file.

Stepping back

A generator earns its keep twice: once when it scaffolds your project, and then continuously, every time it drags that project back up to the framework’s current shape. The second job is worth nothing if regeneration flattens your customisations, because you’ll simply stop running it, and who could blame you.

go-tool-base’s generator gets around that by remembering. It hashes every file it writes into .gtb/manifest.yaml, and on regeneration it re-hashes before overwriting: unchanged files it owns and updates silently, changed files it stops and asks about, and .gtb/ignore lets you mark files as permanently yours. Skipped files stay tracked, so the generator never loses sight of what you’ve made your own.

The point of a scaffold isn’t the first five minutes. It’s that you can still run it in month three without holding your breath.

Your CLI is already an AI tool

Thu, 19 Mar 2026 00:00:00 +0000

“Make it work with AI” has become one of those requests that lands on a developer’s desk with a thud and not much further detail attached. My instinct, the first time, was to brace for a big lump of integration work… a bespoke adapter for this assistant, another for that one, a treadmill of little wrappers stretching off into the distance.

Turns out I’d already done most of the work. So have you, if your CLI tool is any good. Let me explain what I mean.

You already described your capabilities

Stop and think for a second about what a well-built CLI tool actually is. It’s a set of named operations, each with a human-readable description, each taking a set of typed, named, documented parameters. You wrote all of that already, because a CLI without it is unusable by people.

Now look at what an AI assistant needs in order to call a tool. A set of named operations. A description of each, so it knows when to reach for them. A typed parameter schema for each, so it knows how to call them.

It’s the same list! A good CLI is already, structurally, a description of a set of capabilities. The information an AI agent needs isn’t extra work you have to go and do. It’s work you finished the moment your --help output was any good.

The only thing missing is a translator. Something that takes “this is a CLI” and presents it as “this is a set of tools an AI can call”.

MCP is that translator, and it’s a standard

The temptation, when you want your tool to be AI-usable, is to sit down and write an integration. A little adapter for Claude Desktop. Another for Cursor. Another for whatever turns up next month. Each one a bespoke wrapper, each one a thing to maintain, and the list never stops growing because new assistants keep appearing. That’s the treadmill I was bracing for.

The Model Context Protocol exists to kill that list. MCP is an open standard for how an AI model discovers and calls local tools. Implement it once and your tool works with every assistant that speaks it. Write once, not once-per-client.

So go-tool-base implements it once, in the framework, for everyone. (That’s rather the theme of this whole series, if you hadn’t spotted it yet… do the annoying thing once, properly, in a place where every tool inherits it.)

The `mcp` command, and the mapping it does for free

Every tool built on go-tool-base inherits a built-in mcp command. Run it:

mytool mcp

and the tool starts a JSON-RPC server over standard I/O, speaking MCP. That’s the whole user-facing surface. One command.

Behind it, the framework walks your Cobra command tree and maps it straight onto MCP tool definitions:

Each command becomes a tool.
Each command’s short description becomes the tool’s description, the text the AI reads to decide whether this is the tool it wants.
Each command’s flags and arguments become the tool’s JSON Schema parameters.

There’s no second schema to write and then keep in sync (and we all know how well “keep these two things aligned by hand” tends to go). The command tree is the schema. Add a new command to your CLI and it’s a new tool for the agent, automatically, with the description and flags you already gave it. Nobody has to remember to update an MCP manifest, because there’s no separate MCP manifest to forget about.

Configuring an assistant to use it

On the assistant’s side it’s just as undramatic. You tell your AI client (Claude Desktop, Cursor, anything MCP-aware) to launch mytool mcp. From then on the assistant:

Starts your tool in MCP mode when it boots.
Discovers every command as a callable tool.
Calls the right one, with the right parameters, when a user’s request needs it.

Your CLI tool has quietly become something the AI can pick up and use, mid-conversation, on its own initiative.

The safety property worth noticing

Now, “let an AI run things on my machine” is rightly a sentence that makes people nervous. It makes me nervous, and I built the thing. So it’s worth noticing the constraint sitting quietly in this design.

The AI can only call what you defined. The tools it sees are exactly the commands in your tree, and the parameters it can pass are exactly the flags and arguments you declared, validated against the JSON Schema generated from them.

It can’t invent a command. It can’t pass a parameter you never defined. The boundary of what the agent can do is the boundary of what your CLI does, and you drew that boundary already, back when you built the tool. Exposing the CLI over MCP doesn’t widen the surface one inch. It just makes the existing surface reachable. The AI isn’t running things. It’s running your commands, the ones you wrote, tested and shipped, and nothing else.

The gist

A CLI tool, built properly, is already a structured description of a set of capabilities: named operations, descriptions, typed parameters. Which is also exactly what an AI agent needs in order to call a tool. The gap between the two is only a translator, and writing a bespoke one per assistant is a treadmill you don’t need to step onto.

go-tool-base puts the translator in the framework. Every tool gets an mcp command that serves the command tree over the Model Context Protocol… commands become tools, descriptions become descriptions, flags become JSON Schema parameters, with no second schema to maintain. Point any MCP-aware assistant at it and your CLI is an agent-callable tool, bounded to exactly the commands you shipped.

You did the hard part when you built a good CLI. MCP just opens the door you’d already framed.

go-tool-base: I got tired of reinventing the wheel

Wed, 18 Mar 2026 00:00:00 +0000

If you’ve written more than two or three command-line tools in Go, you’ll recognise the shape of the first afternoon. I certainly do! You reach for Cobra for the command tree, Viper for config, and then you start the part nobody ever puts in the README… the plumbing.

Where does config live? A file, an env var, an embedded default? In what order do they override each other? How does the tool tell the user there’s a newer version, and how does it actually update itself? What does logging look like, and is it the same logging the next tool will use? And how do you wire all of that into each command without every command reaching into a pile of globals?

None of it is hard. That’s the problem! It’s not hard, it’s just there, every single time, and every single time I’d find myself reinventing it slightly differently to the last time. Different override precedence here. A subtly different update flow there. Logging that didn’t quite match the tool I’d written three months earlier. Each new tool was a fresh re-litigation of decisions I’d already made and then promptly forgotten.

Now, I’ve banged on about the Boy Scout rule for years (leave the codebase better than you found it), but it has an uncomfortable corollary. If you keep turning up to the same campsite and finding it in the same mess, at some point the honest thing to do is to stop tidying it and go and build a better campsite.

First, just packages

So I started pulling the recurring pieces out into their own packages. Nothing grand. A config package that did the hierarchical merge the way I always ended up doing it anyway. A version package that knew how to compare semver and spot a development build. A setup package that handled first-run bootstrap and self-updating from a release. They lived as separate repos, and if you go digging through my GitHub history you can still find the scruffy ancestors of them scattered about.

Separate packages was the right first move. It forced each piece to stand on its own and earn its keep on a real project before I trusted it on the next one. A package that’s only ever been used in the repo it was born in hasn’t really been tested… it’s just been agreed with.

But separate packages come with a tax. Each one has its own release cadence, its own changelog, its own CI. Worse, they have to agree with each other at the seams, and when they’re versioned independently those seams drift. I’d bump the config package, and the setup package that depended on it would quietly need a matching bump, and the tool that used both would need telling about both. I’d traded “reinvent the wheel” for “keep a dozen wheels in sync”, and I’m really not convinced that’s a better deal.

Then, one library

Once the packages had been used enough (used in anger, on real tools, by people who weren’t me) the shape of them stopped moving. The interfaces settled. The arguments about precedence and defaults were over, because the answers had survived contact with reality.

That’s the point where separate packages stop being a virtue and start being friction. So I forged them into one and called it go-tool-base. One module, one version number, one changelog, and one set of seams that are now internal and can’t drift, because they ship together.

The heart of it is a dependency-injection container, a Props struct, that holds the things every command needs: the logger, the config, the embedded assets, the filesystem handle, the error handler, the tool’s own metadata. Commands are handed Props explicitly rather than reaching for globals, which means a command is just a function of its inputs and is therefore trivially testable. That one decision has quietly paid for itself on every tool I’ve built since.

Around that container sits all the stuff I was so tired of rewriting: hierarchical config, structured logging, version checking, self-update from GitHub or GitLab releases, an interactive TUI documentation browser, AI integration, service lifecycle management. A new tool inherits the lot and gets to spend its first afternoon on the thing that’s actually novel… its own logic.

Finally, a generator

A library still leaves you staring at a blank main.go. You still have to know the conventions, wire the container, lay out the directories, register the commands. All knowable, but all boilerplate. And boilerplate is exactly the enemy I set out to kill in the first place.

So go-tool-base ships a generator. gtb generate skeleton scaffolds a complete, working, idiomatic project: directory layout, the wired Props container, the command tree, CI, the whole lot. gtb generate command adds a new command and registers it for you. The generator also handles upkeep: when the framework’s conventions move, it can regenerate the scaffolding of an existing project without trampling all over the code you’ve written on top. (That last bit turned out to be a properly interesting problem in its own right, and a future post.)

The goal is blunt. Creating a CLI tool should be about the tool, not the scaffolding. The first afternoon should be spent on the part that’s actually worth writing.

One thing I was careful about

There’s a nasty failure mode with “batteries-included” frameworks: the day you outgrow them, they hold you hostage. You either stay inside the framework’s worldview forever, or you face a rewrite. I’ve been burned by that before and I had no intention of inflicting it on anyone else.

So go-tool-base generates idiomatic, standard-library-compliant Go. There’s no magic runtime you can’t see, no clever code you couldn’t have written by hand. If you ever outgrow the framework the generated code stands on its own and you walk away with a perfectly normal Go project. A framework should be a starting point you’re glad you took, not a room you can’t get out of.

Where this leaves me

go-tool-base exists because I was spending the first afternoon of every Go CLI tool rebuilding the same plumbing, and rebuilding it slightly wrong relative to last time. It started life as separate packages so each piece could earn its place on real projects; once they’d stopped moving I forged them into a single library so the seams couldn’t drift; and then I wrapped a generator around it so a new tool starts as a working project rather than a blank file.

It’s a framework for the unglamorous 80% (config, versioning, updates, logging, lifecycle) so you can spend your time on the 20% that’s actually yours.

Over the coming posts I’ll dig into the individual pieces… the generator that won’t clobber your edits, the credential handling, the self-update integrity checks, and a few Go techniques I’m rather pleased with along the way. Stay tuned!

Migrating away from Mediawiki and how to export its data

Wed, 19 Aug 2020 00:00:00 +0000

I like Mediawiki, it is a simple tool capable of doing a lot and can be very flexible and easy to customise. However its not always the right solution! I had a situation where we needed to migrate away from using it for a combination of security and usability reasons. So I thought it would be good to document it.

After reviewing a few things it was decided to move things over to the companies already existing O365 SharePoint as a new site. This sounded laborious as first, but actually turned out to be pretty straight forward.

We start with getting data out of Mediawiki, thankfully we only wanted the most recent revision and not the full history of a page. We use PostgreSQL as a back-end so it was reasonably straight forward to figure out how to extract the data in a sensible query.

SELECT 
 page_id as id, 
 page.page_title as title, 
 pagecontent.old_text as content, 
 page_touched as edited
FROM mediawiki.page
LEFT JOIN mediawiki.slots ON page.page_latest = slots.slot_revision_id
LEFT JOIN mediawiki.content ON content.content_id = slots.slot_content_id
LEFT JOIN mediawiki.pagecontent ON pagecontent.old_id = CAST(OVERLAY(content.content_address placing '' from 1 for 3) as integer)
ORDER BY page_touched DESC;

It tool a little sleuthing to realize that the slots table was the pivotal in extracting the latest page version. With the right join and a little mangling of the content_address field from the contents table to remove the “tt:” from the value and convert to an integer we now have a result set of all the page names and the latest revision of that page. I also added in the date the page was last updated to allow me to see when it was last edited as it was a live system migration and helped me to ensure content remained sync while both were still in play.

Once I had the query it was a simple job of writing an “Exporter” using Go Lang to extract the data and write it to files, I’ll chuck a snippet of code at the bottom of the post for you.

Mediawiki uses wikitext as a format so I needed to convert it to something more widely understood. Having used Pandoc in the past successfully I plumped for this as I knew it would handle a lot of options and was simple to use to convert to the markdown_mmd format

I Installed it via the ubuntu apt package available on my system and wired this in as a hacky exec command into my script… and voila! I had hardcopies of all the Mediawiki pages on my system in both wikitext and markdown_mmd format.

Why markdown_mmd I hear you ask… mainly because it gave me the cleanest conversion for use with the new markdown web page widget for Sharepoint’s modern interface.

Now we have the files we could do a little munging and parsing to convert URLs into the format needed for the new location in Sharepoint, easily done with a bit of regex pattern matching, which I wont go into as yours will be very different from mine… suffice to say looking for "wikilink" in my regex helped massively in finding all the occurrences I needed to update. I used sed but you could use whatever tool you like or add it into your version of the exporter

'SysAdmin/(.+) "wikilink"'

and with a little back referencing to substitute the values we need to keep and its all good.

Next came the import of the data into Sharepoint, but that is a post for another day.

package data

import (
	"bytes"
	"fmt"
	"github.com/jmoiron/sqlx"
	"github.com/rs/zerolog/log"
	"io/ioutil"
	"os"
	"os/exec"
	"path/filepath"
	"time"
	"wiki-export/src/util"
)

type Page struct {
	Id int
	Title string
	Content string
	Edited time.Time
}

type Exporter struct {
	Config util.ExporterConfig
	DB *sqlx.DB
}

func (l *Exporter) Export() {
	stmt := `
		SELECT page_id as id, page.page_title as title, pagecontent.old_text as content, page_touched as edited
		FROM mediawiki.page
		LEFT JOIN mediawiki.slots ON page.page_latest = slots.slot_revision_id
		LEFT JOIN mediawiki.content ON content.content_id = slots.slot_content_id
		LEFT JOIN mediawiki.pagecontent ON pagecontent.old_id = CAST(OVERLAY(content.content_address placing '' from 1 for 3) as integer)
		ORDER BY page_touched DESC
		;`

	page := Page{}
	rows, err := l.DB.Queryx(stmt)
	util.CheckErr(err)

	for rows.Next() {
		util.CheckErr(rows.StructScan(&page))
		wikiFilename := fmt.Sprintf("%s.mediawiki",filepath.Base(page.Title))
		mdFilename := fmt.Sprintf("%s.md",filepath.Base(page.Title))
		path := filepath.Dir(page.Title)
		wikiDir := fmt.Sprintf("%s/mediawiki",l.Config.TargetDir)
		mdDir := fmt.Sprintf("%s/%s",l.Config.TargetDir, l.Config.TargetFormat)

		if path != "." {
			wikiDir = fmt.Sprintf("%s/mediawiki/%s",l.Config.TargetDir , path)
			mdDir = fmt.Sprintf("%s/md/%s",l.Config.TargetDir , path)
		}
		util.CheckErr(os.MkdirAll(wikiDir, 0777))
		util.CheckErr(os.MkdirAll(mdDir, 0777))

		wikiTarget := fmt.Sprintf("%s/%s", wikiDir, wikiFilename)
		mdTarget := fmt.Sprintf("%s/%s", mdDir, mdFilename)

		log.Debug().Msgf("%s => %s -> %s", page.Title, wikiTarget, mdTarget)

		c := []byte(page.Content)
		util.CheckErr(ioutil.WriteFile(wikiTarget, c, 0777))

		cmd := exec.Command("pandoc", "-f","mediawiki", "-t", l.Config.TargetFormat, wikiTarget)
		
		var errorBuffer bytes.Buffer
		var outputBuffer bytes.Buffer
		cmd.Stdout = &outputBuffer
		cmd.Stderr = &errorBuffer
		err := cmd.Run()
		if err != nil {
			log.Err(err).Msgf("ERROR: %s", errorBuffer.String())
			util.CheckErr(err)
		}

		util.CheckErr(ioutil.WriteFile(mdTarget, outputBuffer.Bytes(), 0777))
	}
}

Pre-populating Neo4J using Kubernetes Init Containers and neo4j-admin import

Wed, 15 Jul 2020 00:00:00 +0000

Recently there has been an uptake in the use of Neo4j by the Data Scientists. This is a good thing! they are wanting to use the right tool for the job. However we need to run it inside our k8s cluster as a portable readable data source that has been dynamically populated from a pile of data in a combination of PostgreSQL and MongoDB.

This isn’t a problem for them working locally, they install and spin up a local copy of Neo4j and can interact with it quite happily. They even realised that they can generate CSV’s from PostgreSQL and MongoDB and then import them, blindingly fast, into Neo4j using the neo4j-admin tool that comes bundled. Fantastic!

At least until they come to want to run their Neo instance inside our k8s cluster. That’s where I step in and turn them aside from creating their own custom neo4j image with a bespoke entry point that loads all the data for them in some crazy threaded bash scripting!

“No, No, No!” I tell them. “It’s far easier to just add an init container to your pod, that will preload the data before Neo starts up”.

Init containers, if you haven’t come across before, them are a special type of container that lives inside a k8s pod and are set to run BEFORE your main container runs. In this case it means we can easily sequence a bash script to run the neo4j-admin import before Neo4j is even started. And here is how we did it!

The script

The data scientists had been using Neo4j 3.5.x locally because they had a need for the graph algorithms plugin (https://github.com/neo4j-contrib/neo4j-graph-algorithms) which at the time they were looking didn’t support Neo4j 4.x. The plugin is now deprecated and its replacement (https://github.com/neo4j/graph-data-science) thankfully supports 3.5.x and 4.x.

As Neo4j 4.x introduces a lot of new features and improves performance so I recommended we switch to using that. This meant a refactor of their bash script for neo4j-admin there some very subtle differences and a few caveats to work with. This is what they came up with

#!/bin/bash
DBNAME="neo4j"
if [ "$#" -eq 1 ]; then
 DBNAME=$1
fi

# extract data from SQL
python3 extract_data.py

# remove old db for rebuild
rm -rf "/data/databases/$DNBAME"

neo4j-admin import \
 --database=$DBNAME \
 --delimiter="|" \
 --nodes=Protein=${NODE_DIR}/nodes_protein_header.csv,${DATA_DIR}/nodes_proteins.csv \
 --nodes=UniProtKB=${NODE_DIR}/nodes_uniprot_header.csv,${DATA_DIR}/nodes_uniprot.csv \
 --relationships=HAS_AMINO_ACID_SEQUENCE=${EDGE_DIR}/edges_protein_sequence_header.csv,${DATA_DIR}/edges_protein_sequence.csv \  
 --relationships=HAS_AMINO_ACID_SEQUENCE=${EDGE_DIR}/edges_chembl_protein_biotherapeutic_molregno_header.csv,${DATA_DIR}/edges_chembl_protein_biotherapeutic_molregno.csv \
 --skip-bad-relationships=true \
 --skip-duplicate-nodes=true

The import command here is significantly shorter for example purposes, as the original is about 120 lines long. As you can see it’s pretty straight forward, they had another script in extract_data.py, that I wont bore you with suffice to say that it pulled out all the data they wanted from PostgreSQL and MongoDB, which got saved to disk as CSV files in the relevant directories.

Great, it worked on their local version!

The Dockerfile

ROM neo4j:latest
ENV NEO4JLABS_PLUGINS ["graph-data-science"]
RUN apt update && apt install -y python3
WORKDIR /srv
COPY src /srv/src
COPY headers /srv/headers

The plan is always to keep it simple. We have one image that we can run for both the init container and the main container. This docker file gives a vanilla neo4j instance with python and our scripts for extracting the data loaded into it

The k8s Manifest

apiVersion: v1
kind: Pod
metadata:
 name: neo4j
spec:
 containers:
 - name: neo4j
 env:
 - name: NEO4J_AUTH
 value: neo4j/password
 image: registry.example.com/phpboyscout/rnd_graph:latest
 imagePullPolicy: Always
 volumeMounts:
 - mountPath: /data
 name: neo4j
 subPath: data
 initContainers:
 - name: importer
 args:
 - neo4j_import.sh
 command:
 - /bin/bash
 env:
 - name: DATA_DIR
 value: /import/data
 - name: HEADER_DIR
 value: /srv/headers
 image: registry.example.com/phpboyscout/rnd_graph:latest
 imagePullPolicy: Always
 stdin: true
 workingDir: /srv/src
 volumeMounts:
 - mountPath: /data
 name: neo4j
 subPath: data
 - mountPath: /import
 name: neo4j
 subPath: import
 - name: neo4j
 persistentVolumeClaim:
 claimName: neo4j

Now we can pull it all together with our k8s manifest. From here you can see that we have our default neo4j container that we pass in our default authentication details to and an init container that runs our import.sh script. Both containers have access to a shared volume for the /import and /data folders.

And now we get to…

Troubleshooting

So right off the bat it didn’t work! No surprises there but here are a few things that caused us some issues and how we resolved them.

Database offline

At first glance everything seemed to work. Until we tried to connect to the neo4j database with the default UI, at which point we were presented with the error message

Database "neo4j" is unavailable, its status is "offline."

This took a little sleuthing and shelling into the neo4j container to take a look at the /var/debug.log file which gives significantly more useful information about whats going on with the server. First we were getting stack traces that contained messages like

Component 'org.neo4j.kernel.impl.transaction.log.files.TransactionLogFiles@59d6a4d1'
was successfully initialized, but failed to start. Please see the attached cause 
exception "/data/transactions/neo4j/neostore.transaction.db.0"

From experience this sounded like a permissions issue and lo and behold, checking the files on the filesystem showed that because the import script was run as root the database files were owned by root. We resolved this by adding:-

chown -R neo4j:neo4j /data/

to the bottom of the import script. Next we were then presented with an error that looked like

2020-07-14 16:56:33.919+0000 WARN [o.n.k.d.Database] [neo4j] Exception occurred while
starting the database. Trying to stop already started components. Mismatching store id.

This one seems like it would be an obvious one to google and I did come up with few pages that seemed to describe what was happening to me but gave some varied solutions, from starting and stopping the sever and running neo4j-admin unbind in between to deleting various files. It seemed very strange because we did test this with the 3.5.17 version of Neo and it worked fine.

The solution we needed was to wipe the slate clean properly. The line in our script to remove the previous build of the db

# remove old db for rebuild
rm -rf "/data/databases/$DNBAME"

just didn’t cut it. It turns out that because the 4.x version of Neo4j supports multiple databases the import command writes additional information to the system database and transactions database in the form of some identifiers for each database, BUT if you don’t do something to clear that value for the database your are building it wont match up when the server starts and you get a declaration of Mismatching store id

I’m not sure if the developers are aware of this flaw, so in the mean time we have to expand our cleanup to:

# clean up for fresh import
rm -rf /data/databases/*
rm -rf /data/transactions/*

removing the neoj4, system and store_lock databases and transaction logs from the data store. This solved the problem and the server was able to start and we could connect to neo4j database successful.

Its not an ideal solution, I can foresee definite situations we will have to work around when we get to a point where multiple databases may be needed and are built separately and independently from each other. but it will suffice for now.

Malloc(): Error message goes here

Once it was up and running we noticed that we were getting lots of restarts on the main neo4j container a quick look at the stdout log and we could see each restart ending with something that looked like

malloc(): corrupted top size

instantly this looks like an issue with memory sizing inside the container for the JVM. Thankfully the team at Neo4j have accounted for this and give you a nice tool in the form of

neo4j-admin memrec

which interrogates the databases and gives some sensible values you can set in the output which in our case looked like


# Memory settings recommendation from neo4j-admin memrec:
#
# Assuming the system is dedicated to running Neo4j and has 376.6GiB of memory,
# we recommend a heap size of around 31g, and a page cache of around 331500m,
# and that about 22400m is left for the operating system, and the native memory
# needed by Lucene and Netty.
#
# Tip: If the indexing storage use is high, e.g. there are many indexes or most
# data indexed, then it might advantageous to leave more memory for the
# operating system.
#
# Tip: Depending on the workload type you may want to increase the amount
# of off-heap memory available for storing transaction state.
# For instance, in case of large write-intensive transactions
# increasing it can lower GC overhead and thus improve performance.
# On the other hand, if vast majority of transactions are small or read-only
# then you can decrease it and increase page cache instead.
#
# Tip: The more concurrent transactions your workload has and the more updates
# they do, the more heap memory you will need. However, don't allocate more
# than 31g of heap, since this will disable pointer compression, also known as
# "compressed oops", in the JVM and make less effective use of the heap.
#
# Tip: Setting the initial and the max heap size to the same value means the
# JVM will never need to change the heap size. Changing the heap size otherwise
# involves a full GC, which is desirable to avoid.
#
# Based on the above, the following memory settings are recommended:
dbms.memory.heap.initial_size=31g
dbms.memory.heap.max_size=31g
dbms.memory.pagecache.size=331500m
#
# It is also recommended turning out-of-memory errors into full crashes,
# instead of allowing a partially crashed database to continue running:
#dbms.jvm.additional=-XX:+ExitOnOutOfMemoryError
#
# The numbers below have been derived based on your current databases located at: '/var/lib/neo4j/data/databases'.
# They can be used as an input into more detailed memory analysis.
# Total size of lucene indexes in all databases: 0k
# Total size of data and native indexes in all databases: 17300m

So how to get these values into the container… Thankfully this is handled for you in the form of Environment Variables you can pass into the docker image. A bit of a google and i found this little snippet which is a goldmine for telling us how to translate settings into environment variables.

# Env variable naming convention:
# - prefix NEO4J_
# - double underscore char '__' instead of single underscore '_' char in the setting name
# - underscore char '_' instead of dot '.' char in the setting name
# Example:
# NEO4J_dbms_tx__log_rotation_retention__policy env variable to set
# dbms.tx_log.rotation.retention_policy setting

As for getting the variables into the container, you could do this from the pod and inject it in. I this case because the data we are going to be using is reasonably stable and tested we decided to stick them into the Docker file with the ENV directive.

ENV NEO4J_dbms_memory_heap_initial__size 31g
ENV NEO4J_dbms_memory_heap_max__size 31g
ENV NEO4J_dbms_memory_pagecache_size 331500m

And so far we haven’t had a restart yet!

Adding Ambient Sounds to your Discord Server On LInux

Tue, 30 Jun 2020 00:00:00 +0000

I’m a Dungeon Master! I don’t mean that in the S&M sense! As in the game Dungeons & Dragons (https://dnd.wizards.com), where I run a weekly game as well as take part in a couple of campaigns as a player. It’s a lot of fun and something I would definitely recommend you have a go at if you are so inclined

There is a vast amount of tooling & tech out there that allows you to play remotely such as Virtual Table Tops, Character builders, online resources, etc. One such tool that gets used quite often is a chat service called Discord (https://discord.com) It’s really useful and allows you to easily be part of and manage communities of people…. Think IRC & Slack, but more up to date than IRC and less “workish” than Slack.

As part of my online games I like being able to have ambient music to match the surroundings the players are traveling through, as well as some active elements thrown in for good measure. This is possible in a few different ways using discord but the way I want to set it up can be somewhat frustrating to set up. Let me explain:

I have taken a shine to two tools in particular… Syrinscape (https://syrinscape.com) and Table Top Audio (https://tabletopaudio.com). The former being a windows app with an nice interactive mixing UI that allows you to combine and generate unique sounds, the latter being a lovely web service that has some fantastic loop-able ambient background tracks all 100% free.

I am wanting to be able to pipe the audio from these two services into my Discord server so that I can make use of the fantastic audio they offer. This is the journey of how I managed to get this working, partly as a reminder for me if I ever need to do this again and also to help others that may be looking to do the same.

My Setup

I’ve been a big fan of Ubuntu for a number of years, but since 20.04 I’ve found that the shine I’ve had for it has waned significantly. I wont go into the why and wherefore of it but I’m now running the excellent Pop_OS! from System76 (https://pop.system76.com) its an Ubuntu variant but with the bits I dislike removed. So assume that anything I’m doing is compatible with Ubuntu 20.04.

The Requirements

The ideal solution should see me being able to have a single instance of discord running that allows me to still use my mic to be able to talk, and to have my selected background playing with the ability to control the volumes of both the mic and the background independently.

Finding A Solution

A lot of googling led me to realise that there isn’t a perfect solution to fit my brief. The hardest part being not actually knowing what to google and a lot of the terminology being somewhat foreign to me as I’m not much of an audio engineer. However I finally stumbled upon a blog post by Emma Anderson dated June 2016 and thankfully it gives me a lot of the heavy lifting that I needed along with some explanation of what I’m trying to achieve, though I’m hopefully going to be more verbose here in what this all means and how it works.

PulseAudio

The first thing we need to do is make sure the packages for pulseaudio and pavucontrol are installed. These will allow us to manipulate the way we capture sound and redirect it to the appropriate inputs and outputs.

apt install pulseaudio pavucontrol

Virtual Input & Virtual Mic

What we are going to try to achieve, is to create two new elements inside of Pulseaudio;

a Virtual input that we can channel the applications creating our background sounds which will allow us to control the volume independently.
a Virtual Microphone that we can channel our both our normal microphone and the new Virtual input into.

By creating these elements we can then use the pavucontrol tool to select what needs to be redirected where. so lets get started.

pactl load-module module-null-sink sink_name=VirtualInput
pacmd update-sink-proplist VirtualInput device.description=VirtualInput
pacmd update-source-proplist VirtualInput.monitor device.description=VirtualInput.monitor

Here we have two commands, the first will create our new Virtual Input as what is referred to as a “null sink”. This on its own is not really very useful for us as we also need what is referred to as a “source”, thankfully when we run this command it also created a new “source” for us.

On it’s own that should be more than enough, but running the 2nd & 3rd command makes our live a lot easier because it will apply some very useful labels to both of the newly created sink and source. In this case VirtualInput for the sink and VirtualInput.monitor for the source. Having these in place makes it a lot simpler to configure things with pavucontrol.

Next we need to create our Virtual Mic using some very familiar looking commands.

VM=$(pactl load-module module-null-sink sink_name=VirtualMic)
pacmd update-sink-proplist VirtualMic device.description=VirtualMic
pacmd update-source-proplist VirtualMic.monitor device.description=VirtualMic.monitor

again we have now created a new new pair of sink and source with some nice easy to recognise labels that we will use when we start working with pavucontrol.

The next piece to our puzzle is creating the components that will let us define a connection from the VirtualInput and our physical microphone to the newly created VirtualMic. We do this with two identical commands;

pactl load-module module-loopback sink=VirtualMic
pactl load-module module-loopback sink=VirtualMic

we now have most of the elements that we need to configure everything to work.

Listening to my own Ambience

Before we can start wiring it all together we need to ensure we can also listen back to our own ambience. This involves us creating one more “loopback” module that points to the speakers we are wanting to listen to. Lets find out what our options are by running;

pacmd list-sinks | awk '/index:/ {print $0}; /name:/ {print $0}; /device.description/ {print $0}

This lists all of the available “sinks” that we can use. on my daily driver laptop I get;

 * index: 1
	name: <alsa_output.pci-0000_00_1f.3.analog-stereo>
		device.description = "Built-in Audio Analogue Stereo"

This tells us the “index” for the device, its name and also some kind of description. The important bit for us here is the name as we will need that to create our new “loopback” with the command;

pactl load-module module-loopback sink=alsa_output.pci-0000_00_1f.3.analog-stereo

This creates the last piece for our puzzle!

Connecting it all together

I’m now going to assume you have logged yourself into the Discord client and fired up your copy of Syrinscape… but you should just as easily swap out these for something else of your choice.

Now we can start pavucontrol either from the command line or you can look for it in your applications menu. Once it loads you will hopefully be presented with something that looks like;

For this next step I am specifically starting on the “Recording” tab of pavucontrol this is to allow us to set up what is going to be captured. I have updated the drop-down at the bottom left to show “All Streams” as this will make it quicker to configure… Starting at the top we have two entries for;

Loopback to VirtualMic from: These are the result of the first two “loopback” modules we created with the pactl command we ran previously. They are going to allow us to capture the audio streams from our physical microphone, mine here is the TONOR TC-777 and our newly created VirtualMic.

followed by a single entry for;

Loopback to Built-in-Audio Analougue Stereo from: which is the last “loopback” module that we create to let us hear our own Ambience, Having this set to our VirtualInput means that anything that pipe into our VirtualInput will also come out of our speakers.

and finally;

WEBRTC VoiceEngine: Once you connect to a voice channel in discord this will appear and it allows us to specify which of our devices it should be reading the audio feed from. For our purposes we have this set to our VirtualMic so that we can have our mixed audio feeds

Now that recording is configured we can sort out our playback.

Here we can see the “Playback” tab of pavucontrol, again set to show “All Streams”. This time I’m going to run through the elements here starting from the bottom of the list and working my way up…

WEBRTC VoiceEngine: This again is our connection to a Discord voice channel, as you can see I have this set to play back all of its output via Built-in Audio Analogue Stereo which is how my Operating system has labelled my physical speakers.

Syrinscape.exe: This is the Syrinscape application, that I run through PlayOnLinux (https://www.playonlinux.com), and I will use to generate all of my lovely ambient sounds. This is set to play all of its “audio stream” on our VirtualInput.

The next two items in our list, providing you have configured the Recording tab first should look as in the image. Changing the “Loopback to VirtualMic” entries on the Recording tab will change the labels of these two entries.

Loopback of VirtualInput.monitor on: it seems we have two of these entries and where we can tell the to pip all of the audio we are now capturing on our VirtualInput In this case we want it to go to two places, our VirtualMic so that it can be sent to both our Discord audio channel and also to our Built-in Audio Analogue Stereo speakers.

Loopback of Built-in Audio Analogue Stereo on: is where we now direct the input from our physical microphone and feed that straight into our VirtualMic.

The other entries in the list here are for firefox and the system itself and are not relevant to what we are trying to achive.

Winner Winner Chicken Dinner

That’s is effectively all we need to do… From here on in anything you play via the Syrinscape app will be merged with your microphone input and passed to Discord. You can then use the volume sliders in pavucontrol to adjust the levels of all the inputs to suit your own personal preference.

Though I will make a few small suggestions about how to configure your discord settings. You shouldn’t need to make any adjustments to the input and output devices which should now be set to Default as your “Input Device” if you change this it will override the changes we have made and you will need to go back to the Recording tab of pavucontrol and switch WEBRTC VoiceEngine back to InputMic , but…

I would recommend disabling automatic input sensitivity and lowering the sensitivity slider all the way down to -100dB… this is to allow for the potential low and subtle tones and ambient elements you may want to play… be warned though it makes it very very easy for an low quality microphone (such as the Built-in Audio Analogue Stereo microphone found on my laptop) to pick up other noises such as your systems fans, mouse clicks and typing. A simple way to combat this is to get a reasonable quality external cardioid condenser microphone which eliminates a lot of this unwanted background.

One last thing

That should be it for now… I’ll leave you with one final thing. This is a simple little bash script I threw together that I run in can a terminal to create all the components and if I want will then clean them all up and remove them. If you really want you could set it up as a permanent implementation, but I’ll let you google for that solution!

#!/bin/bash

LB1=""

listenback() {
 echo ""
 echo "Listing all possible output devices"
 pacmd list-sinks | awk '/index:/ {print $0}; /name:/ {print $0}; /device\.description/ {print $0}'
 echo ""
 echo "Please enter the name of the output device to create a loopback for (leave blank to skip): "
 read S1
 if [ "$S1" != "" ]; then
 	echo " * Creating Loopback for '$S1'"
 	LB1=$(pactl load-module module-loopback sink="$S1")
 fi
}

cleanup() {
 while true; do
 read -p "Finished? do you want to clean up and remove modules [Yn]: " yn
 case $yn in
 [Yy]* ) return 0;;
 [Nn]* ) return 1;;
 * ) echo "Please answer yes or no.";;
 esac
 done
}

listenback

echo " * Creating VirtualInput"
VI=$(pactl load-module module-null-sink sink_name=VirtualInput)
pacmd update-sink-proplist VirtualInput device.description=VirtualInput
pacmd update-source-proplist VirtualInput.monitor device.description=VirtualInput.monitor

echo " * Creating VirtualMic"
VM=$(pactl load-module module-null-sink sink_name=VirtualMic)
pacmd update-sink-proplist VirtualMic device.description=VirtualMic
pacmd update-source-proplist VirtualMic.monitor device.description=VirtualMic.monitor

echo " * Creating loopbacks for VirtualMic"
VML1=$(pactl load-module module-loopback sink=VirtualMic)
VML2=$(pactl load-module module-loopback sink=VirtualMic)


echo "All modules have been loaded have been configured! Run pavucontrol to configure your devices."

if cleanup; then
 pactl unload-module "$VML2"
 pactl unload-module "$VML1"
 pactl unload-module "$VM"
 pactl unload-module "$VI"
 if [ $LB1 != "" ]; then
 pactl unload-module "$LB1"
 fi
 echo "All modules have been unloaded"
else 
 if [ $LB1 != "" ]
 then
 echo "Modules $LB1, $VI, $VM, $VML1 & $VML2 remain loaded"
 else
 	echo "Modules $VI, $VM, $VML1, & $VML2 remain loaded"
 fi
fi

Encrypting additional drives with LUKS on Linux

Mon, 29 Jun 2020 00:00:00 +0000

Encryption is king nowadays with everyone having mobile devices. We have a significant number of people on laptops that travel around and also workstations that live in open plan offices. This means we encrypt all of our disks… just in case. 99% of the time is super simple to do as most OS installers give you the option to do it, some now ven enforce it as a default option. This post however is about adding an additional disk to the system and making it automatically mount on system startup.

So let me set the scene, we have a data-scientist that’s running out of disk space for a task they are running on their Ubuntu 18.04 Workstation. At some point the workstation had an upgrade to the HDD in the past to a shiny new SSD, and the old 4Tb spinning disk was left in the chassis that they want to use for this very specific task.

Now this workstation has been through a couple of data-scientists over the last 12 months and unfortunately the LUKS password that had been set up for the old spinning disk has gone walkabouts… so the plan is as follows

flatten the old disk and set up a new partition using the whole disk
generate a new secure encryption key
set up LUKS encryption on the new partition
use Ext4 as a filesystem
enable auto decryption of the disk
add the new partition to the fstab to mount on system startup

N.B Assume that we are running everything as the root user

Flatten the disk

As we cant recover anything we are going to flatten the disk using parted (apt install parted to install) to allow is to create a partition greater than 2Tb, but first we are going to identify the disk we are working with… I tend to favour using either fdisk -l or as a more concise option lsblk -p which gives us a an easy to interpret overview something like:

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk

I can tell from this that we are looking at using the disk that is currently at /dev/sdb and its showing as being 3.7Tb in size.

Great… now to set up our new partition using the command parted /dev/sdb which gives us an interactive shell to work with (you can see the prompts in the output below are prefixed with (parted)

GNU Parted 3.2 
Using /dev/sdb 
Welcome to GNU Parted! Type 'help' to view a list of commands. 
(parted) mklabel gpt

The command mklabel gpt will wipe the partition table for /dev/sdb and give us a clean slate to work from

(parted) unit TB

We now set parted to think in Terabytes as the default reference size using the command above.

(parted) mkpart primary 0.00TB 3.70TB

Now we get to create the actual partition. You can see from the command above that we are using the command mkpart and telling it to create a primary partition type.

(parted) print 
Model: ATA WDC WD4005FZBX-0 (scsi) 
Disk /dev/sdb: 4.00TB 
Sector size (logical/physical): 512B/4096B 
Partition Table: gpt 
Disk Flags: 
 
Number Start End Size File system Name Flags 
 1 0.00TB 4.00TB 4.00TB primary

We can chek everything went smoothly using the print command which gives us confirmation that a new primary partition is present. We can now leave parted with a simple.

(parted) quit

And we can now use fdisk -l or lsblk -p to see that we now have a partition waiting for us at /dev/sdb1.

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk 
└─/dev/sdb1 8:17 0 3.7T 0 part

Generating an encryption key

Our disk is now ready for use, but not yet encrypted, so our next step is to create a key that can be used when we encrypt the disk. As we are going to be mounting it automatically we want to use a keyfile to store the key. You can of course create a key by mashing the keys on the keyboard, but I tend to prefer letting something else do the hard part for me.

First we create somewhere to store the key… I opted for,

mkdir -p /etc/crypt/keys

But feel free to put it wherever you want just as long as its only accessible by the root user. Next we generate the keyfile using the command:

dd bs=512 count=4 if=/dev/urandom of=/etc/crypt/keys/sdb1 iflag=fullblock

Here I am using /dev/urandom as my randomness generator, but you could use any valid generator of your choice. With this set of parameted dd will read the stream of “randomeness” and write 2048 bytes to our keyfile at /etc/crypt/keys/sdb1. If you want to be a little more complex about teh size and shape of your key then have a look at https://man7.org/linux/man-pages/man1/dd.1.html

Encrypting the Disk

Hopefully it will already be installed because you encrypted your root disk at installation, but if not you can run apt install cryptsetup to get going.

The command to do the encryption is actually very simple.

cryptsetup luksFormat /dev/sdb1 /etc/crypt/keys/sdb1

You can see that using the cryptsetup tool we are asking it to execute teh command luksFormat but while it says format in the command this is a little misleading as it doesn’t actually format the disk but just rewrites a portion of bytes at the beginning of the partition to enable encryption. we then tell it the partition we want encrypting, here its /dev/sdb1 and finally we pass in the keyfile we just generated and saved at /etc/crypt/keys/sdb1. If you omit the keyfile it will still encrypt teh disk but will prompt you to enter the key manually.

As soon as you press enter you will be warned of teh danager of what you are doing… so double check you are encrypting the right partition and follow the instructions that should look something like :

WARNING! 
======== 
This will overwrite data on /dev/sdb1 irrevocably. 
 
Are you sure? (Type uppercase yes): YES 
Command successful.

And that’s it… the disk is encrypted and ready to use. There are a few ways you can now work with the disk. the quickest and easiest is to just decrypt the disk manually using cryptsetup to open the disk.

cryptsetup open /dev/sdb1 sdb1_crypt -d /etc/crypt/keys/sdb1

Here we open /dev/sdb1 and give is a new name of sdb1_crypt and we unlock it using the -d argument to tell it the keyfile we generated before.

That is the dis decrypted and ready to roll… you can now use fdisk -l or lsblk -p to confirm that it is now available at /dev/mapper/sdb1_crypt.

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk 
└─/dev/sdb1 8:17 0 3.7T 0 part 
 └─/dev/mapper/sdb1_crypt 253:3 0 3.7T 0 crypt /mnt/4tb-1

This tells us that the newly decrypted disk is now available at /dev/mapper/sdb1_crypt and is a volume of 3.7Tb… Exactly what we were hoping for!

All finished with your encrypted disk… you can just as easily close it again using:

cryptsetup close sdb1_crypt

Setting up the Filesystem

Ok, we have an encrypted partition, we can decrypt it but we cant mount it yet as we don’t have a file system to work with. Let’s take care of that real quick by opening up the partition again.

cryptsetup open /dev/sdb1 sdb1_crypt -d /etc/crypt/keys/sdb1

And now that its available we are going to set up an ext4 filesystem using the command mkfs.ext4 /dev/mapper/sdb1_crypt which, all going according to plan, should look something like:

mke2fs 1.44.1 (24-Mar-2018) 
Creating filesystem with 976753664 4k blocks and 244195328 inodes 
Filesystem UUID: d797be67-c53e-49d3-897e-c624b21a22d3 
Superblock backups stored on blocks: 
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
 102400000, 214990848, 512000000, 550731776, 644972544 
 
Allocating group tables: done 
Writing inode tables: done 
Creating journal (262144 blocks): done 
Writing superblocks and filesystem accounting information: done

we are now good to go… lets try mounting the filesystem with

mount -t ext4 /dev/mapper/sdb1_crypt /mnt

I’m just mounting straight to /mnt but obviously this can be any folder you want. If the command worked we can easily confirm it with a quick df -h:

Filesystem 1K-blocks Used Available Use% Mounted on
udev 32846708 0 32846708 0% /dev
tmpfs 6578140 2308 6575832 1% /run
/dev/mapper/ubuntu--vg-root 1919562064 993193376 828790500 55% /
tmpfs 32890688 200 32890488 1% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
tmpfs 32890688 0 32890688 0% /sys/fs/cgroup
/dev/sda2 721392 276068 392860 42% /boot
/dev/sda1 523248 6232 517016 2% /boot/efi
/dev/mapper/sdb1_crypt 3844637680 0 3844637680 1% /mnt

Excellent… you can now start working with your new partition… however lets un-mount and close the drive quickly with a

umount /mnt && cryptsetup close sdb1_crypt

And then we can move onto…

Automatic Decryption

This is a lot simpler that you may realise… all we need to do is add a new line to the file /etc/crypttab! But first we need one last piece of information we don’t yet have, but we can easily get with the command

sudo cryptsetup luksDump /dev/sdb1 | grep "UUID"

This will use luksDump to get information about the encrypted partition and then uses grep to specifically target the property UUID which we will need to identify the partition in the next step.

Now in your favourite editor of choice add the following line, replacing the spoof UUID here with the one we just found.

sdb1_crypt UUID=1111111111-2222-3333-4444-555555555555 /etc/crypt/keys/sdb1 luks

Here we are giving the decrypted volume a unique label for teh decrypted label to be made available at the appropriate /dev/mapper/* location. We also specify the UUID to identify the partition to decrypt… we could use the path /dev/sdb1 but using the UUID is more explicit and prevents any confusion if another partition happens to present itself as /dev/sdb1 at some point in the future. Third we have the path to our newly generated keyfile and finally we have the encryption mode that we are using for encryption which here is luks. For more info on crypttab have a look at https://www.freedesktop.org/software/systemd/man/crypttab.html

We can now test that auto decryption is working using:

cryptdisks_start sdb1_crypt

which if successful should have an output like:

 * Starting crypto disk... 
 * sdb1_crypt: INSECURE MODE FOR /etc/crypt/keys/sdb1, see /usr/share/doc/cryptsetup/README.Debian. 
 * sdb1_crypt (starting).. 
 * sdb1_crypt (started)...

All that’s left to do now is set up

Auto-mount the filesystem

Hopefully we now are on really familiar ground… we can now treat /dev/mapper/sdb1_crypt as a bog standard ext4 partition that can be mounted via the /etc/fstab by adding the line:

/dev/mapper/sdb1_crypt /mnt ext4 defaults 0 2

As you can see its pretty ordinary, exactly as you would expect, obviously swapping out /mnt with the location of your choice to mount the filesystem. If you are not wholly familiar with fstab then its definitely worth having a look at https://help.ubuntu.com/community/Fstab as it gives a good overview for those who are new to it…

Finally we can check that it all works with:

mount -a

And at this point I am pretty sure I can hear the fat lady singing…

Connecting to Ubuntu 18.04+ using RDP

Mon, 20 May 2019 00:00:00 +0000

We have a mix of different setups that the Software Engineer and Data Scientists use to get their work done. There are some using just Linux on laptops, Some on MacBooks and some on the various versions of Windows.

For those not using Linux as their primary OS we have a bunch of Desktops that run Ubuntu 18.04+ for them to connect to. SSH can do quite a lot but a few of the team work remotely and in house we prefer RDP for that kind of thing rather than VNC.

We have had some issues with connections in the past so this post exists to remind me how next time I need to set it up. First we need to install the xRDP server package.

sudo apt install xrdp

Next we need to ensure that we have the right ports open on the workstation. If like me you also use UFW to manage your firewall rules then open port 3389 using…

sudo ufw allow 3389

The issue left, is that you will get an annoying pop up when you log in about a colour management profile needing to be set up and asking you to provide your password. Even then you may still get some annoying crash pop-ups.

I found a really good solution to this at http://c-nergy.be/blog/?p=12043 which I’ve cribbed and paraphrased below

Create the file /etc/polkit-1/localauthority/50-local.d/45-allow-colord.pkla (using your editor of choice and sudo)and add the following contents

[Allow Colord all Users]
Identity=unix-user:*
Action=org.freedesktop.color-manager.create-device;org.freedesktop.color-manager.create-profile;org.freedesktop.color-manager.delete-device;org.freedesktop.color-manager.delete-profile;org.freedesktop.color-manager.modify-device;org.freedesktop.color-manager.modify-profile
ResultAny=no
ResultInactive=no
ResultActive=yes

We now need to clear any crash dumps from the workstation

sudo rm /var/crash/*

You should then be good to connect to using whatever RDP client you prefer… I like Remmina myself but each to their own.

Technical CV writing is hard

Wed, 15 May 2019 00:00:00 +0000

Recruiting people is effing hard!

That’s all… I’ll get back to reading through CVs now and let you get on with your day!

Medicines Discovery Catapult is, at the time of writing, recruiting Software Engineers, and as “Head of” it falls to me to start filtering through the CVs that land in my inbox. But what a mess! I get all sorts, from 1-page masterpieces that look amazing and all glossy, but tell me nothing about the individual to 10 pages of war and peace that have so much in it spanning 20 years of experience that 70% is actually completely irrelevant to the job they have applied for.

So now I get to say that this is what the perfect portrait of a CV should look like… I’m sorry to disappoint but I don’t think there is such a thing as a “perfect” CV. It’s far too subjective and open to interpretation. Instead, I’m gonna rip apart my own CV and explain why I chose to write it like I have and justify reasons why I would expect to see similar things on the CVs I am forced to read.

If you want the “TL;DR” version you can find a copy of my CV at https://docs.google.com/document/d/1MM_6nXIVU_wbrvkhdGL5xaJEwc46L7JblUqUot_g1ec but if you have gotten this far you may as well stick around and read the rest. I promise it won’t take long.

The Basics

OK! let’s start with some general observations I follow with regard to my own CV. Then we can get into the nitty-gritty of how bad mine is!

Recruiters

Love them or loathe them they are a part of the recruitment ecosystem and they will help you get hired. When you send them a copy of your CV bear in mind that they will most likely try to squeeze it into a format that will include their own branding and a cover sheet with some pertinent details on it. They may also then start redacting information off your CV in an effort to anonymise you. It is also possible that they will use the same CV for multiple roles meaning you will need to insist that they send your tailored CV if you are really bothered about winning a specific position.

Make sure to discuss this with your recruiter before having them put you forward for a role you are keen on.

Keeping Up to Date

Design your CV so that it is easy to tweak and update! Keeping your CV up to date is essential, and I don’t mean just adding your last role when you finally get fed up with your current employer abusing your good nature and rage quit!

Keeping your CV in top form means that you need to review the entire thing whenever you make an update. As your career progresses you will find that your perspective will change on what previous roles consisted of and how they would impact/reflect on the position you now desire.

You will also find that an up-to-date CV is easier to tailor to any specific job you might be applying for

Prettiness

I know his name is Riccardo, but I’m struggling to focus on anything else

Having a pretty CV is great! and if creativity is relevant to the vacancy you are hoping to fill then go for it… make it gorgeous. It is however very subjective so make sure that you understand what your potential employer is looking for. Sometimes less is more, especially considering they may be scanning through hundreds of CVs. If they have to spend half an hour looking for something specific, then it’s gotten lost in all that creativity. If you want to see some gorgeous-looking CVs take a look at https://weare.guru/creative-cvs/ all of them are creative and beautiful…

But… they could take 20 minutes, or more, for someone to read and extract the right information. Which will waste a lot of time for your potential new boss. Plus we are talking about technical CVs and I am not the most creative of individuals, so keeping it clean and well formatted with consistent fonts usage and sizing

Length

I’ve heard a lot of people say different things about how long your CV should be. 1 side of A4… 2 sides of A4 but only if it’s printed double-sided, even “length doesn’t matter” because the more information you put in the better.

I would recommend taking a middle-of-the-road approach. Keeping things concise is paramount… but if you need 3 or 4 pages then that’s OK… as long as you make the content captivating and interesting for the reader, that is what matters

PDF

This one feels like it should be a “no brainier”… Make sure to submit your CV as a PDF!

There are two very good reasons behind this. First, it ensures that the reader will view it exactly as you intended. If you send it over as a Word, Google, OpenOffice or other such documents, then you are not guaranteed the reader will be using the exact same tools. I for one don’t use Word and an awful lot of presentation is lost because someone used a fancy word feature or have a font that I can’t get hold of.

Second, While it is not impossible it discourages recruiters from tampering with it and spoiling all your hard work. I have known only a few recruiters in my time that are willing to learn (or pay for) a good PDF editing suite. It is possible for them to alter things, but they tend to have to jump through hoops.

Beginnings

And so we have the start of My CV. Looks pretty boring, doesn’t it? Black text on a white background. Nothing fancy!

We have some basic contact details that can be used to contact me and a very short profile statement. That is all I feel an employer needs to see before we get into the next section of my CV. On the surface, it doesn’t actually say much but lets scratch a little deeper.

Font

I have specifically chosen a font that I think looks clean and professional, With a nice easy typeface, it becomes easy for an employer to scan the CV. I keep my CV in Google Docs so I went with Raleway, which I think is nice, clean, professional and easy to read.

I’ve also chosen consistent font sizes and spacing;

16 for the main title
14 for headings
12 for subheadings
11 for everything else

Limiting Liability

Middle-aged, liberal, heterosexual white male programmer Swipe right to Hire Me!!! Please!

It may sound odd but I put as little personal information in my CV as possible. Hence why just my name and contact details exist. I don’t mention my age, gender, race, driver status or political preferences. People come with inherent biases, that is just a fact of life. so putting as little as possible negates triggering these biases.

I’ve seen people put all sorts of information into CVs… a personal bugbear is photos. As they can introduce a massive amount of opinion in the eyes of the beholder. Think Tinder but for recruitment!

So unless specifically asked for I would strongly recommend keeping it to a minimum and using the space it would consume in selling the things that matter.

Profile statement

1 paragraph! that’s all I needed to say regarding what is effectively a personal statement. This is not a UCAS application and I don’t need to detail a million things about myself. What is to come next will be the real sales pitch

My profile statement focuses specifically on what my future employer is going to get from me if they hire me. Passion! It is a statement of intent, specifically written to demonstrate that I will strive to bring my “A game” to anything I do in the future and also that I intend to encourage those around me to do the same. In effect, it is the opening line to what is a sales pitch.

I could expand on this to state other goals and ambitions but that would just detract from the main objective of the CV. Plus we will get an opportunity to elaborate later on in the recruitment process.

Skills

Now we come to the meat and bones of the CV. This is the headline! the bit that you will tailor the most in order to impress whoever is scheduling those interviews and guarantee you a chance to shine. If you can make this captivating enough for the person you need to impress they will then be more than happy to read the rest

My CV has a LOT of skills listed. This is mainly because I’m a show off more than anything else and in reality, I will tailor this heavily to suit whatever role I’m applying for. For example, if it doesn’t have a requirement for managerial skills I would drop that section entirely. Depending on the required attributes and skills asked for I would happily add/remove bits to any section of my skills.

With different types of skills, I will define them differently. As you can see I am quite generic with Managerial skills. A lot of people understand these and are looking for confidence in your ability to do certain types of tasks. Technical skills define different aspects of what I do that are not specifically tied to languages. It can be very difficult to be curt and to the point here. I favour labelling a technology area and then listing a few of the most prominent or recent items from my repertoire. A bad example on my CV is databases… I can use quite a few as you can see, but in reality, I’ve been greedy by listing MySQL, MariaDB & Percona… they are all effectively connected to each other, but my need to show off is far too much for me to resist.

Experience

When I first started out as a Dev I had a hiring manager who once told me that most of the time the main thing he wanted to see was;

did the applicant have the skills he wanted
what was the candidates’ personal assessment of their ability
how much genuine commercial experience do they have

Three really simple things that stuck with me. So the next time I came to write my CV I added a table for Skills which had only a few things on it back then but has grown massively over the years. I’ve tweaked and tailored it over the years, adding and removing things that were relevant as needed.

By quite clearly labelling my own perception of my skills, I give the person reading it an indication of my potential value. It allows them to tailor the interviews and technical tests to fit me as an individual. It also acts as a double-edged sword as the higher I grade myself the more chance of falling flat on my face when I get asked a question I can’t answer… So it’s always better to try and be accurate, but also not to be timid, there are always other competitors in the recruitment race.

I favour using really simple words to identify my skill level; beginner, intermediate, expert and expert+. It makes it simple for the reader to gauge and also allows me to mix the terms around

Stating how many years of commercial experience you have lets the reader understand your commercial experience. The number of times I’ve had recruiters do a typical keyword match and see that I mention c# once on my CV and then try to wave a job advert under my nose. I did .NET for 6 months 10 odd years ago. I legitimately can claim that experience, but it should not be the core on which I base my next adventure.

As a useful side effect, by stating a duration in years prompts me to review and update my CV regularly to keep it up to date.

Previous Employment

By this point, I’m hoping that I’ve captured the readers’ imagination. That they now envision a Development God has graced them with a CV worthy of filling any role they have…

And then come back down to earth with a bump! Next up in the firing line is Work Experience. Here I define some of the previous positions I have held in my illustrious career in somewhat chronological order. I say some because I have a general rule of limiting what I put on the CV to either 10 years or 10 positions, whichever comes first. This hasn’t been a hard and fast rule over the years, and it’s fine to flex in order to suit the roles I’ve gone for. There is no reason whatsoever though to go so far back in time as to describe my time as a pot-washer when I was 17.

In some cases, I’ve also omitted some things from my CV. Somewhere in and around 2012 - 2015; I happened to found and run a co-working space in Manchester City Centre. It was an interesting venture that I’m really proud of and am glad to say is still running even though I’m no longer a part of it. But in truth, it does not add anything of value to my CV for the positions I plan to go for in the future.

To the point

I don’t like big sprawling paragraphs of text if you haven’t already got the gist. So for me, bullet points are the way forward. I try to keep it fairly obvious and detail achievements and document things I have done, and not the tools I have used.

I try to explain why I joined the company and what my purpose was. I highlight important successes and demonstrate improvements I made to the company. Short succinct sentences should be chosen to illustrate aspects of your accomplishments that actually relate to your future goals in your new position.

By doing it this way you also make it easier to tweak and change things without having to restructure a whole piece of prose in the future.

That said my own CV has a glaring exception! When I was doing freelance work I was not able to accurately describe everything I was doing due to some pesky Non Disclosure Agreements. So instead I have a simple paragraph providing a positive high-level explanation of what benefits I brought to my clients.

Formatting

There is a lot of information condensed into this section which makes could make it hard to read if left unformatted. I stuck to the font sizes I had chosen previously and decided to use a more subtle combination of indentation, italics, underline and bold to make it more pleasing to the eye and easier to scan.

Each company name acts as a subheading with a font size of 12. I take a little liberty and include on the same line the dates that I was with them. This gives a clear timeline of events that a manager can then refer to quickly when they need it, such as in an interview.

Indenting everything under a subheading makes it easier for the reader to separate out the content easily. I include an address for the company. I am not actually sure why if I’m honest, it’s just something I’ve always done. I put this in Italics mainly because it’s an aside to the core of the information.

The job title will come next in bold of course to help it stand out. Followed immediately by the relevant bullet points. My last role obviously plays the most prominent part as it’s my headliner. In the case of Medicines Discovery Catapult. I’ve held two roles, “DevOps Engineer” and “Head of Software Engineering” so I break these into their own sections within this piece of experience. Prior to that, I was with Wakelet and here I merge the two roles I held there in order to conserve space.

Education

The bulk of the hard work is now done! We have put in the sales pitch and hopefully, we are close to being invited to an interview. Time to put in some supporting information

This may come as a surprise to some people but I didn’t do the whole University thing… I mean I lived in a University city and frequented the student union bars, but was never actually enrolled in a course. To this effect, I bolstered my CV when I started out by going and obtaining some (now significantly outdated) professional certifications.

Regardless of my lack of educational achievements I would always recommend keeping it simple unless it is your first position and you have no work experience (writing a CV in that situation probably needs to be another blog post entirely). List them in chronological order with some dates and a summary of what you obtained… no one needs to know that I completely fluked getting a GCSE in Art.

Were I to have a Degree I would obviously have the institution and dates in there along with my final marks. I would also look at listing the modules I completed that are relevant to my career, providing I did a Computer Sciences degree and not something like Biology. Again, it’s all about keeping to the point and providing specific information to bolster everything else you may have done.

Wrapping up

Time to finish off with a little bit of something personal. I don’t want to spend too much time here but I want to show that there is more to who I am than just work. I decided to keep it simple, a simple list of things that I enjoy doing in my spare time (When I have any, being a father of 3). The idea here is that these can become conversation pieces with the people that may be interviewing you. I’ve ended up a number of times having interviews where I talk about Scouts.

The final flourish here should be something simple but gives someone a helping hand at learning more about you if they feel so inclined. I include a link to my blog and my GitHub account, but you could include anything that may be relevant.

Last but not least… References are always available on request. My referees are varied and have changed over time. Adding them to your CV doesn’t actually impart any other information that could get you hired. Name-dropping is not the right way to get a job.

Conclusion

OK! so it’s not a pretty CV by any stretch of the imagination. It’s a bit on the long side in its full un-tailored, raw form, though not as long as it could be if I wasn’t being diligent in how I want to present myself. But this is the format I’ve used as my CV for at least 15 years now and I would say that I’ve been really successful in getting interviews out of it. I would say I get at least an 80% success rate of conversions from seeing my CV to a first-stage interview (watch me now jinx myself for the future).

A CV will never get you the job! It’s all down to you excelling in an interview situation and proving how awesome you are and that you can do everything you say you can on your CV. All it is meant to do is get your foot in the door. Hopefully, this breakdown of my CV will help you to take a look at your own CV and work on ways to improve your chances.

Dell DisplayLink D6000 & Ubuntu 18.04+ Issues

Tue, 14 May 2019 00:00:00 +0000

I love Ubuntu… I’m pretty fond of dell kit too!

So I was rather chuffed when I started working at Medicines Discovery Catapult because they let me have both. When you look at my desk it looks like it could be an advert for Dell. Laptop, monitors, dock, keyboard and mouse…. its great when you have a corporate account with a Dell reseller

However while I’ve had a lot of success with the D3000 DisplayLink dock on Ubuntu I found that I’m now having to deal with the upgraded D6000… which doesn’t play very nicely with the more recent versions of Ubuntu (we are talking 18.04 and later)

I kept finding that after a random amount of time the D6000 would randomly seem to power down… I would lose the screens, audio, networking and USB. and the only way I could fix it is to unplug it from teh laptop and plug it back in. Not ideal, especially if I’m in the middle of a video call or debugging something on the net

Being the kind of techie I am my first port of call checking my logs… but I couldn’t see anything that would cause this random disconnect. So off to google I went… eventually I found a lot of information telling me it was part of power management causing things to start powering down… In this case it implied that it was something trying to suspend USB… which sounded really plausible!

So a little more research suggested that I should be using laptop mode tools to disable the ability for USB to be suspended. I gave it a go, though I was dubious as in my mind I shouldn’t have needed to install an additional package (albeit a great one for tweaking your power management on a laptop running Linux)

Alas no joy! And I had too much to do to start debugging in depth and ripping apart other peoples code to figure it out.

What did I do? you ask. Well, I just put up with it for a few weeks, but gradually it began to grate on my nerves. However there was that one day where it didn’t turn off… and that left me perplexed… I checked if any updates had been applied in my last apt update && apt upgrade … nothing…. it then dawned on me that I had plugged in the headset I used for conference calling into the audio in/out on the dock instead of directly into the laptop.

Now I had a little more information I was able to deduce (with googles help) that the laptop was actually suspending USB, but that the trigger was actually pulseaudio. At this point it becomes really easy to solve the problem.

Solution

Edit /etc/pulse/default.pa using your preferred editor (and sudo)

Find the line

### Automatically suspend sinks/sources that become idle for too long
load-module module-suspend-on-idle

And comment it out and save!

Lastly, because its run as a user service you need to restart the Pulse Audio daemon using the command

systemctl --user restart pulseaudio.service

or you could just logout and back in again

A reboot and a legacy moniker

Tue, 26 Feb 2019 00:00:00 +0000

So… my last post was a good 2 years ago now…. Hi how have you been?

It’s been a very busy couple of years with a lot of stuff shifting in my personal life meaning things inevitably take a back seat. However its been long enough that I needed to give myself a new start and see about blogging again and getting back into speaking again.

Still PHPBoyScout?

Since 2016 I’ve jumped around in my career a lot! This has exposed me to a lot of new languages and tech, all of which has been awesome but it now means that the moniker of PHPBoyScout probably isn’t accurate all that much any more. That said I still have a love for PHP and my first thought when presented with a new challenge is still “How would I approach this using PHP?”, so I think that means I can keep my twitter handle for a little longer rather than re-brand myself :-D

I’ve always had a penchant for picking up languages pretty quickly and have been an advocate of the idea that as long as you can think in the right way then languages can be learnt easily enough.

Employed and Employable

October 2017 saw me take on a new role with a company called Medicines Discovery Catapult. They are a grant funded not for profit organisation that is specifically focused on shaking up the medicine discovery pipeline by helping SMEs & CROs innovate and collaborate.

My role is very broad as I appear to have fallen in to the role of Head of Software Engineering very quickly. Though I started as a DevOps Engineer it has evolved very quickly. We work with a wide array of languages and tools and nothing is off the table if it helps to solve the problems we are working on. Currently we actively have code being written in Python, Node, Go & Scala (I live in hope that I’ll be ale to bring PHP into that mix eventually). Vast data sets and AI are the name of the game with my team of engineers helping to support the data scientists and Informaticians.

At the time of writing we are also hiring so if you are curious take a look at https://md.catapult.org.uk/about/careers/ and see if you like the sound of what we are doing and the challenge on offer.

Moving forward

The plan moving forward is to post about the tech we are working with, talk about the types of challenges we are facing and also maybe restart my public speaking career. Stay tuned for more and if you don’t hear from me soon then give me a nudge

Project Slayer: The Critical Path

Tue, 26 Feb 2019 00:00:00 +0000

The first of my new round of talk abstracts! In all honesty this isn’t a talk but something more that came out of a very drunken Saturday night at #phpbnl19. There were a bunch of us sat talking and somehow the topic of D&D came up which sent my mind racing with this idea… By the time 2am rolled around I had a fully formed idea along with some willing players to help with the idea. Now I just need to find a conference willing to take a chance on it! If you know a conference that would be interested let me know

Description/Abstract

Our heroes have just completed our latest quest! Having successfully delivered the latest iteration of their project they are approached by the aged and holy sage “PeeEm” with a new quest! How quickly can they implement the sacred and forgotten art of “Lo-ging” into the codebase

Do they accept? If they do will they be able to defeat the trials and tribulations that await them? Will they find treasure and glory, or suffer defeat at the hands of the vile and depraved Stakeholder?

What will people learn

The only way to find out if our heroes will complete this epic quest will be to join us and see if the dice of fate will be kind to them!

Along the way we will will learn some truths about feature implementation and how our heroes handle the challenges that lie ahead. And hopefully gain enough experience to level up!

Additional Information

This is an extremely unique talk! It takes the form of a live Dungeons & Dragons game. I will be on stage playing the part of Dungeon Master looking to guide our team of adventurers through the process of delivering a new feature for a project.

Our heroes currently consist of a heroic bard who will be inspiring our heroes, and audience, with ballads of past glories. A warlock with the demonic power to fork and merge code like no other in existence…. Our sorcerer has the innate and wild magic of Fire(base)! Finally our team is held together with the support of our cleric, worshipping the ancient god Rasmus.

The outcome of this adventure will genuinely be determined by the roll of the dice! It will be a game of 5th edition D&D that we can probably fit into an hour… but 2 or more would be better and far more fun.

Audience participation is expected! Cosplay is hoped for! I will, of course, be in full scout uniform!

Monty Python explains why your project failed!

Sun, 07 Feb 2016 00:00:00 +0000

As part of the attempt to develop my profile as a speaker, I’ve realised that I sometimes need to explain a few of my current talk abstracts a bit too much. This is mainly due to my lack of experience writing them and that the majority of my current talk ideas cover large topics that are not as technical as I would like.

My favourite so far is one title “Python explains why your project failed”. This is a tongue in cheek talk which aims to poke fun at the Developer, PM and of course the client!

The TL;DR

This talk has yet to be accepted by anyone… but will be eventually I hope. In the mean time I wanted to share some of the funny thoughts and comparisons I’ve had coming up with the content for the talk. I plan on doing this by writing a series of blog posts one for each topic or sketch that features in the talk.

The Abstract

Python is fantastic! If you haven’t seen it you really need to. Its simple, elegant, powerful and gives you an amazing perspective on what we do as Developers, it is also hilariously funny…. Yes Funny!!!!

Wait! You thought I was talking about Python the programming language didn’t you? I’m sorry to tell you but we are actually talking about the most awesome of British comedy acts… Monty Python.

Throughout this talk I will take you through the development life-cycle of a project and use the Comedy of Monty Python to illustrate both the Good and the Bad (mainly the bad) aspects of our industry. All the way from Client introduction, requirements gathers, spec writing, team selection, planning and scoping all the way through Development to Testing, Delivery and Support!

The Delivery

This is a little harder to explain as I’ve not given the talk (yet) and I don’t think I could ever match the delivery better than the Pythons themselves.

However I have been known to dress for the occasion, so it’s quite possible that you may find me standing on stage in a red cassock at some point.

I’ve also managed to convince @phpcodemonkey, who is as big a Python fan as myself, that this talk should really be performed as a 2 man show, rather than me monologuing at a room full of people.

The Sketches

Due to the prolific variety of skits and sketches that Pythons created I found it extremely hard to select the few needed to fill a single talk. I have however managed to select a few and will change them around from time to time to suit the audience. I’ve provided a short list of a few of my favourites Sketches and a couple of words describing what they explain:

Spanish inquisition - Client Indecision
Dead parrot sketch - Stubborn Project Managers
Ministry of silly (array) walks - Tool Selection
Brian’s Latin Lesson - Planning and Preparation
We demand a shrubbery - Demanding the Impossible
Black knight - Solution fixation / Code Blindness
Camelot Song - Stakeholder Morale
The People’s Front - Team Fragmentation
The Silly Job Interview - Stakeholder Communications
Four Yorkshireman - Rockstar Developers
Argument clinic - Product Delivery
Architects sketch - Taking Shortcuts and Cutting Corners

The Finale

These are just a few of the potential topics I will be looking to cover in the coming posts, but while your reading them I want you to remember to…

https://youtu.be/WlBiLNN1NhQ

Are you a Good Code Scout? - NomadPHP lightning talk video

Wed, 06 Jan 2016 00:00:00 +0000

https://www.youtube.com/watch?v=Tt0lnauF5lI

Just before the Christmas period I was lucky enough to be able to give my “Are you a good Code Scout?” talk as a lightning talk for NomadPHP. Here is the video that was recorded from it.

Using Gmail aliases with Evolution

Wed, 06 Jan 2016 00:00:00 +0000

If your anything like me you have a large number of email aliases that you use with Gmail which is great. However I use Evolution as a mail client more often than not when using Gnome3 as a desktop.

It’s very easy to set up Evolution to create separate outbound email accounts that you can use for handling all of your aliases. It doesn’t yet support OAuth2 as an authentication mechanism for any account that is not set up using the built-in Gnome Online Accounts integration.

This is a real pain as Google have disabled the more common ‘plain’ and ’login’ authentication mechanisms for use with an SMTP only account. Meaning that any time that you try to connect to smtp.gmail.com:587 with STARTTLS you will get some form of error message to the effect of “Bad Authentication”.

Hopefully I’ll find a workaround at some point in the near future or Evolution will add the facility to enable OAuth2 as an available authentication mechanism.

In the mean time there is a workaround if you visit https://www.google.com/settings/security/lesssecureapps you can enable these less secure authentication mechanisms allowing you to once again connect and send email via email addresses using SMTP

A metaphor about PSR-7 and Middleware for non-developers

Thu, 08 Oct 2015 00:00:00 +0000

Never one to shy away from coming up with a metaphor for explaining something technical I found myself having to come up with one on the spot for PSR-7 and Middleware while at the recent PHPNW15 Conference.

Normally my brain will come up with something completely inappropriate but this time round I found I quite liked the imagery that came to mind.

If you would like to find out more of the specifics about PSR-7 you can take a look at http://www.php-fig.org/psr/psr-7/ which will make a better job of explaining it than I could ever do.

Now on to the metaphor

Imagine a house on fire, a bizarre way to start I know but bear with me. The nearest well with water that can put out the fire is 500 meters away! We then have a human chain stretching between the well and the house with a bucket going back and forth between trying to put the fire out. So lets break this down, the house represents the internet, or more specifically you and your browser. The fact you are on fire means that you are desperately needing water to quench the flames. At this point you send an empty bucket which represents your “request”, along the human chain, which in itself represents the application, to the well.

At the start of the chain the bucket is pretty normal, it’s a bucket of course, its round, made of wood with a rope handle, lets say it has a small leak in it.

As it travels down the chain it’s passed from person to person, everyone in it has the opportunity to do something with the bucket, or not as the case may be and could just pass it to the next person in the chain. Others may attempt to fix the leak in the bucket, someone may choose to replace it with a metal bucket, change the handle or make it bigger. Regardless of what may be done to the bucket in essence it remains a bucket.

Inexorably the bucket will continue to move down the chain to the well. When it reaches the well it changes state because now it has been filled with water. All of the interaction with the bucket thus far, mean that what happens at the well could vary depending on the changes have been made . If its been made bigger, for example, it could be filled with significantly more water, if swapped for a metal one it could imply that the bucket descends the well to get the water quicker because its heavier. Either way it is filled with water and begins its journey back towards the house.

Again it passes through the hands of each person in the chain, but now that its state has changed it now has the opportunity to be modified again. Someone may empty some water out as there is too much in the bucket, others may say that there is not enough and send it back down the line towards the well to be refilled. Either way the bucket continues to change hands over and over until it reaches the house and the contents are thrown on the fire to complete the request for water.

During this whole time the human chain could have been in flux. Some people may have swapped places, left the chain, added to the chain, some extraordinary people may have played leapfrog in the chain and appeared to handle the bucket more than once. Regardless of these changes the chain remains and continues to pass the bucket from one person to the another as long as the requests for water keep coming.

This, in the simplest possible form, explains PSR-7 and the concept of Middleware.

The bucket remains a bucket because PSR-7 says that is what is needed to complete the request for water, it also defines how you should interact with it regardless of what modifications have been made. If the bucket cant be used according to how PSR-7 describes a bucket to be, then the middleware can’t complete the request.

Every person in the human chain can be classed as a piece of middleware all the way from the house to the well and back again. If at any point someone enters the chain that doesn’t agree that the bucket is a bucket or doesn’t know how to handle it, then the it is dropped on the ground and the request fails.

Badges & Stickers

Sun, 04 Oct 2015 00:00:00 +0000

One of the most prominent things I’ve been asked about regarding my promoting being a Good Code Scout, is where can we get the badges?

Following on from a number of questions and subsequent tweets about it

https://twitter.com/stuherbert/status/650591775732670466

Well… I’ve decided that (providing I can get permission from all the right people) I’ll start creating a range of Badges & Stickers for you to earn as a Good Code Scout.

So if your interested in having some stickers or badges let me know using the form below and if I get enough interest I will most definitely get some made up for you.

Update

Unfortunately I’m have no more stickers left, Though I’ll be working on diversifying some of the designs in the near future I wont be looking at ordering any more for a little while yet. Keep your eyes on twitter as I’ll most likely post there when they are available again.

Wow... What a Conference

Sun, 04 Oct 2015 00:00:00 +0000

So I attended the PHPNW15 conference this weekend and what a weekend. I’ve been an attendee of the conference for a number of years and have always enjoyed it immensely. However this year turned out to be something special.

Following on from my first ever appearance as the PHP Boy Scout I decided to submit to the Unconference at PHPNW15.

It was a good talk, an extension of the previous lightening talk I’d given and felt really good to give. Unbeknownst to me however there was mischief afoot. Normally the Unconference talks are rated by the organisers and the one that they selected as the best gets to have a guaranteed slot in next years PHPNW conference. All of which I had genuinely either no idea about or had forgotten had happened in previous conferences,

As you may have guessed from the fact this post exists, I ended up winning that slot.

However…. it appeared that a Speaker had taken ill at the last minute and couldn’t make it at which point I was asked a mere 3 minutes before it was announce that I was also going to be given the hangover slot on track three for the Sunday sessions!!!!!

Suffice to say I had an interesting evening to say the least, in preparing my talk for “the big time”

Amazingly I felt really calm about everything, and even had a good chuckle about managing to find some props to help break the ice!

Everything is ready, I’ve practiced, knowing my talk was going to be a bit short… but that was ok considering the short notice, and I had plenty of anecdotes I could use as filler. I’m sat there waiting for the moment I have to put my head above the parapet and all of a sudden……..

nothing

My mind goes blank!

The long and the short is that I survived, and the feedback I have had has been amazing and I’ll be taking all of it on board to make sure that next time its even better!

The recordings should be available in the near future so when they are I will share a link so you can judge how it went for yourselves. In the mean time I’ve published the revised slide deck for you on slideshare.net/phpboyscout/are-you-a-good-scout-phpnw15-track-3

I’m hoping that I can now find some opportunities to practice for my slot at #phpnw16

[slideshare id=53508432&doc=areyouagoodscout-151004073451-lva1-app6892]

What is a PHP Scout

Fri, 02 Oct 2015 00:00:00 +0000

Recently I’ve had a lot of people asking me what a PHP Scout is! I thought it would be a good opportunity to explain.

To understand what a PHP Scout is it helps to know a little of the background basics of Scouting in general. Knowing this helps to make it easier later on as well as we draw some direct parallels. If you would like to investigate more about the history of Scouting you can find a good starting point at http://scouts.org.uk/about-us/history/.

For now I’m going to give a tl;dr version;

Scouting started in 1908 as a movement for training young people to encourage them to develop physically, mentally and spiritually by Robert Baden-Powell. Over the next 100+ years it has evolved to encompass people of all ages, races, colours and creeds to get involved and try to be the best they can be.

The primary ethos of the movement today is to bring Everyday Adventure to young people and this is achieved through a comprehensive programme scheme that is designed to touch on all aspects of that young persons development. This is then rewarded in a variety of ways with the primary reward being the experience itself, the awarding of badges also strengthens then sense of achievement and desire to work towards the next goal.

All members of the Scouting movement are required to make and frequently renew a promise:

On my honour, I promise that I will do my best to do my duty to {insert deity/monarchy here}, to help other people and to keep the Scout Law.

The key part here is I will do my best. Scouts are continually encouraged to improve themselves in everything they do.

The Boy Scout Rule

Lets start with something easy! There is a pretty common piece of guidance that gets bandied about in a lot of different circles that is normally referred to as “The Boy Scout Rule” which promotes leaving things better than you found it. It came about as common practice for scouts to always try to leave a campsite cleaner and tidier than when they arrived so that its in a good state for the next group.

This is quite generic but can easily be made very specific to us as programmers:

Leave the codebase better than you found it.

So what do I mean by this? Ultimately I mean that regardless of the state of the code you are working on you should always try to find a way to improve it.

This can be something as simple as;

refactoring the code to make it more readable
adding some docblock to explain a file/class/method/function/variable
create a Readme file or add some documentation
remove obsolete code, old backup files, stray files, unused components
fix a failing test
write a new test even

Its not an exhaustive list at all but it gives you an idea of what kind of things you can be doing to improve your codebase. Any good Scout group leaving a campsite would also make sure to put out the fire and close the gate on your way out. Which is exactly what you should be doing by making sure all your Acceptance, Functional, Integration & Unit tests pass and writing a good commit message.

Right tool for the job

In every activity that a Scout takes part in they are always taught the correct way to work with their tools and equipment, such as how to use an penknife property. They are then encouraged to explore different ways of using those to achieve their goals. This is no different for a PHP Scout, by knowing how to use their languages and tools properly they can then use it to maximum effect.

Self Development

As a child you assimilate massive amounts of information every day that helps you to grow and develop. This is creatively harnessed by Scouts through a variety of different activities that are designed to help them learn new skills that can help them grow as people.

Now that we are older our brains don’t have the same capacity to soak up that volume of information. But that doesn’t mean we shouldn’t be trying! A good PHP Scout will continually strive to push the boundaries of what they know, to pick up new skills that can be used to make them more capable. This can be learning a new technique, or language or tool be it via formal training, conferences, social events or even just a good Google.

Granted the Scouts are rewarded with some cool badges, but I’m sure its only a matter if time before some entrepreneurial PHP Scout decides to start creating some achievement badges of their own (see http://phpboyscout.uk/php-scout-membership-badge)

Helping Others

We’ve all heard the adage of a Scout helping someone across the street. It’s a somewhat stereotypical example but extremely apt as it highlights that they are encouraged to take into consideration other peoples needs and to provide assistance wherever possible. Modern Scouting however goes far beyond aiding with avoiding getting run over on a road.

By encouraging Scouts to not only help individuals, communities and groups we make them more considerate of the needs of others as well as developing their sense of self. A fantastic example of this is the 2015/16 initiative A Million Hands which promotes finding ways to identify the needs of others and to take action to provide aid.

This is a fantastic trait to be teaching children and is something that any good PHP Scout would applaud, and would then go forward to do the same things but with the development community. This can be something as simple as;

helping a colleague at work (without being told to do so by your boss)
organising an event with a local user group
contributing to an open source project

All very simple stuff to do and all it takes up is a little of your time! Where is the reward? I here some of you ask! I would say that the act itself is its own reward, and in reality that is true as when working helping others your generate some very positive Karma which will eventually be paid back when the day comes that you yourself need some help. You should (hopefully) also have an opportunity to maybe learn something new and improve your ability to communicate, a soft skill yes, but essential to your growth if your are striving to be better than you are now.

Problem Solving

The Scout motto is a very simple two words… “Be Prepared”, but be prepared to do what? Its quite open ended really it could be anything at all! I like to think that its nearly impossible to be equipped with every possible skill and tool possible to meet any and every task you will encounter through life, though being a Scout does try to help arm you with as many as possible.

Yet as a PHP Scout we should always “Be Prepared” to solve problems. If we are doing our jobs right we should be looking to solve problems through the solutions we provide every day. Quite often I talk to developers and hear them make pigeon holing statements like

I am a WordPress developer

And then complaining that they are bored at work or that they cant get jobs working with anything other than their chosen platform. Now this infuriates me as a PHP Scout would never do this, when asked they profess loud and clear

I am a Problem Solver

and be prepared to prove it by making sure they are prepared by knowing more than one or two platforms or frameworks or even programming languages. This can be encouraged by actively seeking “problems” that you can solve with tools and techniques you are not familiar with.

This is echoed throughout the challenges that are presented to Scouts, where they are tasked with solving a practical problem such as putting up a tent without any instructions, the best way to light a fire with two sticks and a bit of kindling, how to cross a stream with only a few bits of wood and rope. In solving these types of problems the Scouts not only receive the obvious of shelter, food, heat etc but they also become more prepared for the next time a similar scenario presents itself.

Team work

No man is an island as the saying goes and the same goes for being a Scout. By being organised into lodges,packs & patrols they have a ready made team to work with and the only way they can progress is to work together. They may not like the people in their team (and as a Scout Leader I will quite readily admit to putting Scouts into groups with others they may clash with).

We may all have teams that we work with as part of our Jobs, and a PHP Scout will take the opportunity to work with as many different combinations of teams as possible both in and outside of the workplace. By diversifying the people you have to interact with you develop a broader understanding about the problems you may be trying to solve.

This can then be expanded upon as mentioned previously by then branching out into the community and working with user groups and opensource projects.

To Summarise

The ethos behind the Scouting movement is a solid foundation not only for children aged 7-18 but for everyone. By being a PHP Scout you strive to keep improving your ability to create great code, solve problems, work with others and in doing so become a better developer.

As with all Scouts they are Hard Working, Determined, Ingenious & Tenacious and so is a PHP Scout.

The PHP Scout Membership Badge

Tue, 11 Aug 2015 00:00:00 +0000

As the PHP Boy Scout I’m having some badges made and I wanted to introduce the all new PHP Scout Membership Badge.

This badge shows that you are more than just a PHP Developer but also a good PHP Scout. This means that you have all the qualities it takes to be a PHP Scout and will :

always leave the codebase better than you found it
help other PHP developers be good Scouts
get involved with your local User Group & PHP Community
Contribute to at least one open source project

If you would like to find out how to get hold of a PHP Scout Membership Badge then fill out the form below.

[contact-form to='matt@phpboyscout.uk’ subject=‘Someone wants a Membership Badge’][contact-field label=‘Name’ type=‘name’ required=‘1’/][contact-field label=‘Email’ type=‘email’ required=‘1’/][contact-field label=‘I will always leave the codebase better than I found it’ type=‘checkbox’ required=‘1’/][contact-field label=‘I will help other PHP developers’ type=‘checkbox’ required=‘1’/][contact-field label=‘I will get involved with my local User Group’ type=‘checkbox’ required=‘1’/][contact-field label=‘I will contribute to an open source project’ type=‘checkbox’ required=‘1’/][/contact-form]

My first ever public appearance as PHPBoyScout

Thu, 06 Aug 2015 00:00:00 +0000

So it’s finally happened!

I stood up in front of a group of developers and gave a lightning talk about how Scouting Principles should be applied to every day development.

[slideshare id=51344270&doc=areyouagoodscout-150806121954-lva1-app6892]

The amazing thing is that I didn’t get any rotten tomatoes thrown at me! quite the contrary in fact. Even with me doing the talk in full Scout uniform.

Now to see about finding some more places to speak and actually fleshing out the talk into something that can last a full hour and not just shy of 5 minutes.

Goodbye Dev in Charge

Wed, 05 Aug 2015 00:00:00 +0000

Over all the time that I’ve been a developer I’ve had people telling me that I should get in front of an audience and speak. However I’ve always suffered from a rather bad case of ‘Imposter Syndrome’ which meant my automatic response to those kind of statements has always been… I don’t really know enough about any one topic.

This is very true, I’ve spent a lot of my career learning a really broad swathe of technologies and techniques so I can turn my hand to any task that’s been presented to me so far. Even so people continue to try convince me that it would be a worthwhile pursuit.

Now that I work at Magma Digital I find that I’m often talking with @phpcodemonkey about all sorts of things and the topic of creating a talk came up while we were enjoying the most excellent PHP South Coast Conference. He knows I’ve been a Scout Leader for around 4 years now, and he suggested that I do a talk on the ‘Boy Scout Rule’.

I don’t know if he was serious or not at the time but it set my mind racing! This is a topic that I actually know quite a lot about!

So I’m now going to leave behind the Dev in Charge and have now rebranded as the PHP Boy Scout. I’ve already managed to pull together the basis of a talk on how Scouting principles can be used in conjunction with what we do as Developers and have a few other ideas that I’m going to work on over the next few weeks.

Fingers crossed I will be better at doing this kind of thing than I suspect I will be… but nothing ventured and nothing gained!

Free Open Source Website for Scouts

Mon, 12 Jan 2015 00:00:00 +0000

I’ve been a Scout Leader for a few years now and the District I work within have very little by way of internet presence. As a bit of a pet project I started building a simple Scout based website for them to use.

Its nothing too fancy, I created a simple module and theme for the Silverstripe CMS and have now put it into a GitHub Repository to share with the wider scouting community.

I chose Silverstripe because of the speed with which I could develop something usable as well as providing a super simple management interface that can be handled by users of all skill levels.

The module itself extends some very common extensions available for the CMS and makes them scout focused. Features include.

Customisable theme
Multi tiered Event Calendars
Customisable Group/Section Pages
Dynamic Forms
A reliable News/Blog system

These are just a few of the most obvious features and hopefully I will continue to add more.

I’m also offering to help any Scout Groups/Districts/Counties if they are wanting to use these modules and get their sites built and up and running for them free of charge.

If you want to take a look at the code and have a play yourselves you can find it online at

If you want to get in touch or would like more information about having a website built for you please fill in the form below.

[contact-form to='matt@phpboyscout.uk’ subject=‘Request for Scouts Website’][contact-field label=‘Name’ type=‘name’ required=‘1’/][contact-field label=‘Email’ type=‘email’ required=‘1’/][contact-field label=‘Scout Group/District/County’ type=‘url’ required=‘1’/][/contact-form]

Its time for a change

Tue, 02 Dec 2014 00:00:00 +0000

So… Its been a long time since I posted anything of any relevance. This is due to having been super busy with my previous company Zucchi.

However that has all changed now! After three and a half years of running my own company I have decided that its not for me. I gave it my all, but in the end I was becoming too much of a Salesman and I missed getting stuck in with code.

I’ve now moved on and have joined the fantastic team at Magma Digital who have been leaders in PHP software development for somewhere in the region of 14 years as well as heavily involved in the PHP community having been a essential part of the PHPNW user group and conference.

This means I should be able to pick up where I left off all those years ago and start being more active again.

See you soon

Flexbox cross browser

Fri, 16 Aug 2013 00:00:00 +0000

Despite having been around for a while and having been through a couple of revisions, its support across browsers can vary greatly. From “Candidate Recommendation” on Chrome/Opera, “legacy flexbox” on Firefox and no support at all on IE9 and earlier.

Making flexbox work consistently across browsers was a challenge for us on a recent project, but I have found a solution that seems to work quite well.

Below is an SCSS @mixin that will attempt to handle compatibility between CR and legacy cross browsers flexbox.

@mixin flex($content: flex-start, $items: stretch, $direction: row, $wrap: wrap) {
 $packLegacy: $content;
 @if $packLegacy == flex-start {
 $packLegacy: start;
 } @else if $packLegacy == flex-end {
 $packLegacy: end;
 }

 $alignLegacy: $items;
 @if $alignLegacy ==flex-start {
 $alignLegacy: start;
 } @else if $alignLegacy == flex-end {
 $alignLegacy: end;
 }

 $oritentLegacy: $direction;
 $directionLegacy: normal;
 @if $oritentLegacy == row {
 $oritentLegacy: horizontal;
 } @else if $oritentLegacy == column {
 $oritentLegacy: vertical;
 }

/** SAFARI **/
 display: -webkit-box;
 -webkit-box-orient: $oritentLegacy;
 -webkit-box-pack: $packLegacy;
 -webkit-box-align: $alignLegacy;
/** FIREFOX LEGACY **/
 display: -moz-box;
 -moz-box-orient: $oritentLegacy;
 -moz-box-direction: $directionLegacy;
 -moz-box-pack: $packLegacy;
 -moz-box-align: $alignLegacy;
/** LEGACY **/
 display: box;
 box-orient: $oritentLegacy;
 box-direction: $directionLegacy;
 box-pack: $packLegacy;
 box-align: $alignLegacy;
/** IE 10+ **/
 display: -ms-flexbox;
 -ms-flex-wrap: $wrap;
 -ms-flex-direction: $direction;
 -ms-justify-content: $content;
 -ms-align-items: $items;
/** CHROME **/
 display: -webkit-flex;
 -webkit-flex-wrap: $wrap;
 -webkit-flex-direction: $direction;
 -webkit-justify-content: $content;
 -webkit-align-items: $items;
/** NATIVE **/
 display: flex;
 flex-wrap: $wrap;
 flex-direction: $direction;
 justify-content: $content;
 align-items: $items;
} //@mixin flex

@mixin flexItem($width) {
 -webkit-box-flex: $width;
 -moz-box-flex: $width;
 box-flex: $width;
 -ms-flex: $width;
 -webkit-flex: $width;
 flex: $width;

 min-height: 0;
}

Firefox however only half supports flexbox (all revisions) and to get around this I would recommend using Modernizr as this will add the class “no-flexbox” to the tag. This provides us with a simple work around that allows non flexbox supporting browsers render correctly by using specifically crafted and targeted CSS for non-flexbox browsers

I found that IE9 support could be implemented using the flexie javascript plugin. In IE8 Modernizr will add the class “no-flexboxlegacy” which can again allow you to create targeted CSS that wont affect your Flexbox layout.

For a great overview of the “CR” of flexbox, CSS Tricks has an amazingly comprehensive coverage of the functionality here http://css-tricks.com/snippets/css/a-guide-to-flexbox/

Creating Custom Routes in Silverstripe 3.1

Wed, 31 Jul 2013 00:00:00 +0000

We wanted to create a Route to our custom Products Controller in our products module for SilverStripe 3.1, such as: “http://www.examplesite.com/products/”

However looking at the Controller Documentation it was not clear how to create a route without an Action being supplied. In our example above the action is not specified, as we just want to use ‘view’.

Solution:

Create a /_config/routes.yml file containing the following:

---
Name: productsroutes
After: 'framework/routes#coreroutes'
---
Director:
 rules:
 'product': 'Product_Controller'
---

The above will redirect any Url that starts with “/product” to our Product_Controller. Note that everything after the rule, so after “/product”, is used in the next bit for matching.

Now we need to add private static $url_handers to Product_Controller to match our path, so in this example we need to match “$Slug!” which will match “”. Note the ! means the slug is required. Of course we want to direct this to a specific action, in this case “view”, this gives us:

private static $url_handlers = array(
 '$Slug!' => 'view',
);

Now just add “view” to the $allow_actions and add the “view” function. This gives the final Product_Controller as follows:

class Product_Controller extends Page_Controller
{
 private static $url_handlers = array(
 '$Slug!' => 'view',
 );

 private static $allowed_actions = array('view');

 public function view(SS_HTTPRequest $request)
 {
 // Your action code goes here
 return $this->render();
 }
}

Handy note:

You can put ?debug_request=1 on the end of your URL to see how it determines which Controller to use.

Disabling Cache in Silverstripe 3.1

Wed, 31 Jul 2013 00:00:00 +0000

While working with Silverstripe we found ourselves having to run “?flush=1” a lot to clear the Cache. To switch it off, while you work, add the following to your mysite/_config.php:

SS_Cache::set_cache_lifetime('default', -1, 100);

Set up SilverStripe 3.1 using only Git (No Composer)

Mon, 29 Jul 2013 00:00:00 +0000

We recently tried to use composer to set up SilverStripe 3.1, but ended up with a dependency nightmare. In order to work around this we decided to make use of Git submodules.

First set up your Git repository and run:

git init

Next set up a site directory for the code inside your Git repository. Then navigate to SilverStripe Installer in your browser and Download a copy. Extract files, and copy contents to site folder. Now we need to add the CMS and Framework. Navigate in a browser to the Git Hub repositories for CMS and Framework. Now copy the HTTPS clone URL for each project and run the following, to add these as Git sub modules.

git submodule add https://github.com/silverstripe/silverstripe-framework.git site/framework
git submodule add https://github.com/silverstripe/silverstripe-cms.git <path-to-site>site/cms

Now delete mysite/_config.php and load the site. Follow the normal install instructions displayed and you will have a running version of SilverStripe 3.1

Enabling MYSQL_CLIENT_INTERACTIVE with Doctrine 2 on Rackspace Cloud Database

Fri, 26 Jul 2013 00:00:00 +0000

We recently ran into problem using Doctrine 2 connecting to a Rackspace Cloud Database using the MySqli Driver.

Problem:

We have a long running PHP script that can sometimes run for hours at a time whilst processing information. This script requires a connection to a database, but has long periods of inactivity where there is no actual interaction with MySQL. By default MySQL uses the “wait_timeout” setting which states, how long an inactive connection can exist before it is killed. This is normally fine with web pages requests, as it is usually a short lived request. Unfortunately you do not have the ability to alter this setting when using Rackspaces Cloud Database.

Solution:

When using the MySQLi extension you can create a connection in “interactive mode” by passing the “MYSQLI_CLIENT_INTERACTIVE” flag, which will then use the “interactive_timeout” setting. On Rackspace this is set to 8 hours!

Annoyingly Doctrine does not allow you to pass any flags to the MySQLi Connection. So we overrode Doctrine\DBAL\Driver\Connection with our own Driver which then allows us to pass a “flags” parameter through.

Feel free to look at some of the other helpful features in we have added to Doctrine 2 here: ZucchiDoctrine

Installing PECL extensions for Zend Server 6

Mon, 13 May 2013 00:00:00 +0000

Recently we have revisited using Zend Server for some of our projects and decided to give the new version 6 a chance to prove itself.

Overall its a big improvement over version 5. There are still some things that are extremely annoying but we have decided that we can overlook them.

However there is one thing that we couldn’t do without. By default you will find that a number of PECL extensions will not install out of the box (at least this is what we experience using the Debian based install).

To fix this you will need to make sure you install the additional packages in ubuntu

php-5.4-source-zend-server or php-5.3-source-zend-server depending on the php version you are using
autoconf
build-essential

Once this is done you should now be able to install extensions from PECL without too much hassle.

Better Output for MySQL Select Command Using \G

Wed, 24 Apr 2013 00:00:00 +0000

If you ever find yourself using MySQL via command line and end up with something like this:

And thought there must be another way, well here it is: Use \G instead of ; at the end of your select command.

For example:

select * from CHARACTER_SETS\G

Below is an image of the output from this select:

Happy Querying!

Introducing ZuQ - A Simple ZeroMQ Queuing Daemon

Tue, 19 Mar 2013 00:00:00 +0000

We recently had the need to create a queuing system to replace an implementation of RabbitMQ that was being used on a previous project. The reasoning behind this is that the requirements of the project required a very custom implementation of a queuing system that would drastically alter in architecture as the project grew and RabbitMQ just wasn’t going to fit the bill. However to start with we required something super simple and efficient that could be expanded and developed as required. After a little investigation and a lot of recommendation from others we decided to use ZeroMQ as our transport layer for that very reason, as we could build something which could span across multiple servers and was fast.

The diagram above helps describe our basic queuing system. We have a queue daemon that is continuously listening for connections and two clients, one that populates the queue and the other that retrieves from the queue.

The Clients

Each client is written in PHP and uses a 0mq socket to communicate with a service, in this case our queue service. We used a SOCKET_REQ type of socket in order to have a request/response communication with our queue service.

function client_socket(\ZMQContext $context)
{
 // SOCKET_REQ used to create a client that sends requests to and receive from a service
 $client = new \ZMQSocket($context,\ZMQ::SOCKET_REQ);
 $client->connect("tcp://localhost:5555");
 // SOCKOPT_LINGER = 0 Configure socket to not wait at close time
 $client->setSockOpt(\ZMQ::SOCKOPT_LINGER, 0);
 return $client;
}

public function injectIntoQueue()
{
 $context = new \ZMQContext();
  $client = $this->client_socket($context);
  $msg = "This is a message";
  $retries_left = 3;
  $read = $write = array();
  while ($retries_left) {
    // We send a request, then we wait to get a reply
        $client->send($msg);
        $expect_reply = true;
        while ($expect_reply) {
        // Poll socket for a reply, with timeout
            $poll = new \ZMQPoll();
            $poll->add($client, \ZMQ::POLL_IN);
            $events = $poll->poll($read, $write, 2500);
            // If we got a reply, process it
            if ($events > 0) {
            // We got a reply from the server, must match sequence
                $reply = $client->recv();
                if (intval($reply) == $msg) {
                $retries_left = 0;
                    $expect_reply = false;
                }
             } elseif (--$retries_left == 0) {
            break;
             } else {
               // Old socket will be confused; close it and open a new one
                $client = $this->client_socket($context);
                // Send request again, on new socket
                $client->send($msg);
 }
 }
 }
}

You can see from the code above, have a 3 strike rule. The reasoning behind this is that if the client fails to connect to the queue service more than 3 times, we can stop trying to inject into the queue and move on to the next item. As we ultimately intend to adapt the lazy pirate pattern we have made it so that if the socket times out, we can then create a new socket and retry. Without this, as the architecture becomes more complicated we may then end up in a situation where we might have errors, thus the recommend solution is to create a new socket. Once the client has sent its message to the queue, we poll for a response (i.e. which is the message we sent returned back). Once we have a response that is valid, meaning that the queue has been populated, we can stop polling until the next message.

The Frontend Client

protected function getFromQueue()
{
 $context = new \ZMQContext();
 $worker = new \ZMQSocket($context, \ZMQ::SOCKET_REQ);
 $read = $write = array();
 // Set random identity to make tracing easier
 $worker->connect("tcp://localhost:5556");
 // Tell queue we're ready for work
 $worker->send("ready");
 $reply = $worker->recv();
 return $reply;
}

Our Frontend Client is much simpler as it is part of a process that is being continually updated, therefore it doesn’t need the same connection retries are the Backend Client. We simply send “ready” to the queue system and if the queue is populated it will return us the first item.

The Queuing Service

This is a continuously running executable created using c++.

zmq::context_t context(1);
zmq::socket_t frontend (context, ZMQ_ROUTER);
zmq::socket_t backend (context, ZMQ_ROUTER);
backend.bind("tcp://*:5555");
frontend.bind("tcp://*:5556");

We create two sockets of type ZMQ_ROUTER which is an advanced pattern used for extending request/reply sockets. This means when we improve our queuing system we will be able to route packets to specific recipients using an address in the message.

After creating our sockets, we initialise them

// Initialize poll set
zmq::pollitem_t items [] = {
 { frontend, 0, ZMQ_POLLIN, 0 },
 { backend, 0, ZMQ_POLLIN, 0 }
};

//poll the sockets - this seems to poll both sockets at the same time
zmq::poll (items, 2, -1);

Backend Handler

If we get a message from the backend, we check the contents to see if it contains purge at which point we empty the queue, otherwise we push the msg contents onto the queue. Finally we send the message back to the backend to show that the message has been received.

//receive msg from client
if (items [1].revents & ZMQ_POLLIN) {
 //get message from client
 zmq::message_t message(0);
 string client_addr = s_recv (backend);
 string empty = s_recv (backend);
 assert (empty.size() == 0);
 string msg = s_recv (backend);

 //allow the backend to purge the queue
 if(msg == "purge") {
 while (!queue.empty()) queue.pop();
 } else {
 queue.push(msg);
 }

 //send response back to the backend
 s_sendmore (backend, client_addr);
 s_sendmore (backend, "");
 s_send (backend, msg);
}

Frontend Handler

If we get the “ready” message from the frontend client, we pop a message off the queue and return it to the frontend client. If the queue is empty we send an “empty” message back instead.

// Handle activity on frontend
if (items [0].revents & ZMQ_POLLIN) {
 //get message from worker
 zmq::message_t message(0);
 string worker_addr = s_recv (frontend);
 string empty = s_recv (frontend);
 assert (empty.size() == 0);}
 string msg = s_recv (frontend); string queueMsg;
 if(msg == "ready") {
 if(queue.size() > 0) {
 queueMsg = queue.front();
 queue.pop();
 } else {
 queueMsg = "empty";
 }
 }
 //send reply to worker with contents of queue
 s_sendmore (frontend, worker_addr);
 s_sendmore (frontend, "");
 s_send (frontend, queueMsg);
}

That is our complete queuing system using ZeroMQ with PHP and C++.

Summary

Using the above has allowed us to create a very simple in memory queue daemon that we can use to quickly pass data from one system to a another. On the whole it works well and we are looking to expand on it in the near future to increase both its functionality and scalability.

You can find the queueing daemon (christened as “ZuQ”) on github @ https://github.com/zucchi/ZuQ

Glorious Gluster - How to setup GlusterFS on Rackspace Cloud and Ubuntu 12.10

Fri, 15 Mar 2013 00:00:00 +0000

A few of our projects recently called for a distributed file-system that provided high availability and redundancy. After a tip off from a fellow techie and a quick browse around the net it appeared that a solution called GlusterFS appeared to tick all the boxes for what we were wanting.

However setting it up turned out not to be as trivial as I had originally anticipated. I’m going to try and put down the process we have evolved for setting it up on Ubuntu in the cloud

A couple of things to clear up first.

We are using Rackspace for our cloud but beyond the setup of the servers it should still be relevant
There are a number of ways to interact with Rackspaces set up but for this we are going to use the cloud control panel
We use Ubuntu as our preferred server which means that our config tends to be all over the place compared to other guides
You will need to set up a minimum of 2 servers and a separate block storage device for each.
We have set up and broken a few different variations of gluster setup so far and make no guarantees that the setup in this blog is infallable but its the best wehave so far.

Setting up the hardware

First things first. We are going to need to set up are some servers.

Feel free to create any size server you want. Just make sure to select Ubuntu 12.10 (or whatever version you may have that is newer).

You will also need to define a new network to work with. We use this to isolate the traffic between the nodes of our new gluster.

You can create a new network when creating the first of your servers. On the creation page under the networks heading you can find a “Create Network” button.

Hopefully this should be quite self explanatory. Now when you create subsequent servers you will then have the option to attach your new network (“GlusterNet” in my example).

Once the two starting nodes have been created then you need to add some additional block storage to store your data on. Make sure that you create blocks that have sufficient capacity for your needs. Something else to consider is using High Performance SSD storage. Its a little on the pricy side but well worth the expense if you are trying to eak out every ounce of performance from the implementation.

You will then need to attach one to each of your servers.

Once attached you will be able to see the details of the block mount point from the block storage details page.

Make a note of the mount point (in this case “/dev/xvdb”) as we will need that in a minute.

Prepare the Server

Now that we have a the hardware ready we can shell into a server to set it up.

First you need to shell into your server and update its OS as the images provided by most cloud supplier tends not to have the latest patches and updates. In our case it’s as simple as:

apt-get update 
apt-get upgrade

Once that’s done we then need to prepare the Block Storage device ( henceforth refered to as a “brick”)

if you run fdisk -l you should see that an entry that looks something like

Disk /dev/xvdb: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdb doesn't contain a valid partition table

This indicates that our brick needs a partition table and formatting. We can achieve this be doing the following

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xe7da4288.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
 p primary (0 primary, 0 extended, 4 free)
 e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-209715199, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-209715199, default 209715199): 
Using default value 209715199

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

I’ve highlighted the prompts and my responses. All we are doing here is creating a default partition table that has a single partition which uses up the whole disk.

now running fdisk -l should give us something that looks like

Disk /dev/xvdb: 107.4 GB, 107374182400 bytes
43 heads, 44 sectors/track, 110843 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe7da4288

 Device Boot Start End Blocks Id System
/dev/xvdb1 2048 209715199 104856576 83 Linu

As you can now see we have a valid device of /dev/xvdb1 that we can mount_._ However we need to create a valid filesystem on the new brick before we can mount it. I have been doing this with Ext4 rather than XFS (which is the recommened filesystem from gluster), this is mainly down to the fact that when i tried using XFS I kept getting some issues with performance and access. I’m sure that with further investigation I could resolve this but as of yet haven’t had chance to. So far though I have had zero issues using Ext4. To create the filesystem we run:

mkfs.ext4 -j /dev/xvdb1

Next, create a folder to mount to, easily done by executing:

mkdir -p /glusterfs/brick

Finally, the simplest way to mount the device is via your /etc/fstab by adding the line

/dev/xvdb1 /glusterfs/brick ext4 defaults 1 2

and running mount -a as root (this will also mean that it mounts on boot for you automatically as well.)

Next we need to install the latest gluster version. At the time of writing this was v3.3.1. You can find a version to suit your OS at http://www.gluster.org/download. If you are using Ubuntu you can do the following

apt-get install software-properties-common
add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
apt-get update
apt-get install glusterfs-server glusterfs-client

By this point you will now have a single working server to continue on your going to need to set up your second server ready to create your new volume.

Once you have your second (or third, fourth, etc) setup its a good idea to add a reference to each one of them to your /etc/hosts file. This is not really necessary and you can just use the IP addresses of each server but it saves you having to remember each IP and makes it easier to identify.

Remember that we are going to be working with the new network interface you created earlier (i.e “GlusterNet”). to get the IP of your GlusterNet interface a quick ifconfig will show you an interface with an IP that matched the CIDR from earlier. In my case I now have 2 IPs of 192.168.3.1 & 192.168.3.2.

So now I add the following lines to my /etc/hosts file

192.168.3.1 gluster1
192.168.3.2 gluster2

Creating our volume

Now that the servers are prepared we can now play with the the tool gluster.This tool is a life saver in getting everything configured quickly and you can easily get a list of what its capable of by running gluster help. Now Im not going to take you through every command and option and would recomend reading the gluster manual to learn more.

What this tool actually does is help generate and manipulate all the required config that is then stored at /var/lib/glusterd/.

Firstly we need to tell gluster is that we have a pool of servers that will communicate with each other. Gluster refers to these as peers. To do this you need to run gluster peer probe gluster2 on each server for each server that will be used, replacing “gluster2” with the name names you defined in your /etc/hosts file. This will then create the appropriate files at /var/lib/glusterd/peers/

Now that all our peers have been defined we can get to actually creating the new distributed volume. This however requires a little consideration as there are some decisions you need to make.

If we take a look at the help for creating a new volume we can see that we need to decide on what options to use

volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>

- what are we going to name our volume
[stripe ] [replica ] - are we going to crate a striped or replicated volume and how many “bricks” are we going to create this volume with
[transport {tcp|rdma|tcp,rdma>] - What transport protocol do you want the peers to communicate with
- which servers/bricks do you want to use.

for more information on how to create you volume and what all the options mean have a look at these links

http://gluster.org/community/documentation/index.php/Gluster_3.2:_Configuring_Distributed_Replicated_Volumes

http://gluster.org/community/documentation/index.php/Gluster_3.2:_Configuring_Distributed_Striped_Volumes

for our purposes we are going to run

gluster volume create myvolume replica 2 transport tcp gluster1:/glusterfs/brick gluster2:/glusterfs/brick

This now creates a new volume that spans both of our servers. you can confirm that this is the case by running gluster volume info and you should get something that looks like

Volume Name: myvolume
Type: Replicate
Volume ID: d3dd24fd-9482-44c3-9503-24291fad8193
Status: Created
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/glusterfs/brick
Brick2: gluster2:/glusterfs/brick

running this on both servers should give you the same results.

What you will now find is that the gluster command has created a plethora of files at /var/lib/glusterd/vols/myvolume/. As you work with gluster more and more you will find yourself drawn to these files as they control all the different aspects of how the volume works and performs. Most importantly we will need some information from these files when we come to configure a client to mount the volume.

All that is left to do now is start the volume which can be easily done with a quick gluster volume start myvolume

At this point we have now completed setting up our volume but we need to add some security. I would strongly recommend setting up a firewall using ufw to control access to the server. The easiest way to do this is to allow all traffic on your “GlusterNet” network interface as only the servers you attach to that network will have access. you can find a guide to using ufw at https://help.ubuntu.com/12.10/serverguide/firewall.html.

Mounting a Client

Now that we have a working volume we need to add some clients. To do this you will need to create a new server as above that is attached to the “GlusterNet” network but without the block storage (unless you really want it that is).

Make sure to add your gluster dfinitions to your /etc/hosts file

Once you have your new client server ready we can install the gluster client

apt-get install software-properties-common
add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
apt-get update
apt-get install glusterfs-client

I’ve seen a number of different guides that tell you to install glusterfs-server as well but I have as yet had no need to as it all works without it.

Now there are a lot of way that you can mount your new Gluster volume. I have tried a few and have had varying results. What I have found is that the best way is to create a volume file. To do this we create a new file at /etc/glusterfs.vol.

volume gluster1
 type protocol/client
 option transport-type tcp
 option remote-host gluster1
 option remote-subvolume /glusterfs/brick
 option username <username>
 option password <password>
end-volume

volume gluster2
 type protocol/client
 option transport-type tcp
 option remote-host gluster2
 option remote-subvolume /glusterfs/brick
 option username <username>
 option password <password>
end-volume

volume replicate
 type cluster/replicate
 subvolumes gluster1 gluster2
end-volume

volume writebehind
 type performance/write-behind
 option cache-size 1MB
 subvolumes replicate
end-volume

volume cache
 type performance/io-cache
 option cache-size 400MB
 subvolumes writebehind
end-volume

What you will notice is that there is a and required for this to work. You can find these details on one of your peer servers in the file /var/lib/glusterd/vols/myvolume/trusted-myvolume-fuse.vol.

This /etc/gluster.vol file is basically going to inform the gluster-client software about how to connect to the gluster volume and all the available nodes to connect to. This provides us with some level of fail-over so should one node become unavailable the gluster client will seamlessly switch to a different one. It also allows us to define additional “translators” such as the performance-io one that you can see here. I would strongly recommend reading through the available translators to see which may be useful to you.

Now one of the main issues you will find with Ubuntu is that it will fail on boot if you try to add this mount to your fstab. To get around this you can use Upstart. if you create the following file at /etc/init/glusterfs-mount.conf making sure to change to the interface for your GlusterNet network (i.e. eth0 or eth1 or eth2, you get the idea)

author "Matt Cockayne"
description "Mount GlusterFS after networking available"

start on net-device-up IFACE=<interface>
stop on stopping network
stop on starting shutdown

script
    mount -t glusterfs /etc/glusterfs.vol /glusterfs
end script

As you can see we are using a straight mount command. The magic is that this will not be executed until the start clause validates which in this case is not until the network interface for “GlusterNet” is up and running properly. You will also see that we are mounting the /etc/gluster.vol file to /gluster (remember to create this folder to mount to) rather than mounting a network path as you might when mounting an NFS share.

If you wanted you could also add more to your upstart script to handle clean un-mounting of gluster thus allowing you to then use the service gluster-mount (start|stop|restart) commands

A quick reboot of the client server should confirm that it boots successfully and you will now end up with your volume mounted at /gluster. You can now test this by creating a new file. I tend to create an empty file at /gluster/mounted just so I have a quick reference that the folder is mounted. Once that’s created if you now go and take a look at the /gluster/brick on your “peers” you should see that there is now a file called “mounted” sat there looking all smug that it worked.

Caveats

Some important things for you to be made aware of

Never write directly to a brick. Make sure to write to the volume only through a configured client
Beware of split-brain. http://community.gluster.org/q/what-is-split-brain-in-glusterfs-and-how-can-i-cause-it/ http://www.gluster.org/2012/06/healing-split-brain/
RTFM - Read The F***ing Manual. Gluster is big and complex and there is a lot for you to understand. You can download a copy of the manual from here

Docblock, Oh Docblock, wherefore art thou Docblock (hint: Zend Optimizer Plus lost them)

Fri, 01 Mar 2013 00:00:00 +0000

tl;dr> I make a terrible assumption about Zend Optimizer+ and am corrected by Dominic in the comments;

Terrible post title I know but its the best I could come up with.

I’ve just come up for air after spending the majority of the day debugging some issues on our current development sandbox.

Now our sandbox tends to be quite bleeding edge in some circumstances and as such we run a fair few bits of unstable code. On the sandbox in question we have been running PHP 5.4.11 and unfortunately we have struggled to get APC working with it just the way we need it to. The lack of APC tends to make this sandbox quite slow.

We recently saw that Zend have open-sourced their OptimizerPlus extension (https://github.com/zend-dev/ZendOptimizerPlus) and that it was compatible with 5.4…. Fantastic, or so we thought.

So I added the new OptimiserPlus to the sandbox and everything was going swimmingly. That was until we had to run one of the utility scripts that we use to rebuild some of our data structures. These scripts make use of different parts of both Zend Framework and Doctrine which tend to rely on some heavy DocBlock annotations.

Now having used both APC and Zend Server knowing that they done affect this kind of functionality I had expected that OptimizerPlus would be fine…. Wrongo. It took me a good few hours of head scratching trying to figure out what had happened.

It turns out that OptimizerPlus suffers from the same flaws that eAccellerator does and strips Docblocks when caching the bytecode. This results in Reflection returning false when you call methods such as `getDocComment()`.

All in all its not the end of the world I just disable OptimizerPlus and have to wait till I can get APC working. Not my ideal scenario but I can live with it.

Something that does concern me is that there is currently an RFC that has gone to vote (https://wiki.php.net/rfc/optimizerplus) about integrating OptimizerPlus into the PHP 5.5 distribution. While this is great I do worry how many other things may break and will they be picked up and fixed for the 5.5 release.

Update: Since writing this post the RFC has finished being voted upon and has been approved. You can expect to see Optimizer Plus appearing bundled with PHP soon.

Update (15th Mar 13): Thanks to Dominics’ comment I now know that you can tell Optimizer+ to retain your Docblocks by setting your config using

zend_optimizerplus.save_comments (default "1")
	If disabled, all PHPDoc comments are dropped from the code to reduce the
	size of the optimized code. Disabling "Doc Comments" may break some
	existing applications and frameworks (e.g. Doctrine, ZF2, PHPUnit)

zend_optimizerplus.load_comments (default "1")
	If disabled, PHPDoc comments are not loaded from SHM, so "Doc Comments"
	may be always stored (save_comments=1), but not loaded by applications
	that don't need them anyway.

That’ll teach me to write a blog post without investigating more first.

Our Redmine install died, We all cried!

Sat, 23 Feb 2013 00:00:00 +0000

We have been using redmine for quite a long time and a few months ago attempted to upgrade from 1.3 to 2.something. Unfortunately I (quite typically) borked the installation and since then its been hobbling along after my attempts to fix it left it crippled.

Yesterday it finally gave up the fight and my attempts to resurrect the installation were futile. After a quick funeral (the eulogy was very touching), and wake in a nearby emporium of alcoholic beverages to commiserate our loss, I set about trying to figure out what to do next.

Alternatives

Now while Redmine is a worthy tool and has always managed to do what I needed in the past, recently its just not cut the mustard. I’ve kept toying with the idea of creating our own project management system but as with all in-house projects that we dream up its just never going to happen.

A quick google around our options are to either go for a hosted solution (not possible as we have some very specific requirements regarding our SCM that mean we have to host our own repos for client work) or Redmine (or chilli project).

Yes we looked at a number of other management tools and of them all Redmine is still the closes to what we needed.

Installation

So I spin up a new server instance of ubuntu 12.10 on the cloud and get to work installing the latest version.

As root I then run through these steps (you should assume that ALL of these steps require you to be root and files should be owned by root)

# update/upgrade base installation of ubuntu packages
apt-get update && apt-get upgrade

# install the requisite scm tools that we use
apt-get install git-core subversion mercurial cvs

# set up ruby
apt-get install ruby rubygems libruby ruby-dev

# set up apache & mysql
apt-get install apache2 libapache2-mod-passenger mysql-server mysql-client libmysqlclient-dev

# install imagemagick and the magick wand
apt-get install imagemagick libmagickcore-dev libmagickwand5 libmagickwand-dev

# create our user and database in mysql 
# replace uniquePassword with your own password
mysql -u root -p -e "create user 'redmine'@'localhost' identified by 'uniquePassword'"
mysql -u root -p -e "create database redmine"
mysql -u root -p -e "grant all on redmine.* to 'redmine'@'localhost'"
mysql -u root -p -e "flush privileges"

# clone redmine code to target location
cd /usr/local/share
git clone git://github.com/redmine/redmine.git

# set apache as the owner of redmine
chown -R www-data:www-data redmine

# move into our new redmine folder
cd redmine

# set up your database configuration
cp config/database.yml.example config/database.yml
vim config/database.yml

production:
 adapter: mysql2
 database: redmine
 host: localhost
 username: redmine
 password: uniquePassword

# install bundler gem
gem install bundler

# use bundler to set up redmine installation and without specified dependencies
bundle install --without development test postgresql sqlite

# set up our secret token
rake generate_secret_token

# set up our database and load default configuration
RAILS_ENV=production rake db:migrate
RAILS_ENV=production rake redmine:load_default_data

# edit /etc/apache2/sites-available/default
<VirtualHost *:80>
 ServerAdmin webmaster@localhost
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 DocumentRoot /usr/local/share/redmine/public
 <Directory />
 Options FollowSymLinks
 AllowOverride None
 </Directory>
 <Directory /usr/local/share/redmine/public>
 Options Indexes FollowSymLinks MultiViews
 AllowOverride All
 Order allow,deny
 allow from all
 </Directory>

 ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
 <Directory "/usr/lib/cgi-bin">
 AllowOverride None
 Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
 Order allow,deny
 Allow from all
 </Directory>

 ErrorLog ${APACHE_LOG_DIR}/error.log

 # Possible values include: debug, info, notice, warn, error, crit,
 # alert, emerg.
 LogLevel warn

 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

# restart apache 
service apache2 restart

That should be enough for you to have a working installation of redmine ready for you to use/customise

Additional Config

We typically have additional steps that we would configure for our own installation.

# add plugin assets folder
mkdir /usr/local/share/redmine/public/plugin_assets
chown www-data:www-data /usr/local/share/redmine/public/plugin_assets

# enable some additional apache modules
a2enmod rewrite

# disable mod ssl
a2dismod ssl

# install gnutls 
apt-get install libapache2-mod-gnutls

# install ssl certificate bundle and key (this assumes that you have already copied the key and bundle to ~/)
mv ~/my_certificate.bnd /etc/ssl/certs/my_certificate.bnd
chmod 0644 /etc/ssl/certs/my_certificate.bnd
mv ~/my_certificate.crt /etc/ssl/private/my_certificate.key
chmod 0600 /etc/ssl/private/my_certificate.key

# now configure your /etc/apache2/sites-available/default-tls
<IfModule mod_gnutls.c>
<VirtualHost _default_:443>
 ServerAdmin webmaster@localhost
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 DocumentRoot /usr/local/share/redmine/public
 <Directory />
 Options FollowSymLinks
 AllowOverride None
 </Directory>
 <Directory /usr/local/share/redmine/public>
 Options Indexes FollowSymLinks MultiViews
 AllowOverride All
 Order allow,deny
 allow from all
 </Directory>

 ErrorLog ${APACHE_LOG_DIR}/error.log
 # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
 LogLevel warn
 CustomLog ${APACHE_LOG_DIR}/ssl_access.log combined

 GnuTLSEnable On
 GnuTLSCertificateFile /etc/ssl/certs/my_certificate.bnd
 GnuTLSKeyFile /etc/ssl/private/my_certificate.key
 GnuTLSPriorities NORMAL:!DHE-RSA:!DHE-DSS:!AES-256-CBC:%COMPAT
</VirtualHost>
</IfModule>

# Add some Rails / Passenger specific config to /etc/apache2/sites-available/default-tls
RailsEnv production
PassengerDefaultUser www-data
PassengerSpawnMethod smart
PassengerPoolIdleTime 300
PassengerMaxRequests 5000
PassengerStatThrottleRate 5
PassengerHighPerformance On

# change your /etc/apache2/sites-available/default to redirect to ssl
<VirtualHost *:80>
 ServerAdmin sysadmin@zucchi.co.uk
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 RewriteEngine On
 RewriteCond %{HTTPS} off
 RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
 Options FollowSymLinks
 AllowOverride None

 ErrorLog ${APACHE_LOG_DIR}/error.log
 LogLevel warn
 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

# enable your new default-tls vhost and restart apache
a2ensite default-tls
service apache2 restart

# setup &amp; configure email
# when prompted select "internet site" and enter the domain you are hosting redmine from i.e. mysite.co.uk)
apt-get install postfix

# create config file and uncomment the production settings for sendmail
cp /usr/local/share/redmine/config/configuration.yml.example /usr/local/share/redmine/config/configuration.yml
vim /usr/local/share/redmine/config/configuration.yml

production:
 email_delivery:
 delivery_method: :sendmail

service apache2 restart

#install pixel cookers theme cos we like it
git clone git://github.com/pixel-cookers/RedmineThemePixelCookers.git /usr/local/share/redmine/public/themes/pixel-cookers

Debug PHP CLI on Remote Server with Xdebug and PHPStorm

Wed, 06 Feb 2013 00:00:00 +0000

This was a head scratcher when I ran into this yesterday and I thought I would share my solution to the following scenario:

I need to debug PHP Command Line script, located on Remote LAMP Virtual WebServer running in Virtual Box with a Shared Folder, using local PHPStorm 5.0.

The solution:

You first must set PHPStorm to use remote file paths. To set these go to the following:

PHPStorm -> Peferences -> PHP -> Servers

This gives the following display:

Replace the Name, Host and Absolute path on the server, to match your own settings. Note keep the Name and Host the same for ease.

Next add some breakpoints in PHPStorm and set it to listen for any debug connections using the listener icon:

Now login to your Remote Server via SSH etc.

You now need to change settings for Xdebug in either xdebug.ini or php.ini depending on how you installed it. You also need to know the IP of the local machine. This can permanently set in the Network Setting of your VM in Virtual Box, so you will never have to change it. In my example the local machine running PHPStorm is:

192.168.56.1

Now edit the ini file that contains your Xdebug settings and set the following:

xdebug.remote_host = 192.168.56.1
xdebug.remote_connect_back = 0
xdebug.remote_port = 9000
xdebug.remote_handler = dbgp
xdebug.remote_mode = req
xdebug.remote_enable = 1
xdebug.idekey = phpstorm1

Be aware you might have to change the remote_host and the idekey based on your own environment. To better understand what each option does, see Xdebug Settings

Finally, when running the script you must set the following variables:

PHP_IDE_CONFIG="serverName=dev.example.com"

PHP_IDE_CONFIG will tell PHPStorm how to map the Remote File Paths to what it sees Locally. Again replace the URL with the Name/Host you set in PHPStorm. Note: You can export this, if your system is only running one site; mine is not.

You can run this inline with your script:

PHP_IDE_CONFIG="serverName=dev.example.com" ./testscript.sh

This should send you to PHPStorm where you earlier placed breakpoints.

Happy Debugging!

Quick and easy setup of and connection to NRPE on Ubuntu

Wed, 06 Feb 2013 00:00:00 +0000

About NRPE

NRPE (Nagios Remote Plugin Executor) is a useful tool that allows you to execute scripts on remote servers and return the output for ingestion by some form of monitoring software.

Setup

We currently have our own instance of Icinga running to monitor our servers and have recently started to offer access to it for our clients.

The majority of our servers (and our clients servers if we set them up) use one variant or another of Ubuntu. This means we can very quickly get our servers connected to a Nagios/Icinga instance.

First things first we need to install the nrpe server and all the associated plugins

apt-get install nagios-nrpe-server \
nagios-plugins-basic \
nagios-plugins \
nagios-plugins-extra

Next we need to edit the main nrpe config file to be found @ /etc/nagios/nrpe.cfg. What your looking for is the lines

# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames 
# that are allowed to talk to the NRPE daemon.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address. I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1

# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed. This option only works
# if the daemon was configured with the --enable-command-args configure script
# option. 
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! *** 
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=0

You will want to change this to the IP of your Nagios/Icinga instance and set the dont_blame_nrpe value to 1. Feel free to take a look round the rest of the file. Its all quite interesting and generally will documented. Be careful what you change though in case something breaks.

You will also want to look for some lines that are refererd to as “COMMAND DEFINITIONS” and look something like this

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

You can go ahead and comment these out as we will be adding our own definitions shortly. The main reason for removing these is that we will be configuring some specific scripts for our own use later that allow you to configure your requirements and thereshold from within your Nagios/Icinga config.

Configuration of Monitoring Server

Once this is complete you can now configure a new “check command” for use with your nagios/icinga server.

define command {
 command_name check_nrpe
 command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

define command {
 command_name check_nrpe_command_args
 command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ -a $ARG2$
}

Here you can see that we have set up 2 different check commands. The first is a simple command requiring only one argument of $ARG1$ which would be the name of the command we want to run on the remote server. The second command is almost identical except for the fact it takes a second argument which allows you to input a series of “arguments” to be passed to the command on your remote server. each argument should be separated by a space.

Now that you have these you can then configure your hosts and services to make use of it. I would recommend having a trawl through the Nagios/Icinga sites & documentation to find out how to create a config that suits you.

Configuration of Remote Server

Now that we have our monitoring server ready its time to add the command we want to run to the remote server.

To do this your /etc/nagios/nrpe.cfg shoudl hopefully have a line in it that looks like

include=/etc/nagios/nrpe_local.cfg

if it doesn’t have a line like that then add it and edit the `/etc/nagios/nrpe_local.cfg` file to look a little like this

command[check_apt]=/usr/lib/nagios/plugins/check_apt
command[check_users]=/usr/lib/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
command[check_load]=/usr/lib/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p /dev/sda1
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

These are a few simple commands that I tend to use most often. These translate to your “check_nrpe” commands like so

$ARG1$ = everything inside the square brackets [ ]
$ARG2$ = each of the $ARG?$ keys as a single string separated by a space

Once that’s done you should be able restart your nrpe server with `/etc/init.d/nagios-nrpe-server restart`

It really is that simple. Do bear in mind that because you can pass arbitrary arguments into nrpe this was you could leave yourself vulnerable to a bit of maliciousness so its a good idea to make sure your firewall restricts port 5666 (the default port) to IPs you trust.

Compiling Apache 2.4 on Ubuntu 12.04

Tue, 06 Nov 2012 00:00:00 +0000

I’ve decided that I need to up my game when it comes to webservers. However I’m not yet ready to switch to Nginx or one of the other webservers out in the wild as I need something up and running rapidly.

Granted the numbers are definitely against Apache in a lot of benchmarks but historically I’ve always had a good experience and the entry level makes it much more appropriate for me to stick with it.

However Apache 2.2 is rather long in the tooth, thankfully 2.4 has been out for a while now. The problem I have is that I tend to favour Ubuntu as a platform and there is no sign of a 2.4 version appearing on the horizon anytime soon as they are waiting for it to be implemented upsteam in Debian before including it in Ubuntu.

Now there are PPAs available out there but im not overly happy using them (especially on production environments) So the only option is to compile.

First thing is to install all the dependencies we are going to need. Thankfuly ubuntu has a nice and simple way of handling this.

apt-get build-dep apache2

We can then download the source code and start the compilation.

So from the root of our new copy of the source we need to run our configure.

./configure --prefix=/usr/local/apache2 \
 --enable-mods-shared=all \
 --enable-http \
 --enable-deflate \
 --enable-expires \
 --enable-slotmem-shm \
 --enable-headers \
 --enable-rewrite \
 --enable-proxy \
 --enable-proxy-balancer \
 --enable-proxy-http \
 --enable-proxy-fcgi \
 --enable-mime-magic \
 --enable-log-debug \
 --with-mpm=event

You will notice that I’m installing it using the event mpm. Hopefully I’ll be covering more about the event mpm in the future.

Next we need to run make

make && make install

Once that’s complete you should be able to run

/usr/local/apache2/bin/apachectl start

and get the “it works” message through your webrowser when accessing the server IP.

Dont forget to configure apache to suit your specific requirements.

Something that will come up is how to start apache on boot. Seeing as Ubuntu uses Upstart it makes sense to utilise it for controlling apache.

So in the file `/etc/ini/apache.conf` we need to put

# apache2 - http server
#
# Apache is a web server that responds to HTTP and HTTPS requests.
# Required-Start: $local_fs $remote_fs $network $syslog
# Required-Stop: $local_fs $remote_fs $network $syslog

author "Matt Cockayne <matt@zucchi.co.uk"
description "Apache 2.4 HTTP Server"

start on runlevel [2345]
stop on runlevel [!2345]

console output

pre-start script
 mkdir -p /var/run/apache2 || true
 install -d -o www-data /var/lock/apache2 || true
 # ssl_scache shouldn't be here if we're just starting up.
 # (this is bad if there are several apache2 instances running)
 rm -f /var/run/apache2/*ssl_scache* || true
end script

# Give up if restart occurs 10 times in 30 seconds.
respawn limit 10 30
respawn

script
 if test -f /usr/local/apache2/bin/envvars; then
 . /usr/local/apache2/bin/envvars
 fi
 ULIMIT_MAX_FILES="ulimit -S -n `ulimit -H -n`"
 if [ "x$ULIMIT_MAX_FILES" != "x" ] ; then
 $ULIMIT_MAX_FILES
 fi

 /usr/local/apache2/bin/httpd -k start -D FOREGROUND
end script

This is a rather simple upstart script and I will be looking to update it at some point… but it works

Once that’s done you should find that on reboot Apache will start and take advantage of all the management features of upstart including attempting to respawn Apache should it end unexpectedly. You should also be able to then use the following commands to control Apache.

# how to start start apache
start apache
# or 
initctl start apache

# how to stop apache
stop apache
# or 
initctl stop apache

# how to restart apache
restart apache 
# or 
initctl restart apache

# check the status of apache 
status apache
# or
initctl status apache

I generally tend to avoid using the apachectl script found at /usr/local/apache/bin/apachectl once upstart takes control.

Compiling PHP 5.4 on Ubuntu 12.04

Tue, 06 Nov 2012 00:00:00 +0000

So recently I’ve been working with PHP 5.4 a LOT. Unfortunately Ubuntu (my main dev environment) is behind the times. So I’m resorting to compiling PHP manually.

Not a daunting as it may first appear. The really tricky part is working out your dependencies and `configure` script.

Hence the reason for this post as a reminder for myself and others that may want to do a quick compile. (I would recommend that if your compiling for a production/live environment that you make sure you understand what it is your compiling though before just using what’s here)

So where to start. Dependencies first I think

Ubuntu allows you to install dependencies for building source `apt-get build-deps`. We will use this and install any extras we may need.

apt-get install \
libxml2 \
libxml2-dev \
libssl-dev \
pkg-config \
curl \
libcurl4-nss-dev \
enchant \
libenchant-dev \
libjpeg8 \
libjpeg8-dev \
libpng12-0 \
libpng12-dev \
libvpx1 \
libvpx-dev \
libfreetype6 \
libfreetype6-dev \
libt1-5 \
libt1-dev \
libgmp10 \
libgmp-dev \
libicu48 \
libicu-dev \
mcrypt \
libmcrypt4 \
libmcrypt-dev \
libpspell-dev \
libedit2 \
libedit-dev \
libsnmp15 \
libsnmp-dev \
libxslt1.1 \
libxslt1-dev

And now the configure

./configure \
--prefix=/usr/local/php \
--with-apxs2=/usr/local/apache2/bin/apxs \
--enable-fpm \
--with-fpm-user=www-data \
--with-fpm-group=www-data \
--with-config-file-path=/usr/local/php/conf \
--with-config-file-scan-dir=/usr/local/php/conf.d \
--enable-debug \
--with-openssl \
--with-kerberos \
--with-zlib \
--enable-calendar \
--with-curl \
--with-curlwrappers \
--with-enchant \
--enable-exif \
--enable-ftp \
--with-gd \
--with-jpeg-dir=/usr \
--with-png-dir=/usr \
--with-vpx-dir=/usr \
--with-freetype-dir=/usr \
--with-t1lib \
--enable-exif \
--enable-gd-native-ttf \
--enable-gd-jis-conv \
--with-gettext \
--with-gmp \
--with-mhash \
--enable-intl \
--enable-mbstring \
--with-mcrypt \
--with-mysql \
--with-mysqli \
--enable-pcntl \
--with-pdo-mysql \
--with-pdo-pgsql \
--with-pgsql \
--with-pspell \
--with-libedit \
--with-readline \
--enable-shmop \
--with-snmp \
--enable-soap \
--enable-sockets \
--enable-sysvmsg \
--enable-sysvshm \
--with-xsl \
--enable-zip \
--with-pear \
--enable-zend-signals \
--enable-maintainer-zts

Once these are done then we follow the standard make process. Notice we are also running make test… very important as it givges more data for the developers to work with.

make && make test && make install

The next thing is configuring your php.ini file as the install doesn’t have one yet so we copy either the production or development default from the source code to the new conf dir and edit to suit your needs.

cp {php-source-dir}/php.ini-(development|production) /usr/local/php/conf

Thats it. All ready to roll… almost, this installation is the one I use for use with a webserver so you will want to add the appropriate directives to apache.

LoadModule php5_module modules/libphp5.so

AddHandler php5-script .php
AddType text/html .php

Rsync and custom SSH commands

Tue, 31 Jul 2012 00:00:00 +0000

Rsync is a great tool but can be a pain if you have to jump through hoops to connect via ssh such as connecting via a different port.

A simple solution is to use the -e flag (also knows as –rsh=COMMAND). This flag allows you manually define the ssh command to use when connecting

rsync -e 'ssh -p2020' -rav ./* user@server:

Will allow me to connect to a server with SSH listening on port 2020

Nice New Nexus7

Thu, 19 Jul 2012 00:00:00 +0000

This morning I woke up to an email telling me that my Nexus7 that I had ordered 3 weeks ago was… “out for delivery”.

I couldn’t contain my excitement. I sat patiently waiting by my door. Finally 11 o’clock rolls around and there is a knock. I’m handed a brown parcel and hand over the obligatory signature. I close the door behind me and carefully place the box on the desk. I contemplate teasing myself and seeing how long I can hold out before opening it.

That lasted about 20 seconds!!!!

In fact… this video says it all

http://www.youtube.com/watch?v=Xijcwbg8CGQ

I’m not gonna bore you with how awesome it is (and it is awesome). I will however point out a few obvious foibles with it though (not that they would have ever stopped me from buying it).

No way to expand storage
Lack of Flash (may not seem important, but until everyone else catches up there is loads of content I cant use i.e. BBC iPlayer)
A number of apps (games specifically) that I have run on my phone are not yet supported
Google Now feels a little clunky at times and struggles with some of the regional British accents
MTP doesn’t appear to work out of the box with my Linux OS (I’m sure this will be remedied soon)
No obvious way to directly access the front facing camera (easily remedied with an app from MoDaCo)

Registering custom view helpers in ZF2

Wed, 18 Jul 2012 00:00:00 +0000

If you want to register custom view helpers with a module you can do so by using the service location built into the Skeleton Application and creating a module config that looks something like.

return array(
 'view_helpers' => array(
 'invokables' => array(
 // generic view helpers
 'truncate' => 'Zucchi\View\Helper\Truncate',

 // form based view helpers
 'bootstrapForm' => 'Zucchi\Form\View\Helper\BootstrapForm',
 'bootstrapRow' => 'Zucchi\Form\View\Helper\BootstrapRow',
 'bootstrapCollection' => 'Zucchi\Form\View\Helper\BootstrapCollection',
 ),
 ),
);

Bootstrapping ZF2 Forms

Tue, 17 Jul 2012 00:00:00 +0000

So…

With the release of beta 5 for Zend Framework 2 I thought it time for me to tidy up and fix a few modules I created back at beta 3.

Now I’m a big fan of Twitter Bootstrap CSS framework as I’m sure a lot of other people are as well. Seeing that the Zend Skeleton Application comes with bootstrap already included it was easy enough to set up my forms using the old ZF Forms found in ZF1.

However a brand spanking new Forms component has been rolled out with ZF2. The long and the short of this new component meant that I had the opportunity to hand roll a new way of making my forms work with Twitter Bootstrap.

So, a little tinkering, a quick pull request to ZF2 to allow the definition of arbitrary options and I came up with some useful View Helpers that can be dropped into a project and used.

You can find them at https://github.com/zucchi/Zucchi.

So how to use them. Lets start by creating a new form (we’ll keep it simple for now)

class MyForm extends Form
{
 public function __construct()
 {
 parent::__construct('myform');
 $this->add(array(
 'name' => 'price',
 'attributes' => array(
 'type' => 'text',
 'required' => 'required',
 'placeholder' => '0.99',
 ),
 'options' => array(
 'label'
 'bootstrap' => array(
 'help' => array(
 'style' => 'block'
 'content' => 'The price you wish to use'
 ),
 'prepend' => array('$'),
 'append' => array('¢'),
 ),
 ),
 );

 $actions = new Collection('actions');
 $actions->setAttribute('class', 'form-actions');

 $actions->add(array(
 'name' => 'submit',
 'attributes' => array(
 'type' => 'submit',
 'value' => 'Save',
 'class' => 'btn btn-primary'
 ),
 'options' => array(
 'bootstrap' => array(
 'style' => 'inline',
 ),
 ),
 ));

 $actions->add(array(
 'name' => 'reset',
 'attributes' => array(
 'type' => 'reset',
 'value' => 'reset',
 'class' => 'btn'
 ),
 'options' => array(
 'bootstrap' => array(
 'style' => 'inline',
 ),
 ),
 ));
 $this->add($actions);
 }
}

You’ll notice that I have highlighted some lines. Thanks to the ability to set arbitrary options we can define a “bootstrap” option which we can then use to allow us to pass data into our new bootstrap view helpers. You can also see that I have added a save and reset button to a collection. I’ll explain that later.

So what next… Rather than go into the mechanics of how to work with forms I’ll refer you to the ZF documentation and this excellent blog post

We then pick up by looking at your view, and the helpers I have created.

BootstrapForm($form, $formStyle)

One of the few things I miss from the ZF1 implementation of Forms is the self rendering aspect! So what did I decide to do? That’s right I created a view helper to render everything in one command.

The $this->bootstrapForm() takes two parameters. The first is quite obviously the form. The second is the style of form. This is directly related to the form types that can be found http://twitter.github.com/bootstrap/base-css.html#forms. You can use any of ‘vertical’, ‘inline’, ‘search’ & ‘horizontal’. If you dont specify a formStyle then it will default to ‘vertical’

Caveat: This helper will then iterate through all of the associated elements and render them first. Only after the direct elements have been generated will it then move onto Collections or Fieldsets (as soon as I work out how I’ll fix this).

BootstrapRow($element, $formStyle)

This is a straightforward modification of the FormRow helper that come bundled with the new component.

We have a few differences now though. We have a second parameter as with the BootstrapForm view helper and the output is generated using sprintf and a set of templates that mimic the structures of the different form styles from bootstrap.

This helper can be used by itself to generate an element row and is used by the BootstrapForm helper

We can also now take advantage of the “bootstrap” options we set earlier.

Bootstrap Options

style

The style of form element to use regardless of what style may be passed into the view helper (you can see an example of this in the buttons from the MyForm example above)

help

This works in the same way as “description” did from ZF1 but allows you to define it either as a string or an array with the keys “style” for either ‘inline’ or ‘block’ and “Content” which should be self explainatory

prepend

Takes advantage of Bootstraps ability to prepend blocks to an input field. This can be defined as a single string, or an array of strings to allow you to add multiple blocks should you want to

prepend

Takes advantage of Bootstraps ability to append blocks to an input field. This can be defined as a single string, or an array of strings to allow you to add multiple blocks should you want to

These options get evaluated and spat out from the new renderBootstrapOptions() method as part of the “render”.

BootstrapCollection($element, $style, $wrap)

Again this is a direct rip off of the FormCollection helper found in the ZF2 Form component witha few modifications. The main difference is that is makes use of the BootstrapRow helper and has methods and properties to allow the setting of the form style to use.

You can see from the MyForm example above that we set a Collection called ‘actions’. This is a pretty standard way of grouping elements together. You can also see that we set a class for the Collection which may look familiar to those that have used Twitter Bootstrap for a while.

What our helper will then do is wrap the buttons in a div with the appropriate class attached. If you were to define a label for the Collection/Fieldset You would then also find that the fieldset and legend tags are also spat out with our <div class="form-actions"> sandwiched between them and the elements.

Result

So what we now get when we use MyForm with out helpers.

$this->bootstrapForm($form, 'horizontal');

Should now look something like this

How you can use it

As of right now you can get the library from its repo on github @ https://github.com/zucchi/Zucchi and can be found on packagist for use with composer

Edit: The bootstrap stuff has moved to a new location as a separate ZF2 module. you can find it @ https://github.com/zucchi/ZucchiBootstrap or @ packagist for use with composer

Loaded Testing

Sat, 30 Jun 2012 00:00:00 +0000

I recently had to do some load testing for a site recently that would allow me to test in excess of 100k requests in a 60 second period…

So I decided to do some testing using JMeter as it seemed like a suitable tool for doing what I needed and I had used it for some simpler testing in the past.

After a little fumbling around I managed to get a test plan designed that would simulate 10k users actually navigating the site and adding to a cart etc, with a number of various interactions. It wasnt perfect but it would correctly simulate over 100k requests.

So feeling quite pleased with myself I started the test from my laptop. Now I’m not a big gamer, I’m known to play a little World or Warcraft from time to time but that’s about it. So when it comes to computing power i tend to opt for battery life over sheer grunt.

Suffice to say, my laptop fell flat on its face, and if it hadn’t it turns out that the connection I was using just wasn’t up to the task of handling that much traffic adequately.

So plan B…

I quickly fired up the largest AWS instance available and got a copy of jmeter installed. A little tinkering with my test plan and some googling on how to run jmeter without a gui and a quick

./jmeter -n -t test-plan.jmx

and it appeared to be running.

(Please bear in mind that I’m being overly kind… it took a LOT of tinkering and twice as much Googling to work out how to get the test results out so i could actually get some idea of WTF was happening during the test)

So… client “happy”… I decided to go and find a better way to do my load testing in the future.

Sticking with JMeter I managed to find this gem of a page

http://jmeter.apache.org/usermanual/remote-test.html

tl;dr > use your local install of jmeter to trigger tests to run on one or more remote “nodes” and then have all the results sent to your local install.

So I set to work!

Building a Node

First I need to set up an AWS instance that we can use and duplicate so I can quickly build a cluster of nodes on demand. I’m a big fan of Ubuntu so I spin up a micro instance of 12.04 server. Next I shell into the instance and install the default Java runtime from apt

apt-get install openjdk-7-jre

Yes I know there are other more appropriate runtimes, but i dont really care… i just need it to work and it does.

next I grab a copy of the latest stable from http://jmeter.apache.org/download_jmeter.cgi and un-tar it to /usr/local/jmeter

(N.B. JMeter is available through the apt but I had issues with that version and you need to make sure that both your local version and all the nodes run the same version of jmeter)

We can now test that the install is working running /usr/local/jmeter/bin/jmeter-server and you should get some output that looks similar to

Created remote object: UnicastServerRef [liveRef: [endpoint:[10.???.???.???:38939](local),objID:[46522b57:138381f1023:-7fff, 2635011707874933136]]]

Which tells us that the server is running.

BUT unfortunately its not going to work just yet. Because we are using Amazons EC2 we are going to relying on their NAT for routing. Out of the box JMeter just wont work properly.

However there is something we can do to combat this. We can set the parameter RMI_HOST_DEF that the /usr/local/jmeter/bin/jmeter-server script will include in starting the server.

export RMI_HOST_DEF=-Djava.rmi.server.hostname=$(wget http://169.254.169.254/latest/meta-data/public-hostname -q -O -)

I’ll explain what we are doing here. Amazon have been quite clever by providing a meta-data endpoint that you can poll from within your instance to get key pieces of data… Including the public dns record.

We can use this endpoint and using wget pipe that into the RMI_HOST_DEF param (ensuring that we prepend -D) and then export that so it becomes available to the /usr/local/jmeter/bin/jmeter-server script.

Now to get the server to start on boot.

a quick upstart script should solve this

# Upstart script to initialise jmeter-server

description "JMeter Server"
author "Dev in Charge "

start on started networking
stop on stopping networking
stop on stopping shutdown

console output

script
 # get the current public DNS record
 export RMI_HOST_DEF=-Djava.rmi.server.hostname=$(wget http://169.254.169.254/latest/meta-data/public-hostname -q -O -)

 # start jmeter in server mde
 /usr/local/jmeter/bin/jmeter-server
end script

saving this to /etc/init/jmeter-server.conf will mean that it will auto-start jmeter-server on boot and allow you to manually control the process using start jmeter-server and stop jmeter-server

and thats it… instance configured

All you need to do now is save the instance as an AMI and you have an on-demand image for spinning up a cluster of remote JMeter servers for you to play with.

Configuring your local installation

Now that the server side is working we need to configure our local installation to allow it to connect.

First things first however, make sure you are using the same version of JMeter as you are running on the server.

We need to edit the jmeter.properties file that can be found in the bin folder of the installtion you downloaded. Look for the parameter remote_hosts This needs to be set with the public dns of the remote server(s) your connecting to. for example

remote_hosts=ec2-176-34-164-170.eu-west-1.compute.amazonaws.com,ec2-123-34-456-789.eu-west-1.compute.amazonaws.com

Thats your local version configured. You will now be able to tell your local version to run tests on any or all of your specified remotes.

However if your like me you work behind a router/firewall. If so this isnt the end of the story. When you send a test plan to a remote from your local install it will also send the IP address of your local machine for it to send the results back to. JMeter does this by looking up where your current hostname resolves to. In my circumstance it resolved to 127.0.1.1. The reason it did this is down to the fact my systems host file had the line

127.0.1.1 devincharge.local

To resolve this I had to change it to my external IP address

89.345.871.79 devincharge.local

And set up port forwarding from my router to my local machine for all ports from 1024 to 65535. Now, you can if you want use specific ports so you dont have to port forward everything from your router, but i’ll leave that for you to lookup as there are plenty resources on how to do this for you to google and I’ve waffled on for far too long already.

Happy testing

Posts on PHP Boy Scout

Two bugs that taught me the rules

Bug one: the rule-less job that skips your merge requests

Bug two: the import block that only works at the root

The shape they share

Worth remembering

Reviewed, then applied

The gap between “reviewed” and “ran”

Plan as an artifact

Applying is a human decision

The guard on the gate

The bottom line

One graph, not micro-stacks

The pressure to split

What one graph gives you

What splitting costs

When splitting is genuinely right

If ordering between graphs is ever needed

Boiling it down

CI you include, not copy

The .gitlab-ci.yml you keep copying

GitLab has a feature for exactly this

Why a monorepo of components

Authoring discipline

The self-test you cannot fully write

The same instinct, again

The gist

One image for the whole toolchain

The same pile of tools, in every repo

One image, one source of truth

Publishing with crane, not a second build

Soft-failing the scanner on purpose

What it comes down to

A 403 you can't fix in IAM

A 403 on the first real run

The instinct, and why it wastes an afternoon

What is actually in the token

The fix, and the lesson worth more than the fix

The habit it left behind

Pure-Rust Git, no git binary

A VCS subsystem with two halves

A capability ahead of its consumer

Three ways to do Git, and the one worth picking

Repo is a foundation, not a façade

Stepping back

Routing security findings without the noise

Detection is the easy half

Forward the severe, leave the rest

The duplicate you would otherwise send twice

Two tripwires on the root account

Why a quiet alert stream matters here

The short version

Why go-tool-base left GitHub for GitLab

It started with a wrong number

The platform had been quietly failing

The canary left the mine

Why GitLab, and not just “somewhere else”

What I’m not going to claim

Boiling it down

Why I hand-rolled every module

The promise of a wrapper module

What kept going wrong

The CloudTrail bucket, concretely

A wrapper is a deal, and deals have terms

The cost, paid on purpose

To sum up

Hardening the account that will hold the keys

Ready is not the same as safe

The baseline, in one downstream stack

The operator role, and the inversion

The carve-out, because honesty

Harden the room, then move the keys in

Worth remembering

No access keys in CI

The access key you don’t want

Federation instead of a stored secret

Two halves of one handshake

The gotcha: don’t set a profile

The bottom line

Secrets that scrub themselves from RAM

The `.gitlab-ci.yml` you keep copying

`Repo` is a foundation, not a façade

`SecretString` closes both

`forbid`, not just `deny`