Independent notes
Evidence-led notes on safe change, recovery, cloud security, and the systems behind production software.
Latest writing
Browse archiveI wanted a small lab for one narrow question: when a public endpoint receives more requests than it should pass to an origin, can the edge shed the excess while the health signal stays useful?
The answer in this lab was yes. The interesting part was not making requests fail. It was making the rejection deliberate, measurable, and contained.
GLM-5.3 is much cheaper than Kimi K3 for text-heavy coding agents, and its published results are strong. That does not make it a drop-in replacement. Kimi K3 can inspect screenshots and other visual input; GLM-5.3 cannot. In an operations workflow, that boundary matters before any benchmark chart does.
I compared the vendors' current model cards, pricing pages, and the public Terminal-Bench leaderboard. This is a source-led comparison, not an independent bake-off. The published scores use different harnesses and budgets, so they are useful for choosing what to test, not for declaring a winner.
Part 3 of Operating AI Agents Like Production Systems. A green HTTP check can tell you that an agent endpoint answered. It cannot tell you whether the agent completed any useful work, whether it is stuck behind approval, or whether the tool runner stopped executing actions.
I built the last fixture in this series around that distinction. It counts completed executions, review-queue items, and total requests. The input is intentionally tiny: synthetic event records, not model traces or production telemetry. The goal was to make one quiet failure visible: every request can receive an answer while the execution rate stays at zero.
That failure is easy to miss if the dashboard only tracks request success and latency. The endpoint is alive. Users may even see polished answers. The business action never happened.
Part 2 of Operating AI Agents Like Production Systems. An approval is only useful when it names the action it approves. A generic approval can be replayed against a different refund, a different customer, or a different operation.
I used the same synthetic refund from the first tutorial: $240 on invoice-42, with a policy that requires human approval above $100. The first version of the workflow only asked whether an approval existed. That is too loose. A reviewer could approve one request while the agent sends another payload to the billing tool.
This tutorial adds a small action fingerprint: operation, target, and amount. It is not a cryptographic authorization protocol. It is the smallest testable rule that makes a mismatch visible before an external action is allowed.
Part 1 of Operating AI Agents Like Production Systems. A helpful agent answer and an operable agent action are different artifacts. Before an agent can change an external system, a reviewer needs the input references, policy result, exact proposed action, approval state, and execution state.
I built a tiny synthetic refund workflow because it makes the gap obvious. A model can say, “The $240 refund was submitted,” yet leave no way to tell which rule it applied, whether a person approved the charge, or whether the billing tool ran. That sentence may be useful to a customer. It is useless in a review.
The post that prompted this series argued for evidence bundles. This one turns the idea into a small contract and a test. I ran it in a locked-down Docker container against synthetic JSON only. It did not call a model, payment provider, approval service, or production API.
A faster training loop can shorten the time between an idea and an artifact. It does not tell us whether that artifact should be released.
I started with a smaller question than “does distillation work?” The environment available to me has Docker but no GPU, so it cannot establish a throughput result for a large teacher model. It can still answer a useful delivery question: what exactly is available to run, and how much confidence does that give us?
Hugging Face's TRL documentation describes DistillationTrainer as on-policy knowledge distillation. The student generates its own completions; the trainer then compares the teacher's next-token distribution on those completions. That targets the gap between fixed training outputs and the outputs a student produces at inference time.
The upstream project has also published a “40x faster” claim for a particular distillation setup. That is an upstream benchmark claim, not a release decision for another model, dataset, hardware profile, or product.
An agent becomes a different kind of system the moment it can do more than answer. Give it a terminal, a cloud API, a ticketing system, or a deployment tool and its output can change a real environment.
That is where I stop treating the prompt as the main safety mechanism.
A prompt can explain intent. It can tell an agent to be careful. It can ask for confirmation before a destructive operation. But it cannot enforce an allowlist, prove that an approval happened, prevent an unrecognised command from reaching an executor, or tell me whether the claimed outcome exists outside the chat transcript.
Those jobs belong to a control plane.
A service can be quiet for two very different reasons. Maybe it is healthy and nobody is using it. Maybe the checkout pipeline stopped or the nightly report never ran. Error rate can still be zero.
That distinction matters because most alert rules start with an error condition. 5xx exceeds a threshold. Queue depth grows. Latency crosses a limit. Those are useful alerts. They answer a narrow question: did the system produce evidence of a bad thing?
They do not answer a second question that is often closer to the outage: did the work we expected actually happen?
A mature codebase rarely announces that it needs refactoring. It makes a small change feel oddly expensive. A new route requires touching three modules whose names no longer describe their responsibilities. A dependency update turns into archaeology. An incident fix needs a second person on the call because nobody is certain which branch of the conditional protects the customer path.
That is the moment I find the bonsai metaphor useful, with one important correction: the job is not to make code look pretty. The job is to preserve a living system while shaping it so the next change has somewhere safe to go.
A failed GitHub Actions run is not a reproduction recipe. It is evidence from a particular revision, runner image, dependency state, and execution context. The useful first move is to preserve that evidence without pretending the environment can be recreated from one log line.
I released ci-capsule v0.1.0 after building it around that constraint. It is a small, read-only CLI for failed GitHub Actions runs: collect a local evidence bundle, sanitize bounded credential-shaped values, recover a static command only when the workflow source supports it, and say unavailable when it does not.
Browse by problem
View all topics