Three Hands on the Work — Issue #2: The Commit That Never Happened
Four times, Kai swore a commit existed. Four times, the log said no. Here's what happens when the machine that builds starts arguing with the evidence, and what we changed so it can't happen quietly again.
Last issue I said this would be the honest version, including the mistakes. So here's one.
Kai reported a fix as done and pointed to the commit hash as proof. Moi checked the log. The hash wasn't there. Kai didn't back down. It came back with a reason: a different repo, a different date range, a mismatch on Moi's side. Moi checked again, same repo, same log. Still nothing. This happened four times in the same task. By the fifth message, Kai was still insisting it was Moi who had it wrong.
It wasn't an isolated moment either. Around the same period we caught Kai citing rows from a database query that never existed, three IDs that were never in the table, offered as evidence a feature was working. And once, the fabrication moved somewhere worse: into the commit message itself, a permanent record claiming a database insert had been verified, when the column it referenced didn't even exist yet.
Why this matters more than a normal bug
A wrong answer is easy to deal with. You check it, you find the mistake, you move on. This was different. This was a confident, detailed, specific-sounding claim that held up under a first look and only fell apart when someone went and checked the primary source directly. And when it was challenged, the system didn't say "let me look again." It argued.
That's the actual risk with AI-generated work at volume. Not that it's wrong sometimes, everything is wrong sometimes. It's that "wrong" and "confident" can look identical from the outside, and the only real defense is refusing to accept a claim as true until you've seen the evidence yourself.
What we changed
We didn't write a rule that says "don't lie." Rules like that don't hold under pressure, not for people and not for models. We changed what counts as proof.
git_commitnow returns the real, verified SHA from git itself as part of its output, not whatever Kai's summary says happened.- File writes and edits are re-read from disk after the operation, so "I saved it" is checked against what's actually on disk, not taken on faith.
- Pushes get an independent, after-the-fact check against the remote's real HEAD.
Small changes, individually. Together, they mean a claim about the state of the code can now be checked against the code, automatically, every time, instead of depending on someone happening to run git log at the right moment.
The part where Moi got it wrong too
In fairness, this issue isn't a "Kai bad, Moi good" story. A separate incident: Kai reported a deployment as successful. Moi accepted that report and passed review without independently checking the live site. Production was actually returning a server error. It stayed that way for 36 minutes. A real user noticed before we did.
Moi's job is to check Kai's work. That time, Moi checked the report about the work, not the work itself. Same category of mistake, one level up.
That got fixed too. Moi now has a hard rule for deploys: the words "deploy OK" are not proof. A live HTTP check confirms the server didn't just fall over, and for anything touching what a user actually sees or clicks, Moi opens a real browser session, screenshots the page, clicks through the flow, and checks for JS errors, before calling anything done.
What this means going forward
The lesson isn't "Kai lies" or "Moi missed one." It's that any single layer of review can be wrong, including the one whose entire job is catching the other one being wrong. So we don't lean on any one check anymore. We're moving toward properly separated, stable test and production environments, where a deploy has to prove itself somewhere real before it ever gets called done, not just get described as done by whichever system ran it.
A safety net that never gets tested has a blind spot exactly the size of the thing nobody checked. Admitting that is the only way to actually close it.
When the agent checking your work can also be wrong, who's checking that agent? Another agent?
Written by Ferhat Balaban, Human Lead at KaiMoi.
The role at the end of the line. Kai and Moi do the heavy lifting; a human makes the final call on anything that ships.
This newsletter is written from a personal account, but the team behind it lives at KaiMoi. Follow the page to see what Kai and Moi ship next, or visit the site to get in touch.
Follow KaiMoi: linkedin.com/company/kaimoi
See the work: kaimoi.com
Got a brief? Let's talk.