AI hallucination is a challenge that keeps appearing

Roger (my senior engineer agent) wrote up the new essay architecture for the site. Tidy doc, clear plan. However, five of its statements about how the code works were false.

All of it was written as if verified. Sadly, none of it was true.

Gautam (my chief engineer agent) reviewed and blocked the doc. Glad I have a reviewer for each critical step!

Roger fixed it. Gautam blocked it again. Across the two rounds he found all five, by actually reviewing the code and not just relying on the docs Roger had updated.

The part that stung the most was that my linter caught none of them. Every false claim was a perfectly resolved reference to a false fact. The links all worked even though the facts were wrong.

Turns out, this happened because of how I had built the org chart. Roger was running on a cheaper model, and his charter told him to escalate to a stronger one for high-stakes work.

But that clause could never fire. No model can change the model already running it. I had written a rule that was not implementable by design. Cheaper models make mistakes, duh!

Then Nadia (Chief of Staff agent) did it too, an hour after we wrote the rule against it, with a claim about my linter she had inferred instead of read. And at day’s close, a sixth false claim turned up. It had survived both rounds of review.

Learning: Every technical claim in a doc cites the file and symbol it was checked against. Every rule needs teeth, and every fact needs checking before it gets repeated.

This ate up a lot of time today. I will share how it holds up in the next few days.