Built a scorecard for how often I correct my AI
I spent most of today on something bigger than a feature: designing what a genuinely self-improving second brain would look like. One that learns from its own mistakes and keeps itself honest, instead of leaning on me to catch everything.
A big piece of that is a scorecard. I want to measure how well the AI actually understands what I’m asking and gets it right without my help. Say I ask it to write a post like this one. Does it come back in the voice I’ve spelled out in my style guide, or do I have to send it back and fix it? If it needs corrections or a redo, that counts as a miss.
That “how often does it get it right without me stepping in” number is my real north star. To measure it, I first had to define a loop: one full attempt at a task, from me asking to me accepting the result. Every correction or redo before I accept counts against that loop.
But something has to read each session and count the loops, and it can’t be my main AI. An AI grading its own work is going to flatter itself (that’s my assumption, anyway). So I wanted a separate, local model doing the scoring.
Which one, though? I ran a little bake-off: Gemma 4, and Qwen with its “thinking” mode on and off. For this specific job, Qwen 3.6 with thinking off won easily. It was 15-20x faster than the big Gemma and just as accurate, and turning thinking on actually made it both slower and worse. So Qwen became the judge.
And just like that, I have a scoring system. I can finally track how well my AI is doing, and watch that correction rate fall as the brain gets smarter. It’s genuinely a cool thing to have running.
The goal I’m setting: cut the corrections I make per loop by 40 to 50 percent over the next three months. Now I can actually tell whether I’m getting there.