Discussion about this post

User's avatar
Foundation's Edge's avatar

Your field-report section is the whole ballgame: the environment teams build around a model now explains 60-80% of performance variance, and identical models score 15+ points apart on SWE-bench under different harnesses. The AI-needs-context point generalizes - context is not a convenience but constitutive, the difference between a scaffold that supports capability and a cognitive niche that creates it. Your breakdown/recovery stat pairs neatly with METR: developers felt 20% faster while running 19% slower, so confidence and capability have separated and recovery matters twice over. Wrote up the environment-first view here: https://seldondance.substack.com/p/the-cognitive-niche #AIAgents #AIQuality

Meridian Signal's avatar

The recovery section stood out, Jakob. Being able to correct one assumption without starting over would make a big difference. As assistants take more actions, I would want that same control over what they have already changed.

1 more comment...

Ready for more?