
Tips and tricks on writing robust evals for coding agents that test apps whose code we've never seen: different UI/UX contract strategies, UI mutation testing, browser-based E2E verification gotchas.
The Agents on Rails is an LLM benchmarking project governed by the Rails Foundation. They commissioned Evil Martians to design and implement the evaluation tasks and the tooling to run trials at scale. You can follow the Ruby on Rails blog for progress and insight reports, with the up-to-date leaderboard on the official website. On our blog, we're going to share the tech and craft behind those numbers and anecdotes—everything that could be useful for anyone planning to get started with building their own LLM benchmarks or project-specific eval suites. In this first post of The Agents on Rails Files series, we'll share how we designed black-box verification checks for Stage 3 of Agents on Rails so that trials stay fair and results stay reliable. As a bonus, we'll see what happened when we ran the same evals against apps written in Rust.





