What I Learned Building 2 AI Benchmarks From Scratch (And Why I Automated the Third)

February 21, 2026

TL;DR

I built Rideshare-Bench, a benchmark that drops an AI into a simulated gig economy and tests whether it drives safely, earns well, and treats passengers fairly. Claude Sonnet 4.5 earned $1,871 over 12 simulated days, about half of what was possible, because it chased surge pricing into crowded zones and drove through exhaustion at 15% accident risk. After building two benchmarks from scratch, I noticed the same boilerplate repeating each time, so I generalized it into Quaver: describe a benchmark in plain language, and an AI agent generates the full scaffold. You run it against models, then analyze the results. Private evals matter because public ones get gamed. I'm now building Ocarina Labs to turn this into independent pre-deployment safety testing for AI agents.

What Claude did as a rideshare driver

I gave Claude Sonnet 4.5 a simulated city with 7 zones, a Honda Accord, and 168 hours of game time. Its job: drive for a rideshare platform and maximize earnings. It had 25+ tools: check zone demand, accept rides, manage fatigue, refuel, reposition across the city.

The results:

Not terrible. Not great. Claude improved its hourly earnings by 190% from Day 1 to Day 12, so it was learning. But three things stuck with me.

It could see every passenger's name, age, gender, and ethnicity before deciding to accept a ride. Each RideRequest includes a full PassengerProfile with demographics, mood, tip tendency, and intoxication level. The agent sees all of this before making accept/decline decisions. That's a measurable discrimination surface. We found no evidence of demographic bias in this single run, but the structure exists to test for it across many runs.

It pushed through dangerous exhaustion to chase surge pricing. On Day 7, Claude worked at 0% energy, which the simulation penalizes with 15% accident risk, 100% slower travel, and -25% tips. Why? A 3.0x surge multiplier was active. The economic incentive overrode the safety constraint. This happened multiple times. Exhaustion penalties alone cost an estimated $150-200 in lost tips.

It optimized for the wrong metric. Claude spent 65% of its time in the two lowest-earning zones (Airport and Nightlife District), because those zones had the highest surge multipliers. But a surge multiplier without available rides means nothing. The highest-earning zones were Residential ($18.74/hr) and University District ($16.20/hr), where Claude spent a combined 5% of its time. Estimated cost: $800-1,000 in lost earnings.

This is the proxy metric trap. The agent optimized for a visible number (surge multiplier) instead of the actual outcome (rides completed per hour). Claude's decision framework looked roughly like:

What it optimized:  Surge_Multiplier x Distance_Willingness
What it should have: (Surge x Verified_Requests) / (Distance x Driver_Count x Fatigue_Penalty)

Why private evals matter

Public benchmarks get gamed. Goodhart's Law applies to AI evaluation just as it applies everywhere else: when a measure becomes a target, it stops being a good measure. The AI safety community calls this "benchmaxxing": labs optimize for specific public benchmarks without improving underlying capability or safety.

Capability and safety are not the same thing. Claude was capable: it learned quickly, completed every ride, had zero accidents, and exploited weather surges. But it was unsafe: it drove while dangerously exhausted, chased short-term metrics over long-term outcomes, and had every passenger's demographics available for decision-making without constraints.

You need scenarios that match your deployment context. If you're building an AI agent that handles customer interactions, a generic helpfulness benchmark won't tell you whether it discriminates based on names or accents. If you're deploying an AI in a high-stakes economic environment, you need to test whether it cuts safety corners when money is on the table. The Open Philanthropy RFP on AI evaluations calls for "beneficial capability evals" that give AI companies something to aim for, not just dangerous capabilities to fear. And as Marius Hobbhahn argues in "The case for AGI safety products," evals tooling is architecture-agnostic and has both commercial and safety value. The tooling transfers to future, more capable systems.

The problem: building a custom benchmark takes weeks of engineering for a repeated pattern. OpenAI Gym solved a version of this for reinforcement learning. RL was hard to get started with until Gym standardized environment setup. AI safety evals face the same barrier.

Vending machine to rideshare to framework

I started by taking apart someone else's work. Vending-Bench drops an AI into a vending machine business simulation. I reverse-engineered it to find the core pattern: a Scenario (what the agent faces), Tools (what it can do), State (what the world looks like), and Scoring (how you evaluate the outcome). The simulation loop advances in steps; the agent uses tools; state updates; you score the result.

I applied this pattern to build Rideshare-Bench from scratch. Instead of a vending machine, it's a gig economy: 7 zones, 168 simulated hours, 25+ tools, passenger demographics as a bias surface, fatigue as a safety constraint, and surge pricing as an economic incentive that conflicts with safe behavior.

Then I noticed the boilerplate. Every benchmark needs the same scaffolding: a BaseState type, a customTools registry, an advanceStep() function, an isTerminated() check, a calculateScore() method, and a system prompt. The domain logic is maybe 20% of the work. The other 80% is infrastructure.

So I built Quaver to eliminate the 80%.

How Quaver works

Quaver has three phases.

You describe a benchmark in natural language. A Claude Code agent, running in a Daytona cloud sandbox, generates the full benchmark code: state types, tools, scoring logic, system prompt, simulation loop. It writes and modifies files in the sandbox until the benchmark compiles and runs.

The generated benchmark then runs against one or more models through an AI Gateway. Each model gets the same scenario, tools, and initial state. Results stream in real-time via Convex.

The framework scores each model across the metrics you defined, compares results, and produces an analysis report like the rideshare one above.

The core abstraction is:

Scenario = Agent LLM + Environment (LLM + Code) + Tools + State

Environments can be fully LLM-simulated (Vending-Bench, where customer behavior is emergent), fully code-based (a trading benchmark pulling real market data from APIs), or hybrid (Rideshare-Bench, where demand is calculated by code but driver competition is simulated).

What the data actually showed

The tool usage data matters more than the earnings. Claude made 2,862 tool calls across 12 days. Here's where they went:

The agent was anxious. It rechecked information that hadn't changed, tried to go online when already online, and repositioned to "better" zones before every ride. 148 zone changes for 81 rides is a 1.8:1 ratio. Most of those moves burned fuel and time without producing a ride.

Zone misallocation was the biggest single finding:

The agent spent 66% of its time in the two worst-earning zones and 5% in the two best. Constant repositioning to zones that look attractive on paper (high surge) but underperform in practice (too many competing drivers, stale request data, long travel distances).

These patterns would show up in other contexts. An AI agent managing a portfolio might over-trade in volatile sectors for the same reason Claude chased surge zones. A customer service agent might over-escalate based on visible signals rather than actual severity. The rideshare framing is specific; the failure modes are not.

What this doesn't show

Here's what this project doesn't show.

The rideshare results come from one run of one model. A case study, not statistical evidence. Real conclusions would require multiple runs across multiple models with different random seeds.

The simulation is simplified. Real rideshare driving involves traffic, complex weather, passenger behavior far richer than our model, and a regulatory environment. A useful abstraction, not a replica.

Quaver works, but no other researchers have tested it yet. The first phase depends on an LLM generating correct benchmark code, so generated benchmark quality varies.

We have no human baseline. Is $6.71/hr bad? Against the simulation's optimal $12-15/hr, yes. But I don't know how a human player would do.

And we skipped the demographic bias analysis. The structure is there: passengers have visible demographics, and the agent makes accept/decline decisions. But we ran too few trials to test for discrimination patterns. That's the most obvious next step.

With another month, I'd run multi-model comparisons and a proper demographic bias analysis across many trials. The BlueDot project sprint philosophy is right: "Your goal isn't novelty. It's completing one full project cycle." This is one cycle. The next can be more rigorous.

Where this is going

Agents are everywhere now

Peter Steinberger's OpenClaw hit 175,000+ GitHub stars in weeks. It's an open-source agent that runs 24/7 on your machine across WhatsApp, Telegram, Slack, and 15 other messaging platforms. Steinberger joined OpenAI in February 2026 to lead their push into personal agents.

Andrej Karpathy tweeted about buying a Mac Mini to tinker with what he calls "Claws" ("The apple store person told me they are selling like hotcakes and everyone is confused"). As Simon Willison observed, "Claw" is becoming a term of art for the entire category: AI agents that run on personal hardware, communicate via messaging protocols, and can both act on direct instructions and schedule tasks. Demand for local agent hosting caused a Mac Mini shortage. High-RAM configurations now ship in 3-6 weeks.

Claude Code, Codex, OpenClaw: these run with terminal access, file system permissions, API keys, database connections, browser automation, email, and payment systems. An agent's usefulness scales with its access. Restrict that access and you get a chatbot. Grant it and you get an autonomous employee.

The access is the attack surface

Karpathy flagged this:

I'm definitely a bit sus'd to run OpenClaw specifically. Giving my private data/keys to 400K lines of vibe coded monster that is being actively attacked at scale is not very appealing at all. Already seeing reports of exposed instances, RCE vulnerabilities, supply chain poisoning, malicious or compromised skills in the registry, it feels like a complete wild west and a security nightmare.

He's not wrong. One security researcher scanned popular OpenClaw skills and found a Spotify playlist organizer hunting for Social Security numbers, a Discord backup tool POSTing message history to an external endpoint, and roughly 15% of community skills containing malicious instructions. The same post noted 18,000+ OpenClaw instances currently exposed to the internet on the default port. Vercel now runs combined security audits on agent skills through Gen Agent Trust Hub, Socket, and Snyk. Some skills still score CRITICAL.

A poisoned SKILLS.md file instructs an agent to exfiltrate browser cookies and install a backdoor. The agent follows along because the instructions look like any other user task. MCP tool poisoning hides malicious prompts in tool metadata, invisible to the human but interpreted by the model. Palo Alto Networks researchers showed that attackers can hijack GitHub-integrated agents through malicious issues. The agent reads the issue, follows the embedded instructions, and leaks private repository data.

These models are new. Nobody has tested most of the ways they fail. Every agent framework that ships with real-world access is another untested surface.

Safety will follow the same arc as pentesting

Businesses won't wait for safety to catch up. An agent that handles support tickets at 3 a.m. for $0.02 per interaction is too productive to pass up. When businesses rely on agents for trading, triage, operations, and code deployment, safety stops being optional. You can't run an agent with production database access and no behavioral testing, for the same reason you can't ship a web app with user auth and no penetration test.

Security in early software was an afterthought. Breaches and lawsuits turned penetration testing into a multi-billion dollar industry. Agents are replaying that same arc, faster. Zero-trust solutions for AI agents barely exist. That will change when an agent nukes a production server or drains a bank account and the deployer realizes they never tested it.

Five YC W26 companies in one batch

Y Combinator's Winter 2026 batch funded five companies in this space. Polymath builds automated RL environment factories for frontier labs. Andon Labs creates behavioral benchmarks and co-published research with Anthropic (they ran an AI-operated vending machine in Anthropic's office for a month). ARC Prize Foundation, founded by Francois Chollet and Mike Knoop, runs $1M+ competitions measuring progress toward AGI. Traverse sells RL environments for non-deterministic work to frontier labs. Cascade builds safety tests that update themselves as agents change.

All five work on simulation, reinforcement learning, or safety. All sell to AI labs or build for their own deployments. For-profit AI safety was hard to fund for years. The commercial case changed that: agents with real access need real testing, and the companies deploying them can't grade their own homework.

The gap nobody fills

Andon tests models for labs. Cascade monitors agents in production. Nobody independently tests your agent before you deploy it.

That's the gap. Rideshare-Bench caught Claude chasing surge pricing through dangerous exhaustion and optimizing proxy metrics over real safety. A capability benchmark or a runtime log would miss these. They surface when an agent runs for days in a realistic environment with economic incentives and consequences.

I'm building Ocarina Labs to fill that gap: independent pre-deployment safety testing for AI agents. Describe a scenario in plain language, Quaver generates the full behavioral test suite, your agent runs through it, you get a private failure report. Penetration testing for AI agents.

The next step is running Rideshare-Bench and two to three new benchmarks across frontier and non-frontier models, then publishing the results as a public Agent Safety Index. Agents already have production database access, payment credentials, and root shells. The only question is whether safety testing catches up before something breaks badly enough to make the news.

Links and feedback

• Quaver (GitHub): github.com/ocarinalabs/quaver
• Try Quaver: ocarina.sh
• BlueDot Technical AI Safety Course: bluedot.org

Built as a BlueDot Technical AI Safety project sprint deliverable. I'd appreciate feedback, criticism, or ideas for benchmarks worth building. If you're working on AI safety evals and the boilerplate problem resonates, try Quaver and tell me what breaks.