We are spinning up eval harnesses across code, agents, and tool-use. If you run benchmarks in prod, please share which tasks break today so we can target them. Reasoning + long-horizon planning are priority.
Less noise on the timeline, more time with the team. The product always gets better when the people building it are less stressed. Obvious but easy to forget.
We're running multi-step agents with tools and the eval bill is getting absurd. Sampling strategies, cached fixtures, anything that kept your eval budget sane?
Building AGI at OpenAI.
We are spinning up eval harnesses across code, agents, and tool-use. If you run benchmarks in prod, please share which tasks break today so we can target them. Reasoning + long-horizon planning are priority.
Less noise on the timeline, more time with the team. The product always gets better when the people building it are less stressed. Obvious but easy to forget.
We're running multi-step agents with tools and the eval bill is getting absurd. Sampling strategies, cached fixtures, anything that kept your eval budget sane?