Eighteen months ago we ran our first eval harness against a handful of agent prototypes. Today twelve of those prototypes are in production serving real users. Here's what survived the transition and what didn't. Tool selection beats model selection We assumed the model would be…
We are spinning up eval harnesses across code, agents, and tool-use. If you run benchmarks in prod, please share which tasks break today so we can target them. Reasoning + long-horizon planning are priority.
Today we released Grok-4 weights under an Apache 2.0 license. The model is competitive with the frontier on reasoning and coding, behind on long-context recall, and ahead on Chinese and Spanish. Here's the thinking. Open weights are good for the field Closed labs argue that…
Pulled together the tools 200+ AI engineers actually use day-to-day. Eval frameworks converged, observability is still fragmented, and structured-output is now a solved problem.
We're running multi-step agents with tools and the eval bill is getting absurd. Sampling strategies, cached fixtures, anything that kept your eval budget sane?
Foundation models, fine-tuning, agents, evals.
Eighteen months ago we ran our first eval harness against a handful of agent prototypes. Today twelve of those prototypes are in production serving real users. Here's what survived the transition and what didn't. Tool selection beats model selection We assumed the model would be…
We are spinning up eval harnesses across code, agents, and tool-use. If you run benchmarks in prod, please share which tasks break today so we can target them. Reasoning + long-horizon planning are priority.
Today we released Grok-4 weights under an Apache 2.0 license. The model is competitive with the frontier on reasoning and coding, behind on long-context recall, and ahead on Chinese and Spanish. Here's the thinking. Open weights are good for the field Closed labs argue that…
Pulled together the tools 200+ AI engineers actually use day-to-day. Eval frameworks converged, observability is still fragmented, and structured-output is now a solved problem.
We're running multi-step agents with tools and the eval bill is getting absurd. Sampling strategies, cached fixtures, anything that kept your eval budget sane?