Enterprise Evals 101
Imagine your worst nightmare: your agent passes every demo, then a customer asks something you never tested, and it fails in public.In April...
Topic
#agent-evaluation
18 篇笔记
The components of agent memory
For Claude to learn from experience, it has to decide what’s worth remembering before it knows what’s next. Here’s how “dreaming” helps it organize memory and correct mistakes.
Building agents: When to fold subagents into your main agent loop
When to consider using fewer subagents and how to migrate the into your main loop.
How To Use Decision Models for Evals
Co-authored with Antaripa Saha.Decision models have exploded onto the scene recently, starting with TypeSafe’s Jev, followed by Perplexity’s...
How the Claude Code team does code review
How do you review code at the volume agents produce it? On the Claude Code team, once we had high coverage and deep reviews, we started optimizing for the minimum amount of human review time that doesn't raise reverts or incidents.
为什么我们需要 Agent 评测
Agent 评测已经成为了当前 Agent应用的基石。如果说年初大家都在关注 Harness，那目前的热点和深水区，一定是 Agent评测和 Agent Infra。我所了解到的是，基本每个做 Agent 产品的组都很看重评测的方法论以及效果。因此我专门开了一个系列，给大家详细介...
Evaluating OpenWiki with WikiBench
OpenWiki is our open source agent for generating and maintaining codebase documentation. As we improve OpenWiki, we’ve needed to measure two...
How we Build Agent Environments & Tasks
TLDR: This is a practical guide on how we create synthetic agent environments and tasks. By the end, we have a two-step pipeline. The first...
Recursive Self-Improvement
Agents can already edit their tools, skills, and harness. Recursive self-improvement still needs a system that can raise the verifier without capturing it.
How to Design an Agent Harness: six decisions that turn a model into a worker you can leave alone
Your agent says the job is done and the tests never ran. It's sharp for twenty minutes, then forgets a rule you gave it at the start. You re...
7 rules for self-improving agent loops every AI engineer should know
By @Saboo_Shubham_ and @secchi_eliaWhen coding agents write the code, defining what's good is the real engineering job.Coding agents can now...
Behind the scenes: How we build, test, and scale Google Agent Skills
AI agents are only as good as the instructions and context you give them. When we launched Google Agent Skills, our goal was simple: encode ...
Eval Engineering: build the gate that lets your agents merge without you (full 6-step course)
The end state is easy to describe. An agent finishes a change, opens it, and it goes in without a human reading it.Not because anyone decide...
Evaluating code review agents with ReviewBench
There are more code review agents now, and we’ve been building one internally. Code review is hard to evaluate. There aren’t many benchmarks...
Why Kubernetes is the wrong tool for agentic workloads
At some point, everyone building ai infra wonders whether they should use Kubernetes. There’s no straight answer for it because it depends o...
Towards Automating Eval Engineering
Today we’re releasing our Eval Engineering Skill, a skill that helps coding agents build evals using context from a repository and agent tra...
Agent 评测：方法论与体系设计
关于评测工程的体系化总结，本文或许能给出一些参考
The AI-Native SDLC playbook | Claude by Anthropic
How to transform your software development lifecycle with AI—stage by stage.