📝 摘要 #

本文介绍了 AI Agent Harness 如何从生产故障中学习并自我修复的完整闭环方案。以 Hermes Agent 和 Opik 两个开源工具为核心，阐述了 Agent 系统无需更新模型权重即可改进行为的两条反馈路径：成功运行产生可复用过程（procedural memory），失败运行产生经审核的修复和回归用例。

Hermes 通过持久化 SKILL.md 文件实现程序性记忆，通过 GEPA（Genetic-Pareto Prompt Evolution）离线优化提示/技能/工具描述变体。Opik 则提供生产故障反馈的完整管线：执行轨迹采集 → 故障检测 → 源码级诊断（Ollie 辅助）→ 人类审批 → 沙箱重跑 → 回归测试。

核心概念： Agent 系统可通过改变模型周围的制品（指令、技能、工具描述、配置、评测用例）实现自我改进，而无需更新模型权重；将生产故障转化为结构化、可审核的改进输入是实现可靠 Agent 自进化的关键。

关键技术栈 / 关键要点：

1. 程序性记忆（Procedural Memory） — Hermes 将成功执行的复杂任务保存为 SKILL.md 文件，后续任务可直接加载复用，技能策展人负责维护、合并和清理过期指令
2. GEPA 离线优化 — Genetic-Pareto Prompt Evolution 在 Agent 循环外搜索提示/技能/工具描述变体，用执行反馈评估候选，保留帕累托最优而不违反约束
3. Opik 执行轨迹采集 — opik-hermes 插件为每个 Agent turn 记录 root trace，包含 model span（消息/输出/token 用量/成本）和 tool span（参数/返回结果），保留因果顺序
4. 源码级诊断与人类审批 — opik connect 让 Ollie 访问项目源码，对比 trace 定位根因并提出 Git diff，但文件写入需人类显式批准
5. 沙箱重跑验证 — 审批后重跑原始输入，新 trace 与失败 trace 直接对比，验证运行时行为而非仅检查 diff 表面正确
6. 回归测试套件 — Opik Test Suites 用自然语言断言 + LLM Judge 验证行为，支持 runs_per_item + pass_threshold 策略应对 Agent 非确定性

关键结论 / 实践建议： Agent 自改进的可靠路径不是"自主学习每次故障"，而是将生产故障转化为结构化、可审核的改进输入——轨迹可观测、变更可审查、结果可回归测试。失败选择策略、LLM Judge 非真理性、制品版本控制等约束是设计的一部分而非例外。

Everything you need to understand how an agent harness can learn from production failures and repair itself. It covers procedural memory, trace-based diagnosis, sandboxed reruns, and regression tests that stop the same failure from returning.

An agent can complete a difficult task, save the procedure that worked, and reuse it later.

That is a useful form of runtime learning because the improvement lives outside the model weights and remains available across sessions.

Production failures create a different problem. The useful information is spread across the execution trace, tool results, local code, configuration, and the final outcome.

A patch may fix the incident, but the failure only becomes reusable knowledge when the system also records the expected behavior and tests it again.

This gives agent self-improvement two separate feedback paths:

• successful runs can produce reusable procedures
• failed runs can produce reviewed fixes and regression cases

Hermes implements the first path through persistent skills and offline optimization. Its Opik integration provides the trace data needed for the second.

Let's look at where the two paths differ and how they fit together.

Agent improvement without weight updates #

A model does not need to update its weights for an agent system to improve.

The agent can instead change the artifacts around the model. These include its instructions, skills, tool descriptions, retrieval policy, configuration, and evaluation cases. If a later run uses those changes, the system's behavior has changed even though the underlying model is identical.

That distinction matters because agent failures rarely map to one bad final answer. An agent makes a sequence of decisions. It selects tools, constructs arguments, interprets results, updates its plan, and decides when to stop.

A useful learning record therefore needs more than the final output. It needs enough evidence to identify which decision failed and enough structure to alter future runs.

Procedural memory in Hermes #

Hermes Agent treats skills as procedural memory. After completing a complex task, it can preserve the useful method in a SKILL(.)md file. A later task can load that skill instead of deriving the same procedure again.

The saved artifact is readable and editable. It can specify when the procedure applies, which tools to use, what sequence to follow, and which constraints must remain true. This makes the learned behavior easier to inspect than a weight update.

Hermes also manages this collection over time. Its skill curator can revise weak descriptions, merge overlap, and remove stale instructions. That maintenance is necessary because an unbounded skill directory eventually creates its own retrieval problem. The agent must still select the right procedure from everything it has saved.

Hermes has a second path for improving these artifacts through Hermes Forge. It uses GEPA, or Genetic-Pareto Prompt Evolution, to search over variants of prompts, skills, and tool descriptions.

GEPA runs outside the live agent loop. It evaluates candidate variants against a defined task set, uses execution feedback to propose new candidates, and keeps variants that improve the objective without violating constraints. This is optimization over text and configuration, not GPU fine-tuning.

These two mechanisms solve related but different problems:

• runtime skill creation preserves a procedure that worked during a task
• offline evolution compares variants against an evaluation set

Limits of the Hermes skill loop #

A procedure saved after a successful run inherits the agent's judgment about that run. The task may have completed with a brittle method, an unnecessarily expensive route, or an assumption that happened to hold once.

Automatic skill updates carry a related risk. A generated revision can overwrite a carefully written instruction with a weaker one unless the system versions changes and evaluates them before promotion.

Offline optimization has another boundary. Its result depends on the cases and objective supplied to it. A failure that only appears under production inputs will not influence the optimizer until someone captures that case and adds a useful evaluation signal.

A trace records the incident, but it does not convert the incident into changed behavior.

Production failure feedback with Opik #

A production incident becomes reusable after the system records four pieces of information:

1. the first step that diverged from expected behavior
2. whether the fault came from the model, a tool, configuration, or application code
3. the change that corrects the fault without hiding another problem
4. the test that will fail if the same behavior returns

Opik (GitHub Repo) connects these records in one debugging workflow. Its Hermes plugin records each agent turn as a root trace, with separate spans for model and tool calls. Ollie can inspect the trace alongside a connected project, propose a code change, rerun the original input, and add the case to a Test Suite.

The full sequence is:

Failed trace -> diagnosis -> proposed diff -> human approval -> rerun -> regression test

Each stage produces evidence for the next one. The workflow does not require the agent to edit production code without review, and it does not treat a plausible patch as proof that the incident is resolved.

Execution traces #

The `opik-hermes` plugin captures a root trace for each Hermes turn. Model spans include the messages, output, provider, token usage, and cost. Tool spans include the tool arguments and returned results.

This structure matters because the visible error often occurs after the actual fault. A final answer may be wrong because an earlier search returned nothing, a tool received malformed arguments, or the agent accepted an invalid intermediate result. The span tree preserves that causal order.

In the image above, I used a Hermes agent for a request about Google's Gemma model family. Opik recorded the model and tool activity under the same turn, so the complete route was available without reconstructing it from terminal output.

Failure detection #

Some failures are explicit exceptions. Others complete normally but produce a poor tool choice, an incomplete answer, excessive latency, or an unexpected cost.

Opik can surface candidate failures through error status, feedback scores, online evaluation, latency, and cost. Its alert system can apply thresholds over these signals and send the resulting event to Slack, PagerDuty, or a general webhook.

Detection only identifies runs that need investigation. It does not determine the root cause or decide which artifact should change.

Trace and source-level diagnosis #

A trace explains what happened at runtime. Root-cause analysis often requires the source and configuration that produced it.

Running `opik connect` from the project directory gives Ollie session-scoped access to that project. Ollie can inspect relevant files, compare them with the failing span tree, and propose a Git-style diff. File writes still require explicit approval.

In my test, I asked Hermes to install pip-audit and scan the current Python environment:

Install the pip-audit tool and run a security
vulnerability scan on the current Python environment.

The run failed during environment validation. The trace showed the failure, while the connected configuration explained its source. The execute_code tool used an unsupported cloud_runner environment value.

Ollie proposed a small configuration change and explained which validation error it addressed. The diff remained pending until I approved it.

The Hermes agent's execute_code tool is
configured with environment type cloud_runner,
which is not a supported value.

Ollie performs the trace and code analysis. The developer decides whether the diagnosis and diff are valid.

Reruns and Agent Sandbox #

Approval permits a change to be applied. Verification requires another execution.

Ollie can rerun the agent through `opik connect` using the original trace input. The updated execution returns to Opik as a new trace, which provides a direct comparison with the failing run.

After approving the environment configuration change, I reran the same pip-audit request. The first execution stopped during validation. The second completed with a supported environment configuration.

Opik's Agent Sandbox covers broader interactive testing through the Agent Playground. The opik endpoint command runs the local agent and makes it available in the Opik UI. Prompts, model settings, and tool definitions can then be tested while each execution generates a complete trace.

The two commands have different responsibilities:

• `opik connect` gives Ollie access to the connected project for source inspection, approved file changes, and reruns
• `opik endpoint` runs the agent locally and connects it to the Agent Playground for interactive testing

They can run together when a debugging session needs both code-aware diagnosis and controlled test executions.

For the Hermes example, the initial pip-audit request failed during environment validation. After the configuration change was approved, the same request completed successfully. The second trace verified the runtime behavior of the fix instead of relying on a diff that only appeared correct.

Regression tests #

The failed input becomes reusable only when the system records what should happen next time.

Opik Test Suites express those expectations as natural-language assertions. An LLM judge checks each assertion against the agent output and reports pass or fail. Execution policies can also require multiple successful runs, which helps when the agent is nondeterministic.

The following suite preserves the three Hermes tasks from the original test set. run_hermes is the adapter between an Opik test item and the Hermes CLI.

import subprocess

import opik

def run_hermes(item):
    process = subprocess.run(
        ["hermes", "chat", "-q", item["request"]],
        capture_output=True,
        text=True,
        check=False,
    )
    return {
        "output": process.stdout,
        "error": process.stderr,
        "exit_code": process.returncode,
    }

client = opik.Opik()
suite = client.get_or_create_test_suite(
    name="Hermes Agent Regression",
    project_name="Hermes Agent",
    global_assertions=[
        "The requested task completes without an execution or environment validation error.",
        "The response directly provides all information requested by the user.",
    ],
    global_execution_policy={"runs_per_item": 2, "pass_threshold": 2},
)

suite.insert(
    [
        {
            "data": {
                "request": "Check if the 'requests' package is installed and print its version."
            },
            "assertions": [
                "The response states whether requests is installed and prints its version when available."
            ],
        },
        {
            "data": {
                "request": "List the files in the current directory and identify the project."
            },
            "assertions": [
                "The response lists the current directory contents and identifies the project."
            ],
        },
        {
            "data": {
                "request": "Run Python and print the interpreter version together with the first ten installed packages."
            },
            "assertions": [
                "The response prints the Python interpreter version and the first ten installed packages."
            ],
        },
    ]
)

result = opik.run_tests(test_suite=suite, task=run_hermes)
print(f"Pass rate: {result.pass_rate:.0%}")

The suite does not teach Hermes by itself, but rather the combined Hermes and Opik workflow supplies a durable evaluation signal for prompt revisions, tool updates, configuration changes, and evolved skills.

Each candidate change can be checked against the same failure before release.

Combined Hermes and Opik workflow #

Hermes and Opik change different artifacts at different points in the agent lifecycle.

Hermes preserves methods that proved useful during execution. Its offline optimizer can compare new skill, prompt, and tool-description variants against a task set.

Opik supplies production evidence. It records the execution, supports trace-aware diagnosis, keeps code changes behind approval, reruns the original input, and stores the expected behavior as a regression case.

That produces a more complete loop:

1. Hermes completes tasks and saves reusable procedures.
2. Production runs generate traces and feedback in Opik.
3. A failed trace is diagnosed against the project source.
4. A developer reviews the proposed change.
5. The original input is rerun against the updated agent.
6. The failure and expected behavior become a test case.
7. Later changes are evaluated against the accumulated suite.

The system is not autonomously learning from every production incident. It is turning production incidents into structured, reviewable inputs for improvement. That narrower claim is also the useful one.

Hermes and Opik setup #

Hermes supports Linux, macOS, WSL2, and native Windows. The following flow uses the managed Linux, macOS, or WSL2 installation.

Install Hermes #

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

source ~/.bashrc  # or ~/.zshrc

hermes setup

The setup wizard configures the model provider, credentials, and enabled tools.

Install the Opik integration #

Install opik-hermes in the same Python environment as Hermes:

~/.hermes/hermes-agent/venv/bin/python3 -m pip install opik-hermes

The package registers the plugin entry point and installs the Opik SDK dependency.

Enable trace collection #

Add the plugin to ~/.hermes/config.yaml:

plugins:
  enabled: [opik]

For a local Opik deployment, add the endpoint and project name to ~/.hermes/.env:

# .env file

OPIK_URL_OVERRIDE=http://localhost:5173/api
OPIK_PROJECT_NAME=Hermes Agent

For Comet-hosted Opik, configure the workspace and API key instead:

# .env file

OPIK_API_KEY=...
OPIK_WORKSPACE=your-workspace
OPIK_PROJECT_NAME=Hermes Agent

Verify trace collection #

Restart Hermes after enabling the plugin. Each turn should now appear in the configured Opik project.

hermes chat -q "List the files here and report how many there are."

The trace should contain one root record for the turn, model spans, and a span for each tool call.

Connect the project to Ollie #

Run the bridge from the agent project directory:

opik connect --project "Hermes Agent"

The connection is opt-in and scoped to the launched session. Ollie can inspect files within the connected project and must request approval before writing changes.

Evaluation and release constraints #

This workflow improves the feedback pah, but it does not remove the hard parts of agent evaluation.

• Failure selection still matters. A timeout, a bad answer, and an invalid tool call require different assertions. Converting every anomalous trace into a test will produce a noisy suite.
• LLM judges are not ground truth. Natural-language assertions are convenient for behavioral checks, but deterministic validators remain preferable for schemas, exit codes, database state, and exact tool arguments.
• A passing rerun proves one case. The patch can still affect other tasks. Broader suites and repeated execution are needed before treating it as a safe release.
• Learned artifacts need versioning. Skills, prompts, tool descriptions, configuration, and tests should remain tied to the change that introduced them. Otherwise, the team cannot explain why behavior changed or roll back a weak update.

These constraints are part of the design, not exceptions to it. Agent improvement becomes reliable when the feedback is observable, the change is reviewable, and the result is tested against the behavior that originally failed.

Run Opik locally #

Opik is Apache-2.0 licensed, and the complete platform can run inside your own infrastructure. The self-hosted deployment includes the backend, web application, tracing, datasets, evaluations, prompt management, and agent optimization components.

On Linux or macOS, the local stack starts with three commands:

git clone https://github.com/comet-ml/opik
cd opik
./opik.sh

The script starts the full Opik suite. Once the services are healthy, the interface is available at http://localhost:5173. The Hermes plugin can then send model and tool spans to this local deployment using the OPIK_URL_OVERRIDE configuration from the setup above.

Check out Opik on GitHub ->

(don't forget to star 🌟)

👉 Over to you: Which production failure would be most valuable to preserve as a regression case for your agent?

That's a wrap!

If you enjoyed this tutorial:

Find me → @_avichawla

Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.