Want to learn evals? buildevals.com →

The self-improving loop for AI agents.

Trace. Experiment. Test. Evaluate. Improve. Monitor. Run again.

agent.run
4.21s
plan
0.72s
tool:search
0.94s
tool:db.query
0.58s
tool:policies ⚠
0.47s
generate
1.10s
run_a91f · support-bot · 1 span flagged 
HEALTH
agent
support-bot
last 24h
run health0.92
eval pass rate94%
tool errors3%
error recovery86%
drift (PSI)ok
baseline outliers2
WORKS WITH
Vercel AI SDK·Vercel Eve·✳Claude Agent SDK·MCP·{ }REST API·OpenAI Agents SDK·🦜LangChain·LangGraph·🦙LlamaIndex·CrewAI·AutoGen·Mastra·Pydantic AI·Semantic Kernel·🤗smolagents·DSPy·Haystack·Google ADK·Cloudflare Agents·Agno·any OpenTelemetry SDK·Vercel AI SDK·Vercel Eve·✳Claude Agent SDK·MCP·{ }REST API·OpenAI Agents SDK·🦜LangChain·LangGraph·🦙LlamaIndex·CrewAI·AutoGen·Mastra·Pydantic AI·Semantic Kernel·🤗smolagents·DSPy·Haystack·Google ADK·Cloudflare Agents·Agno·any OpenTelemetry SDK·
// THE PROBLEM

Improving an agent is manual work today.

You test by hand. You browse traces to find where the agent went off task, where a tool call failed, where the answer quietly got worse.

Then you change a prompt, a model, a tool. Everything underneath moved too, so there is nothing left to compare against. And once the change ships, nothing keeps measuring whether it still works, or whether it actually got better.

RunAgain closes that gap. Trace what happened, evaluate what changed, replay it exactly as it ran so the two are comparable, turn every issue into a test, improve the agent automatically, and measure it against your goals and real conversions.

You build the agent.

We build the loop around it.

// THE FEEDBACK LOOP

Trace. Experiment. Test. Evaluate. Improve. Monitor. Run again.

01 TRACE
Every span, tool call and token, captured live.
02 EXPERIMENT
A/B prompt
Fork prompts, models and tools behind flags.
03 TEST
✓ mocks
✓ sims
Deterministic runs against mocks and sims.
04 EVALUATE
0.91
LLM-judge, rubric, assertion and dataset scoring.
05 IMPROVE
Diff regressions, fix the weak spans, ship.
06 RUN AGAIN
The loop closes; every pass raises the score.
// OBSERVABILITY

Log every run.
Chaos in, order out.

Every agent run, production or local, streams into RunAgain as a structured trace: every step, prompt, tool call, token and millisecond. Search across runs, diff any two, and replay the weird ones.

  • ✓ get started in 5 mins
  • ✓ full trajectories, not just the final answer
  • ✓ all the details in one place
00:01.2  span.start plan · model=claude-fable-5
00:01.9  tool.call web_search("return policy france")
00:02.8  tool.result 200 · 4 documents · 1.2kb
eval.inline faithfulness 0.94 ✓
00:03.1  tool.call db.query(orders.recent)
00:03.7  tool.result 200 · 12 rows
00:04.0  span.start generate
eval.inline grounding 0.41 ✗ · cited 30-day, source says 14-day
00:04.6  drift.check baseline Δ +0.9% ok
00:05.1  span.end run_a91f · 4.21s total
00:01.2  span.start plan · model=claude-fable-5
00:01.9  tool.call web_search("return policy france")
00:02.8  tool.result 200 · 4 documents · 1.2kb
eval.inline faithfulness 0.94 ✓
00:03.1  tool.call db.query(orders.recent)
00:03.7  tool.result 200 · 12 rows
00:04.0  span.start generate
eval.inline grounding 0.41 ✗ · cited 30-day, source says 14-day
00:04.6  drift.check baseline Δ +0.9% ok
00:05.1  span.end run_a91f · 4.21s total
● live trace ingest, illustrated
// EVALS ON AUTOPILOT

Setting up tests and evals is hard. So we do it for you.

RunAgain watches your traces and drafts the boring parts automatically: tests from real runs, mocks from recorded tool calls, eval suites from failure patterns. You review each suggestion and approve.

NEW TEST
refund-flow: user asks twice

12 production runs looped on duplicate refund requests. Drafted a regression test from run c9e0 with mocked stripe.refund.

NEW EVAL
faithfulness · LLM-as-judge

Answers started citing docs that weren't retrieved. Drafted a judge prompt scoring answer-to-context faithfulness on every run.

NEW MOCK
tool:search → recorded responses

search API is flaky (7% timeouts) and paid. Recorded 214 real responses so tests and experiments replay them for free.

// EVERYTHING BETWEEN "IT RAN" AND "IT WORKS"

What you get.

01
Get alerts and suggestions on Slack

Failed evals, tool errors and drift post to your channel as digests, with drafted tests and fixes ready to review.

02
Mocked environments

Run agents against tool responses simulated from previous runs: deterministic, fast, and no burned API credits.

03
Every eval technique

LLM-as-judge, rubrics, assertions, trajectory scoring, pairwise diffs, dataset regression, human review, or bring your own.

04
Auto-generated suites

Tests, mocks and eval suites drafted from your real traces. You review and approve.

05
Drift alerts

Get paged when quality, cost or latency drifts from baseline, before customers churn over bad behaviour.

06
MCP-first

RunAgain ships as an MCP server: run experiments, tune evals and approve suggestions straight from Claude, and export your best runs as fine-tuning datasets.

// INTEGRATE IN NO TIME

Connect. Trace. Monitor.

1Start from your repo
// we wire the SDK in with you during onboarding
→ one small diff to your agent entrypoint
✓ first trace within minutes
2Or do it your way
$ npm i runagain
// or let your agent do it:
// on npm: runagain·@runagain/ai-sdk
3Monitoring, out of the box
✓ traces flowing
✓ 4 built-in evals scoring
✓ drift monitor armed

Be first to run again.