# RunAgain

> The self-improving loop for AI agents. RunAgain traces every run in production, scores what changed, debugs and replays it, and turns every issue into a test that outlives the next change.

RunAgain (runagain.ai) is an observability, testing and evaluation platform for AI agents. Get a demo: https://cal.com/tamas-szuromi/30min

## The feedback loop

1. **Trace**: log every agent run; every span, tool call and token, captured live from production. Zero-config capture from the Vercel AI SDK and Claude Agent SDK, or plain HTTP.
2. **Experiment**: fork prompts, models and tools behind flags. Replay any run in a mocked environment; tool responses are simulated from previous runs, so iterations are fast and free.
3. **Test**: deterministic runs against mocks and sims. Recorded mocks freeze flaky and paid APIs.
4. **Evaluate**: LLM-as-judge, rubrics, assertions, trajectory scoring, pairwise comparison, dataset regression and human review.
5. **Improve**: diff regressions, fix the weak spans, export your best runs as fine-tuning datasets, and get alerted the moment quality, cost or latency drifts.
6. **Run again**: the loop closes; every pass raises the score.

## Why

- Improving an agent is manual work today: you test by hand and browse traces to find where the agent went off task, where a tool call failed, where the answer quietly got worse.
- Then you change a prompt, a model, a tool. Everything underneath moved too, so there is nothing left to compare against. And once the change ships, nothing keeps measuring whether it still works, or whether it actually got better.
- RunAgain closes that gap: trace what happened, evaluate what changed, replay it exactly as it ran so the two are comparable, turn every issue into a test, improve the agent automatically, and measure it against your goals and real conversions. You build the agent. We build the loop around it.

## Solutions

- [Agent observability](https://runagain.ai/solutions/agent-observability): OpenTelemetry traces for agents covering every span, tool call, token and cost, searchable and diffable; multi-turn sessions.
- [Monitoring & alerting](https://runagain.ai/solutions/agent-monitoring-alerting): PSI drift detection, per-agent run baselines, event-driven Slack alert digests.
- [Debug AI agents](https://runagain.ai/solutions/debug-ai-agents): likely failure cause named per run, diffs against the last good run, trace-to-regression-test in one step.
- [Enrich agent traces](https://runagain.ai/solutions/enrich-agent-traces): attach user/session/tier/version metadata as typed queryable fields; scope evals and alerts by business data; capture outbound HTTP via the sandbox proxy.
- [Simulation & evals](https://runagain.ai/solutions/agent-simulation-evals): LLM judges, assertions, rubrics, dataset and tool-trajectory scoring, online with scoping and sampling, plus human annotation with judge alignment.
- [Mocked environments](https://runagain.ai/solutions/mocked-environments): record real tool/API responses as fixtures, replay deterministically, gate CI on score regressions.
- [Experimentation](https://runagain.ai/solutions/agent-experimentation): experiments over versioned datasets with Wilson confidence intervals and McNemar significance testing.
- [Continuous improvement loop](https://runagain.ai/solutions/continuous-improvement-loop): traces → datasets → experiments → CI gates, with monitors holding the line.
- [Goal optimization](https://runagain.ai/solutions/agent-goal-optimization): define business outcomes as scores, segment by your data, prove candidates against the goal.

## Markdown mirrors

Every content page on this site has a markdown twin: append .md to its URL. Examples: [index.md](https://runagain.ai/index.md), [solutions.md](https://runagain.ai/solutions.md), [use-cases.md](https://runagain.ai/use-cases.md), [evals.md](https://runagain.ai/evals.md), [for-agents.md](https://runagain.ai/for-agents.md), [solutions/agent-observability.md](https://runagain.ai/solutions/agent-observability.md). Served as text/markdown; ideal for agents and LLMs that prefer clean text over HTML.

## Eval catalog

The full list of supported evals, scorers and judges lives at [runagain.ai/evals](https://runagain.ai/evals): 12 heuristic scorers, 11 built-in LLM judges, 6 RAG and grounding judges, 8 retrieval ranking and search metrics, 3 embedding scorers, 4 online eval kinds, plus rubrics, dataset scoring and human annotation with judge alignment.

## Use cases

- [LLM cost tracking](https://runagain.ai/use-cases/llm-cost-tracking): cost computed on every span from token counts (cache and reasoning included), attributed by user, session, agent, version or any metadata.
- [Reduce LLM costs](https://runagain.ai/use-cases/reduce-llm-costs): baselines flag abnormally expensive runs, drift monitors catch token growth, experiments prove the cheaper config scores as well.
- [Time-travel debugging](https://runagain.ai/use-cases/time-travel-debugging): full traces preserve every prompt and tool result; recorded fixtures replay deterministically; failing runs become permanent tests.
- [Regression testing in CI](https://runagain.ai/use-cases/agent-regression-testing-ci): the runagain-eval CLI gates merges on per-scorer thresholds and McNemar-significant regressions.
- [Prompt A/B testing](https://runagain.ai/use-cases/prompt-ab-testing): variants over versioned datasets, scored and compared with Wilson intervals and significance tests.
- [Hallucination detection](https://runagain.ai/use-cases/hallucination-detection): factuality and faithfulness judges score sampled live traffic; grounding failures alert to Slack.
- [Conversation tracing](https://runagain.ai/use-cases/multi-turn-conversation-tracing): set a session id and turns roll up into replayable sessions with per-conversation cost and scores.
- [LLM-judge calibration](https://runagain.ai/use-cases/llm-judge-calibration): human annotation queues paired against judge scores; the alignment table measures agreement.
- [Tool-call failure analysis](https://runagain.ai/use-cases/tool-call-failure-analysis): trajectory scoring with recovery invalidation, failure classification, tool-error digests.
- [Structured output validation](https://runagain.ai/use-cases/structured-output-validation): property checks on JSON answers plus shape-drift alerting.
- [Tracing coding agents](https://runagain.ai/use-cases/tracing-coding-agents): Claude Code sessions become traces with prompts, tool calls and accurate cost, zero code changes.
- [Agent & LLM alerting](https://runagain.ai/use-cases/agent-llm-alerting): event-driven Slack digests for failed evals, tool and trace errors, baseline anomalies and drift.

## Integrations

- Vercel AI SDK (zero-config tracing)
- Claude Agent SDK (zero-config tracing)
- MCP: RunAgain ships as an MCP server, so you can run experiments, tune evals and approve suggestions straight from Claude, and export runs as fine-tuning datasets
- REST API for everything else

## For agents

[For agents](https://runagain.ai/for-agents) is a reference page for assistants and crawlers reading this site: an accurate one-line description of RunAgain, the markdown mirrors, a machine-readable manifest, and the HTTP contact endpoint. It is documentation about the site, not instructions to the reader.

## Pricing

- **Free**: 1,000 events / mo, 14-day data retention, alerts for failures, experiments, the self-improving agents loop, 50 evaluations.
- **Pro ($399/mo)**: everything in Free, 1,000,000 events / mo, 6-month data retention, shared Slack channel, self-improving agents loop, unlimited evaluations.
- **Custom**: everything in Pro, custom event volume and retention, SSO, audit log, custom evals, custom SLAs.

Full detail and an FAQ (data handling, setup effort, what counts as an event, limits, frameworks, replay side effects, custom evals, the self-improving loop): [Pricing](https://runagain.ai/pricing)

## Links

- [Website](https://runagain.ai)
- [Documentation](https://docs.runagain.ai/)
- [Pricing](https://runagain.ai/pricing)
- [npm: runagain](https://www.npmjs.com/package/runagain)
- [npm: @runagain/ai-sdk](https://www.npmjs.com/package/@runagain/ai-sdk)
- [Book a demo](https://cal.com/tamas-szuromi/30min) (30 minutes, for humans)
- [Get in touch](mailto:tamas@runagain.ai)
- [For agents](https://runagain.ai/for-agents)

## Contact

- Email: tamas@runagain.ai