Search documentation

Search the Fumadocs-backed documentation index.

Overview

What AXP does and where to start.

AI agents are a new kind of user for your product. AXP helps you find out how well they can actually use it.

Unlike eval tools that score model output against datasets, AXP runs sandboxed experiments that measure whether agents can operate your real product surfaces: CLIs, SDKs, APIs, web apps, MCP servers, and docs, in real complex environments.

The goal is to use those experiments to observe agents and turn agent behavior into repeatable product evidence.

It is built for product, DevRel, and engineering teams improving how agents use their product, and for teams evaluating competing products before they adopt them.

Use AXP to test new features, improvements, and bug fixes before you release them. You can answer questions like:

  • How do agents use your product?
  • Do changes to your product make agents better or worse at completing tasks?
  • Which models or coding agents perform best with your product? Which perform worse?

You can also run AXP experiments to evaluate competing products that you are considering adopting, buying, or investing in. You can answer questions like:

  • How does your product perform against an agent inventing a solution from scratch?
  • How does your product perform against your competition?
  • In the contexts your customers work in, do agents suggest your product?

These questions map to common experiment patterns: discovery and install, core user flows, product optimization, interface comparison, competitive analysis, and product marketing. See Experiment Design.

How AXP Works

01
Design an experiment
Choose the question, variants, and tests.
02
Run your experiment
Execute each variant in its own sandbox.
03
Analyze results
Compare variants and decide what to change.

Iterate: feed each result into your next experiment.

AXP has three main pieces:

  1. Design an experiment: define the task, the variants you compare (different agents, models, environments, or product versions), and the tests (application and introspection) that measure success.
  2. Run your experiment: execute each variant in its own sandbox on the AXP platform with axp run (local Docker is available with axp local run), capturing logs, file changes, test results, token usage, and cost.
  3. Analyze results: compare data across runs and variants so you can see what worked, what failed, and what to change next.
Experiments
The spec that defines the task, variants, setup, and tests you want an agent to run.
Runs
A completed execution of an experiment, including variant outputs, costs, logs, and artifacts.
$0.57
Avg cost
5m00s
Avg wall clock
321
Tool failures
100%
Test pass rate
Group by: Variant
X-axis: Wall clock
Y-axis: Cost
Cost vs wall clock, grouped by variant
0m00s
13m20s
Results
The summaries you generate from runs to compare variants and decide what to change next.

Installation

The fastest way to get started is the official AXP CLI installer:

bash <(curl -fsSL https://dl.514.ai/install.sh) axp
axp --help

Platform runs require an account (currently closed alpha). Request access, then sign in with axp auth login. See Installation for details.

Supported Coding Agents

AXP currently supports Anthropic Claude Code, OpenAI Codex, and Cursor.

Need support for another coding agent? Ask us to add it.

Next steps

  1. Install AXP and sign in.
  2. Run your first experiment in Getting Started.
  3. Design your own with Experiment Design.