Experiment YAML

Experiment YAML is the file format passed to axp run and axp local run.

The current parser accepts schema_version: 2 and rejects unknown fields at every level. For the canonical machine-readable schema, use experiment.v2.schema.yaml.

Schema URL

Use the version-pinned schema URL for editor validation:

# yaml-language-server: $schema=https://docs.514.ai/schema/experiment.v2.schema.yaml

Published schema URLs:

axp experiment schema prints the latest supported schema to stdout.

Field index

Minimal example

# yaml-language-server: $schema=https://docs.514.ai/schema/experiment.v2.schema.yaml
schema_version: 2
id: cli-install
name: "CLI install"

agents:
  - name: claude
    model: anthropic/claude-sonnet-4.6

prompts:
  - id: install
    prompt: |
      Install the CLI and write its version to /workspace/version.txt.

products:
  - name: cli
    type: CLI
    setup: npm pack

tests:
  application:
    - name: version-file-exists
      script: test -f /workspace/version.txt

limits:
  max_turns: 25
  max_time_seconds: 300
  max_cost_usd: 0.50

Top-level experiment

An experiment defines what you want to learn about agents using a product surface. It names the agents, prompts, products, environments, and tests AXP uses to compute and run variants.

Top-level environment_variables, secrets, and files apply to every variant. MCP servers and setup checks are setup-owned fields.

Variant axes

Variant axes are the experiment inputs AXP combines into runnable variants. This lets you define the dimensions you care about once, then compare results by agent, prompt, environment, and product.

variants = agents × prompts × environments × products

environments and products are optional. If an axis is absent, it is omitted from the variant coordinate.

Agents

agents defines which coding agents run the experiment. Each agent can use its default model or pin an explicit model and model controls.

Accepted shapes:

# String form: one agent, provider-default model.
agents: claude

# List of strings: multiple agents, provider-default models.
agents:
  - claude
  - codex

# Object form: one agent with an explicit model.
agents:
  name: claude
  model: anthropic/claude-sonnet-4.6

# List of objects: multiple explicit agent/model pairs.
agents:
  - name: claude
    model: anthropic/claude-sonnet-4.6

model is optional but recommended. If omitted, AXP uses the provider-default model for that agent.

Agent object

Model object

A model object configures the model used by an agent. Optional controls are passed through when the selected agent supports them.

agents:
  - name: claude
    model:
      name: anthropic/claude-opus-4.8
      effort: high
      context_window_size: 1M
      thinking: true
  - name: codex
    model: openai/gpt-5

Prompts

prompts defines the tasks agents try to complete. Each resolved variant receives one final prompt.

Accepted shapes:

# String form: one prompt.
prompts: "Build the report."

# List of strings: multiple prompts.
prompts:
  - "Build the report."
  - "Build the report and explain your steps."

# Object form: list of named prompts.
prompts:
  - id: detailed
    prompt: "Build the report and explain your steps."

Prompt object

A prompt object gives a task stable metadata, so Results can filter and group by the prompt that produced each run.

Bare prompt strings get positional ids: p0, p1, and so on.

Environments

environments defines the sandbox conditions around the agent, such as installed tools, fixtures, or external integrations. Use environments when you want to compare how agents perform under different surrounding conditions.

Accepted shapes:

# String form: one setup script with a generated environment name.
environments: "pip install -r requirements.txt"

# Object form: one named environment.
environments:
  name: workspace
  setup: "pip install -r requirements.txt"

# List form: multiple named environments.
environments:
  - name: workspace
    setup: "pip install -r requirements.txt"

Environment object

An environment object names one sandbox setup condition. Its setup prepares the sandbox before the agent receives the prompt.

Bare environment strings are shorthand for an environment whose setup is that string.

Products

products defines the agent-facing surface under test. A product can be a CLI, API, MCP server, SDK, docs surface, or other tool the agent uses to complete the prompt.

Accepted shapes:

# String form: one setup script with a generated product name.
products: "npm install -g my-cli"

# Object form: one named product.
products:
  name: my-cli
  type: CLI
  version: "1.2.0"
  setup: "npm install -g my-cli"

# List form: multiple named products.
products:
  - name: my-cli
    type: CLI
    version: "1.2.0"
    setup: "npm install -g my-cli"

Product object

A product object names what you want to compare. Product metadata is recorded so Results can filter and group by product and product version.

Product type

Allowed values: CLI, MCP, API, Skill, SDK, Schema, Docs, Marketing, Agents.md, Other.

Setup

setup prepares the sandbox for an environment or product. Use setup scripts to install tools, create fixtures, start services, or expose resources the agent needs.

Each variant has its own isolated /workspace. A setup script cannot rely on files or side effects created by another variant.

Accepted shapes:

# String form: one setup script.
setup: "npm install"

# List of strings: multiple setup scripts, run in order.
setup:
  - "npm install"
  - "npm test -- --help"

# Object form: one named setup with optional scoped resources.
setup:
  name: install-cli
  script: "npm install"

# List of objects: multiple named setups, run in order.
setup:
  - name: install-cli
    script: "npm install"
  - name: smoke-cli
    script: "npm test -- --help"

# Mixed list: strings and setup objects can be combined.
setup:
  - "npm install"
  - name: smoke-cli
    script: "npm test -- --help"

Setup object

A setup object is the named form of setup. Use it when setup needs its own files, environment variables, MCP servers, or setup checks.

When both a product and environment contribute setup, product setup runs before environment setup.

Setup checks

setup_checks verify that setup produced a usable sandbox before the agent starts. A failed setup check stops that variant before any agent work happens.

setup_checks:
  - name: cli-on-path
    script: my-cli --version

Setup check object

MCP servers

mcp_servers exposes MCP servers to the agent for variants that use this setup. Use this field when the product surface or environment includes an MCP tool.

mcp_servers:
  - name: fixture-sentinel
    type: stdio
    command: /workspace/fixture-mcp.py
    args: []
  - name: axp
    type: http
    url: http://localhost:3001/mcp

MCP server object

An MCP server object defines one server and its transport. The required connection fields depend on type.

Transport-specific rules:

• stdio uses command, optional args, and optional env.
• http and sse use url and optional headers.
• Mixing stdio-only and endpoint-only fields is rejected.

MCP stdio env entry

env forwards declared secret values to a stdio MCP process. Each entry is either a bare secret name or an object:

env:
  - GITHUB_TOKEN
  - name: GH_AUTH
    from: GITHUB_TOKEN

MCP HTTP and SSE header object

headers attaches HTTP headers to an http or sse MCP server. Use placeholders when a header needs a declared secret value.

headers:
  - name: Authorization
    value: "Bearer ${SUPABASE_SERVICE_ROLE_KEY}"

MCP env entries and header placeholders reference environment variable names visible to the variant. Values can come from literal environment_variables, axp://secrets/<slug> references, or the deprecated secrets list.

Rules enforced at axp experiment validate time:

• Every stdio env[*].from and every ${NAME} placeholder in a header value must reference a name visible to the variant.
• Bare $NAME is treated as a literal; only ${NAME} is a placeholder.
• HTTP header names are unique per server, case-insensitively.
• command / args / env are only valid for type: stdio; url / headers are only valid for type: http / sse.

Resolved secret values are written into the agent session frame and can appear in run artifacts. Treat artifacts as sensitive whenever an experiment forwards secrets to MCP servers.

Extensions

extensions refine the variant set when the base axes do not express the combinations you need. Use extensions to narrow an axis, swap products or environments for a subtree, or append prompt guidance to a slice of variants.

If an experiment declares any extensions, AXP creates only extension-derived variants.

Extension object

An extension object is one node in the refinement tree. Nested extensions are recursively cross-multiplied with their parents.

prompts:
  - id: analyze
    prompt: "Read /workspace/task.md and write /workspace/report.json."

extensions:
  - id: with-cli
    products:
      - name: cli
        type: CLI
        setup: "curl -fsSL https://clickhouse.com/ | sh"
    prompts: ["Use the ClickHouse CLI for the analysis."]
  - id: without-cli
    products:
      - name: no-cli
        setup: "true"
    prompts: ["Do not use the ClickHouse CLI; use another local method."]

Environment variables

environment_variables injects environment variables into variants at runtime. Use it for non-secret runtime configuration and supported secret references.

environment_variables:
  - name: LOG_LEVEL
    value: debug
  - name: GITHUB_TOKEN
    value: axp://secrets/prod-gh
  - name: DATABASE_URL
    value: axp://secrets/staging-db

Environment variable object

Store secret values once with axp secrets set <slug>, then reference them with axp://secrets/<slug>. Both axp run and axp local run resolve these references before injecting env vars into the sandbox.

A referenced slug that does not exist in your org fails at preflight.

Reserved names:

• ANTHROPIC_API_KEY
• ANTHROPIC_BASE_URL
• OPENAI_API_KEY
• OPENAI_BASE_URL
• CURSOR_API_KEY
• MODEL
• MAX_TURNS
• IS_SANDBOX
• TRACEPARENT
• any name beginning with AXP_, CLAUDE_CODE_, CODEX_, CURSOR_, or OTEL_

Files

Top-level files stages host files or directories into every variant's /workspace before setup runs.

setup.files is accepted in experiment YAML, but current runs only stage top-level files. Put host file staging at the top level when you need the files delivered during a run.

files:
  - name: my-cli
    source: ../build/mycli
    dest: tools/mycli
  - source: https://example.com/fixtures/data.bin
    sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    dest: fixtures/data.bin

File entry object

Notes:

• Directory sources copy their contents under dest/.
• File sources land as a single file at dest.
• URL sources must be publicly fetchable and land as a single file at dest; unpack archives in setup if needed.
• Sandboxes are Linux containers; macOS binaries staged from a laptop will not run there.
• Directory walks honor .axpignore, not .gitignore.
• --file NAME=SOURCE binds or overrides the source of a named entry.
• --file SOURCE::DEST stages an ad-hoc entry into every variant.
• Missing or unbound sources abort real runs at preflight. --resolve-variants renders MISSING / UNBOUND annotations without failing.
• Staging failures roll the variant up as status=error / exit_reason=staging_failed; the rest of the run keeps going.
• Platform and local runs both stage top-level files.
• Treat experiment YAML like a script you run: sources are read with your permissions and may point anywhere on the host.

Tests

tests defines how AXP scores each run. Application tests check the output state; introspection tests check how the agent got there.

tests:
  application:
    - name: report-exists
      script: test -f /workspace/report.json
  introspection:
    - name: under-thirty-tool-calls
      script: '[ "$(jq ".tool_calls | length" "$AXP_TRACE_PATH")" -lt 30 ]'

Test object

Limits

limits sets the run caps for each variant execution.

Limits object

Secrets deprecated

secrets is a deprecated alias for host environment variable names injected into variants at runtime.

Prefer environment_variables for literal values. Some MCP secret-forwarding fields still reference declared secret names.

secrets:
  - GITHUB_TOKEN

Secret names must match ^[A-Z_][A-Z0-9_]*$.

Validation rules

An experiment is invalid if:

• the YAML contains a field not defined by the schema
• schema_version is not 2
• any required field is missing
• an id that must be kebab-case is not kebab-case
• an agent model id contains ::
• a declared axis is empty
• duplicate ids or names appear where uniqueness is required
• the resolved variant set is empty
• a resolved variant has an empty prompt
• two resolved variants collide on variant_id
• no tests are defined
• test names are duplicated
• an env var or secret name is invalid or reserved
• a file entry has an invalid name, source, sha256, or dest
• an MCP server mixes transport-specific fields
• an MCP server references a secret not visible to the variant
• any limit is not greater than zero

YAML syntax boundaries

The experiment data model is JSON-compatible even though the authoring file is YAML.

• YAML comments are allowed.
• YAML anchors and aliases are allowed when the resolved value is JSON-compatible.
• Custom YAML tags are unsupported.
• Non-string mapping keys are unsupported.