GitHub CI

Connect a GitHub repository to an AXP experiment so that commenting /axp run on a pull request runs that experiment against the PR's code and reports the results right in the PR — a rich comment plus a non-blocking status check.

This is the CI/PR trigger: the same run engine as axp run, driven by a PR comment instead of your terminal, so agent-experiment results sit next to code review.

How it works

1. Whenever the PR's code changes, your repo's CI builds the product and uploads it to AXP with axp ci upload, keyed to the commit.
2. A developer comments /axp run on the PR.
3. AXP posts instant feedback (a "queued" comment and a pending axp run check), then waits — up to ~20 minutes — for that commit's build to arrive (the rendezvous).
4. AXP runs the repo's linked experiment against that build, then edits the comment into a per-variant result and flips the check to pass or fail.

A run triggered from GitHub has no developer laptop to stage local files from, so the PR's software reaches the sandbox as a content-addressed artifact that CI uploads. The experiment consumes it through a source-less files: slot — a files entry with a name and dest but no source. CI fills that slot per commit; everything else about the experiment is unchanged.

Prerequisites

• An AXP account with the CLI signed in (axp auth login). See Installation.
• An experiment that tests your product.
• Admin on the GitHub organization or repository, to install the AXP GitHub App.

Set up a repository

Setup is one-time per repository. After it, any PR accepts /axp run.

1. Install the AXP GitHub App

In the AXP platform, open your organization's Settings → GitHub, choose Connect, and install the AXP GitHub App on the account and repositories you want. When you return from GitHub, the installation is linked to your active AXP org automatically.

The link is owned by your org, so it survives the App being uninstalled and reinstalled. The App requests only what the trigger needs: read repository contents (to read the experiment at the PR's commit), write pull-request comments, and write status checks.

2. Declare a build slot in the experiment

Add a source-less files: entry to the experiment — a slot with no source, which CI fills per commit:

files:
  - name: cli # the slot handle you pass to `axp ci upload`
    dest: tools/mycli # where the build lands in each variant's /workspace

Reference tools/mycli from the experiment's setup and prompts exactly as you would any staged file. Declare one slot per product artifact your CI builds.

3. Register and link the experiment

Register the experiment on the platform, then point the repo's PRs at it:

axp push my-experiment.yaml
axp integrations gh link my-experiment owner/repo

axp push uploads the experiment definition without running it (the experiment id comes from the YAML — here my-experiment). axp integrations gh link then points /axp run on owner/repo's PRs at that registered experiment. The experiment must already be registered, so push first; re-linking a repo overwrites its target. Inspect the current links with axp integrations gh list. See axp integrations.

4. Build and upload the product from CI

Add a workflow that, on each pull request, builds the product and runs axp ci upload once per source-less slot. axp ci upload resolves the slot's workspace destination from the linked experiment and authenticates with an org API key stored as an Actions secret named AXP_API_KEY.

name: axp
on:
  pull_request:

jobs:
  upload-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          # Build the PR's head commit — the commit AXP keys the artifact to.
          ref: ${{ github.event.pull_request.head.sha }}

      - name: Build the product
        run: cargo build --release # produces ./target/release/mycli

      - name: Install the AXP CLI
        run: |
          bash <(curl -fsSL https://dl.514.ai/install.sh) axp
          echo "$HOME/.fiveonefour/bin" >> "$GITHUB_PATH"

      - name: Upload the build for the PR trigger
        run: axp ci upload cli ./target/release/mycli --commit "${{ github.event.pull_request.head.sha }}"
        env:
          AXP_API_KEY: ${{ secrets.AXP_API_KEY }}

axp ci upload reads the repo from $GITHUB_REPOSITORY automatically (set in GitHub Actions); override it with --repo owner/repo elsewhere.

Generate the AXP_API_KEY value from your organization's API-key settings in the platform, and add it as a repository (or organization) Actions secret. axp ci upload re-uploads nothing when your org already has an identical build, so running it on every commit is cheap.

Trigger a run

Comment /axp run on a pull request. The command must be on its own line and exactly /axp run — trailing arguments are not accepted.

AXP immediately:

• posts (or updates) its comment to **axp run** ⏳ queued — waiting for the build to upload…, and
• creates a non-blocking axp run status check on the PR's head commit.

It then waits for CI to upload every build slot for that commit, runs the experiment against the build, and edits the comment into a result when the run finishes.

Anyone with write access to the repo can trigger a run — the comment author must be an OWNER, MEMBER, or COLLABORATOR. Bot and agent accounts can trigger it too, so /axp run works from your own automations.

Read the results

When the run finishes, AXP edits its comment in place into:

• a headline — Experiment set completed. or Experiment set failed. with a See results → link to the run on the platform;
• a one-line summary — 1/1 variants passed · 5/5 tests · $0.15 · 2.5k tokens; and
• a per-variant table with inline deltas versus the variant's most recent prior run:

| Variant     | Status | Tests | Cost          | Tokens      | Runtime      |
| ----------- | ------ | ----- | ------------- | ----------- | ------------ |
| claude · p0 | ✅     | 5/5   | $0.15 (-0.01) | 2.5k (-0.5k)| 4m 3s (-1m)  |

A variant passes only if it ran cleanly and every test passed; a variant that errored, was canceled, or produced no results shows as a failure row. The axp run check mirrors the same verdict.

The check is non-blocking by default. To make a green /axp run required before merge, mark the axp run check required in the repository's GitHub branch-protection rules — no change on the AXP side.

Re-run, and one run per PR

Re-run the experiment either by commenting /axp run again or by clicking Re-run on the axp run check.

Each PR keeps a single run. Starting a new run — or pushing a new commit to the branch — cancels any run still in flight and marks its old comment and check as superseded. Runs always execute against the PR's head commit, so a re-run after a new push evaluates the new code (once CI uploads its build).

Troubleshooting

See also

• axp ci upload — the CI-side command reference.
• axp integrations — inspect integrations and link repos.
• Files — source-less slots and how artifacts are staged.
• Runs — what a run captures, and remote vs. local execution.