Scenario 4 — CI/CD Integration

A regression catcher you can't run in CI is a regression catcher you won't run. Wire agentprdiff check into the same workflow as your unit tests.

The official GitHub Action (easiest)

The fastest path: one step that installs agentprdiff, runs check on your suites, and posts the behavioral diff — assertion flips, cost/latency deltas, output diffs — as a comment on the pull request, updated in place on every push:

.github/workflows/agents.yml

name: agent-regression
on: [pull_request]
permissions:
  contents: read
  pull-requests: write   # needed to post the behavioral-diff comment
jobs:
  agentprdiff:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: vnageshwaran-de/agentprdiff@main
        with:
          suites: "suites/*.py"
          runs: "1"              # bump to 3 + min_pass_rate for flaky cases
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          AGENTPRDIFF_JUDGE: anthropic

When suites use semantic() with a real judge, tell the action to install the matching SDK extra — the judge imports it lazily and a missing package turns every semantic assertion into a failure:

        with:
          suites: "suites/*.py"
          version: "[anthropic]"     # pip spec: agentprdiff[anthropic]

Reviewers see the diff where they already are — in the PR — instead of digging through job logs. Inputs: suites, version (pip spec, e.g. ==0.5.0), install (set "false" if your workflow already installs agentprdiff), root, runs, strict-judge (default "true"), comment, github-token, python-version; output regressed is "true"/"false". Requires agentprdiff ≥ 0.5.0. Notes: fork PRs get a read-only token, so the comment is skipped and the diff stays in the job log; the action pins nothing about your agent's dependencies — install those yourself before the action step, with install: "false" if they include agentprdiff.

Rolling your own workflow

.github/workflows/agents.yml

name: agent-regression
on: [pull_request]
permissions:
  contents: read   # least-privilege; GHAS flags workflows without this.
jobs:
  agentprdiff:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt agentprdiff
      - env:
          # Match whatever env var your production agent reads.
          OPENAI_API_KEY:    ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          # Make the semantic-judge mode explicit (don't rely on autodetection).
          AGENTPRDIFF_JUDGE:  anthropic
        run: |
          agentprdiff check suites/*.py --strict-judge --json-out artifacts/agentprdiff.json
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: agentprdiff, path: artifacts/ }

--strict-judge makes CI fail if the semantic judge silently fell back to fake_judge (e.g. a missing secret) instead of green-lighting keyword matching — the failure mode you least want to discover in production. It will become the default in v1.0.

Artifact upload happens on if: always() so a failed check still hands you the JSON to inspect locally. The regression panel printed to the terminal is preserved in the workflow log.

If you --json-out artifacts/..., add artifacts/agentprdiff*.json (or the broader artifacts/) to your project's .gitignore. The workflow upload doesn't prevent a contributor from accidentally git adding it locally.

GitLab CI

.gitlab-ci.yml

agentprdiff:
  image: python:3.11-slim
  stage: test
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  variables:
    AGENTPRDIFF_JUDGE: "anthropic"
  before_script:
    - pip install -r requirements.txt agentprdiff
  script:
    - agentprdiff check suites/*.py --json-out artifacts/agentprdiff.json
  artifacts:
    when: always
    paths: [artifacts/]

Set OPENAI_API_KEY / ANTHROPIC_API_KEY as masked CI/CD variables in your project settings.

CircleCI

.circleci/config.yml

version: 2.1
jobs:
  agentprdiff:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run: pip install -r requirements.txt agentprdiff
      - run:
          name: agentprdiff check
          command: |
            agentprdiff check suites/*.py \
              --json-out /tmp/agentprdiff.json
      - store_artifacts:
          path: /tmp/agentprdiff.json
workflows:
  version: 2
  pr:
    jobs:
      - agentprdiff:
          context: agent-secrets    # holds OPENAI_API_KEY etc.

Buildkite

.buildkite/pipeline.yml

steps:
  - label: ":robot_face: agentprdiff"
    command: |
      pip install -r requirements.txt agentprdiff
      agentprdiff check suites/*.py --json-out artifacts/agentprdiff.json
    artifact_paths: "artifacts/agentprdiff.json"
    env:
      AGENTPRDIFF_JUDGE: anthropic

What the JSON artifact looks like

One file per invocation, wrapping one envelope per suite:

{
  "reports": [
    {
      "suite": "customer_support",
      "mode": "check",
      "summary": {
        "cases_total": 4,
        "cases_passed": 4,
        "cases_regressed": 0,
        "has_regression": false
      },
      "cases": [
        {
          "suite_name": "customer_support",
          "case_name": "refund_happy_path",
          "trace": { "...": "full Trace JSON" },
          "grader_results": [
            { "passed": true, "grader_name": "contains('refund')", "reason": "..." }
          ],
          "delta": {
            "baseline_exists": true,
            "cost_delta_usd": 0.0,
            "latency_delta_ms": 12.3,
            "tool_sequence_changed": false,
            "output_changed": false,
            "assertion_changes": ["..."]
          }
        }
      ]
    }
  ]
}

Stable schema, easy to grep:

jq '[.reports[].summary.cases_regressed] | add' artifacts/agentprdiff.json

(Before 0.5.0 the file held a single suite's envelope at the top level — and each suite in an invocation overwrote the last. The reports wrapper fixed that; update old jq paths accordingly.)

Conditional skip when secrets are missing

You may want CI to warn instead of fail when an API key isn't set — useful in fork PRs where secrets aren't injected:

- env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  run: |
    if [ -z "$OPENAI_API_KEY" ]; then
      echo "::warning::OPENAI_API_KEY missing; skipping agentprdiff check."
      exit 0
    fi
    agentprdiff check suites/*.py --json-out artifacts/agentprdiff.json

The scaffolded workflow (agentprdiff scaffold <name>) does exactly this out of the box.

Pre-commit hook (local)

.pre-commit-config.yaml

- repo: local
  hooks:
    - id: agentprdiff
      name: agentprdiff
      entry: agentprdiff check suites/*.py
      language: system
      pass_filenames: false
      stages: [pre-push]

Stage as pre-push rather than pre-commit so a single noisy edit doesn't re-run a heavy suite on every save.

Updating baselines from a PR

Two reasonable workflows:

1. Author re-records. Checkout the branch, run agentprdiff record suites/*.py, commit the resulting JSON diff under .agentprdiff/baselines/, push. Reviewers see the trace deltas in the normal PR diff.
2. Bot-driven re-record. A /regen-baselines slash-command on the PR triggers a workflow that runs agentprdiff record, opens a follow-up PR with the updated baselines, and links it from the original PR. Useful in larger teams where author re-records get forgotten.

Either way, the PR diff under .agentprdiff/baselines/ is the review surface. Don't auto-accept new baselines silently — that defeats the whole point.