Test each pull request like a user with an AI tester

Tested deploys put an AI tester on each pull request. It opens the pull request's branch preview in a browser, signs in, uses what the pull request changed the way a person would, and writes checks that run again on later pushes. The result shows on the pull request as the ox tests check, and with tests set to On, production deploys only once staging passes the saved checks. Everything here is also an ox tests command (the full list).

Turn tested deploys on

Tested deploys are off until you turn them on, first for your account in Settings, Beta features, then for a project. They work on every plan, Free included. Before ox lets you, the project needs:

• Branch previews on, since the tester uses the pull request's preview.
• A repository connected through the ox GitHub App, since runs follow pull requests.
• A staging copy that tracks the branch production deploys, where every commit bound for production is checked first.
• A server with room for the test browser and one preview in the half of its memory that previews share. ox names the size it needs when there is not.

The first time, add --accept-data-sharing, which agrees that the AI tester reads your pull requests and previews (the terms list what is sent):

ox settings beta tested-deploys on
ox tests shop on --accept-data-sharing
ox tests shop

There are three settings. On holds the deploy while a check is red. Report, shown as "tell me, never hold", runs the same tests and only tells you. Off stops them. Turning tests off, for the project or for your account, completes the ox tests check a stopped run had open as neutral, so a required check does not block the merge. ox tests shop prints the setting, the testing credit, how signed-in testing works, GitHub's status, the held commit, the queue and the last runs.

Tell the tester what your users do in one sentence or a few, up to 500 characters. It reads this as data:

ox tests shop about "Shop owners list products, and buyers pay with a card."

Let the tester sign in

Most apps keep their real pages behind a sign-in, so the tester signs in before it tests. It tries these ways in order, and ox tests shop signin says which one works or what to do. ox tests shop signin check proves it again with a run.

1. Test accounts from the copy's variables

Many apps make an admin from variables when they start, such as ADMIN_EMAIL and ADMIN_PASSWORD. Name those variables once, and ox signs in with them, once per run on each copy, so an app that limits sign-ins per account keeps letting the tester in. You give the variable names, never a value: ox reads the values from the copy's variables, never shows them to the AI, and refuses any other text before it sends anything.

ox tests shop accounts set name=admin,role=admin,email-var=ADMIN_EMAIL,password-var=ADMIN_PASSWORD
ox tests shop accounts

A project has at most 4 accounts. A name is lowercase letters, digits and hyphens, starting with a letter, up to 12 characters. Use email=ADDRESS when the address is not secret, and user-var when your app signs in with a user name instead: staging then gets a made user name, not a tester@ address. Use user=NAME when the user name is fixed in your app, such as an admin always named admin: a user name is not secret, so it needs no variable. A preview's database starts empty, so this works when your app's release or seed step makes that account from those variables. Make that step safe to run on every deploy, and have it look the account up by its own email, or by its name when your app signs in with a user name. A step that makes the admin only on the app's first start works on staging too: a new staging copy's first deploy already holds its own value for each of these variables production keeps to itself or lacks, the one you give or one ox makes. When staging was already running before such a variable got its value, the Tests card says so and offers Remake staging.

2. A sign-in helper command

The sign-in helper is a command that runs on your server with your app's settings and makes a test user. ox gives it OX_USER_NAME, OX_USER_EMAIL, OX_USER_PASSWORD and OX_USER_ROLE, and it must exit 0. It runs again on later runs, so it should succeed when the user is already there.

ox tests shop users "python manage.py make_test_user"
ox tests shop users --clear

For an app that signs in with a link or a code sent by email, or only with another site, let the helper print a sign-in link your app honors as its last line, OX_SIGNIN_URL= followed by the link. The link must be on the copy's own host. ox opens it instead of filling in a form.

3. The app's own sign-up

When anyone can sign up, the tester makes its users through your app's sign-up form.

When none of these works, the run still tests the pages a visitor sees, and its summary starts with signed-in pages not tested and the reason.

When a check the tester wrote for the change could not reach the app because its user did not sign in during that run, the run is not a plain pass either: its summary starts with signed-in pages not tested, the user that failed and the fix, and the check on GitHub is neutral. The run still passes, so production is not held.

Keep preview keys test-only

A preview starts with production's variables, and the tester clicks what a user would, so a payment or mail key there must be a test key. While a copy still holds production's value of a key that looks like a credential, its runs wait. Mark the key production only, give the copy its own value, or confirm it is a test key:

ox tests shop keys
ox tests shop keys confirm MAIL_KEY

A live Stripe key, or a value that names production's address, can never be confirmed.

Read a run, then fix it or mark it Not a bug

A run's summary says what a user would hit, what was checked and works, and what was not tested. A pass that tested none of the change, unless the tester said nothing shows, starts with Not tested: no check covers the change and its GitHub check is neutral. A change that touches a worker or cron job, when the tester wrote no check, is not a plain pass: the summary says Not tested: a background job changed and names the job, and the check on GitHub is neutral. When the change is too large to list every file, it says a job may have changed. That never holds production. For each red check, ox tests shop run ID prints the error, what was expected and received, the failing step's line and the page's text at the failure. ox tests shop artifacts RUN DIR saves the failed checks' pictures.

A red check holds production at the commit that has it. When the red is a bug, push a fix. When it is not a bug, mark it Not a bug, with a reason of up to 200 characters if you want to give one. Once every red of the run is marked, the run counts as passed and the held deploy goes ahead. A check the pull request proposed then leaves the branch. ox remembers the exact result you accepted: the page, the element and the two values. After the merge, a red with that same result anywhere in the project reads "matches Not a bug" and does not hold anything; any other failure stays red. Undo it on the Tests card to be asked again. ox counts each Not a bug as a wrong red when it measures how often the tester is wrong.

ox tests shop runs --branch fix/login
ox tests shop not-a-bug 412 signup.spec.mjs --reason "The new limit is on purpose."
ox tests shop not-a-bug 412 --all

A held production says why and what to do on the project's overview, in its Projects row, on the Tests card, in ox tests and in the alert, in the same words:

Production is held at 3f9a2c1: signup.spec.mjs failed on staging. Bug? Push a fix. Not a bug? Not a bug deploys it (ox tests shop not-a-bug 412 --all).

When the pull request means to change what a saved check expects, such as a button's label, the tester updates the check without asking you. Its title, description or a commit message must name each new value in words the tester quotes, the old check must still pass on staging and the new one fail there, and the new one must pass 3 times out of 3 on the preview. A run updates at most 12 checks asserting at most 5 new values, so a rename updates every check it touches, its summary opens with each update and the words it quotes, and the ox tests check shows neutral rather than green, so you read it before you merge.

A commit with nothing to test does not wait. When the project has no saved checks yet, or staging serves no web address, the run on staging ends Skipped, and production deploys the commit, not tested. The run on the Tests card says why it was skipped.

A preview runs its branch merged into main (what a preview runs), so a branch that is behind main still runs the checks main saved. When the branch conflicts with main, those checks cannot run there: the summary says "3 saved checks not run: conflicts with main", and the ox tests check is neutral, never green. Resolve the conflict and push. When main saves checks after the preview was built, the summary says to push again to run them.

When staging serves a commit that lacks the commits that saved some checks, such as after the branch was rewritten, the run leaves those checks out, says how many, and holds the commit. Check again would leave them out again, so push a commit that has them, pause them, or use Deploy anyway.

When a held commit has no red to judge, such as when staging failed to deploy, ox tests shop deploy-anyway --reason R lets it ship untested once. A commit held by a red is refused, and the refusal says to use Not a bug. A commit that staging will not deploy by itself, or that waited for staging for over 150 minutes, longer than a head run waits for a slot and runs, is held that way too, and ox alerts you. quarantine pauses a saved check and restore resumes it.

The check on GitHub

When the ox GitHub App may write checks (checks: write), each pull request's run shows on it as the ox tests check, and marking a check Not a bug turns it green. When a run is red, the check says that merging with it red holds production until a fix passes or someone clicks Not a bug, with a link to the run in the console. Without that permission, the Tests card asks you to accept Checks: Read and write in the ox GitHub App's permissions. For a public repository or a password-protected copy, the check holds only the state, the counts, and each red check's name and finding in a user's words.

Cron jobs

When a project has cron jobs, the tester can run them. For a change to a job, it runs the job on the preview, checks its exit status, then checks what the job should have changed on a page or through the app's API. A saved check that runs a job is replayed like any other, so each copy it runs on runs its own jobs.

A job runs at most 5 times in one batch of checks, one at a time, and ox waits up to 30 seconds for it to end, never past the time the check has left. A job that exits with an error is the check's answer: a check that expects success is red. A job ox could not run, because it is already running, the project is stopped or busy with a deploy, it did not end within 30 seconds or before the check ran out of time, or no job has that name, is reported as an infrastructure error and never as a red check. ox shows the tester a job's exit status and never its output. A saved check that runs a job is left out of a replay on a server whose ox agent is too old to run jobs, and the run says so.

What it costs

AI testing is prepaid credit at cost: you pay what the model costs ox, with no markup. Credit never expires and is not refunded. Each pull request's run uses what its AI calls cost, from your account's testing credit, which you buy USD 100 at a time in Settings or with ox plan credit. When your credit is less than the most a run may cost, or at USD 10 of AI in a day for your account, a pull request's run only replays the saved checks; at a daily limit for all of ox, runs wait for the next day. Production deploys never wait on your credit. ox emails you once when your credit falls to 30% or less of what you had after your last top-up.

ox plan usage
ox plan credit

For AI coding agents

A coding agent can push, wait for the run and read what failed, all with --json:

ox tests shop run --branch fix/login --commit "$(git rev-parse HEAD)" --wait --json
ox tests shop artifacts 412 ./shots

run --wait exits 0 when the run passed (Not a bug included), 1 when it failed, 5 when it ended in error, 6 when it was skipped or superseded with no run after it, 7 when the wait ended before the run did, and 8 when the plane refused. Any token can read runs. A token with deploy access can also run again, and every other change needs full access, so an agent that pushes and waits cannot mark its own red Not a bug.

What is sent and kept

The checks run on your own server against the preview and the staging copy, never against production, and ox never stores a test user's password. To write and judge the checks, ox sends the pull request's text and changes, and the text of the preview's pages, to DigitalOcean's AI inference. The privacy policy lists all of it, with how long each piece is kept, and Security covers how the copies are kept apart.