GitHub App
Outqube CI runs your tests on pull requests and reports an Outqube check and one comment per pull request. Permissions, settings, URL modes and check states.
Outqube CI is a GitHub App. It runs an app's tests on your pull requests and reports the result as a check named Outqube, with one comment per pull request. To set it up step by step, see Test every pull request with the GitHub App.
Permissions
The app asks for the least it needs. The webhooks it receives are pull_request, deployment_status, check_run and check_suite.
| Permission | Access | Why |
|---|---|---|
| Metadata | Read | Required by GitHub for every app. |
| Pull requests | Write | Reads pull request files and labels, writes the comment, and creates the label a repository filters on when you ask. |
| Checks | Write | Creates and updates the Outqube check. |
| Deployments | Read | Finds the preview URL of a pull request's commit, in the deployment mode. |
| Organization members | Read | For an organisation account only. Lets Outqube verify that you are an admin of the organisation when you connect. |
It has no Contents permission and reads code only as a pull request's diff, and only when you turn that on. Adding a permission later makes every installer approve again.
One installation belongs to one Outqube organisation. Uninstalling the app on GitHub, or removing a repository from it, unlinks it in Outqube and deletes what Outqube kept about it. Every change is in the audit log.
Link a repository
Each repository runs one Outqube app, and an app can have several repositories. In Settings, GitHub, select Link next to a repository. Its own page opens, with a What will happen summary at the top that describes, in plain sentences, what the settings below it will do. It starts filled in: the organisation's only app (or the one named like the repository), the app's only suite (or the one named smoke, else the first), and On every pull request push on the app's first environment URL. Often Link repository is all it takes.
| Setting | What it does | Default |
|---|---|---|
| App | Whose tests run. | The only app, or the one named like the repository |
| Suite | Which tests a pull request runs. Without one, only runs your CI starts are reported. | The only suite, or smoke, or the first |
| When to test | What starts a run and which URL it tests. See the next section. | On every pull request push |
| Target branches (More options) | Pull requests into these branches run. Pick from the repository's branches or type a name; * matches several, such as release/*. Empty means all. | All |
| Skip draft pull requests (More options) | Drafts run once they are ready for review. | On |
| Only test pull requests with a label (More options) | Only pull requests with the label run. See Test only labelled pull requests. | Off |
| Comment on the pull request (More options) | One comment, updated on each push. | On |
| Cancel runs for older pushes (More options) | A new push stops runs still going for earlier commits. | On |
| Use the pull request's changes in failure reports (More options) | Reads the diff to point at likely causes. The code is not stored. | Off |
Selecting a linked repository, or Edit, opens the same page. There, Status pauses a link (nothing runs and no check is posted) or makes it active again, and Unlink stops the repository. Disconnect on Settings, GitHub unlinks the whole installation.
When to test
| Choice | What starts a run |
|---|---|
| On every pull request push | Opening, pushing to, reopening or marking a pull request ready runs the suite on the same URL every time. |
| When a preview deploy is ready | A successful deployment of the pull request's commit in an environment whose name matches the Preview environment name (Preview, or Preview* with a wildcard) runs the suite on the deployment's URL. |
| Only when my CI runs the Outqube CLI | Nothing starts from GitHub. Runs your CI starts with the CLI report here, when they carry the repository and the commit's full SHA. The page shows a GitHub Actions workflow for the chosen suite that reads the key from the OUTQUBE_API_KEY secret. |
Target branches, drafts and the label apply to the runs Outqube starts, and a commit runs once: a reopened pull request or a second matching deployment does not run it again. Pushes to a branch without a pull request run nothing. Outqube tests a URL only when its host is on the environment's base URL or allowed domains; for a deployment URL it also requires the environment's scheme and port.
Allowed domains
Only allow a domain that your project alone deploys to, such as *.preview.example.com. Never allow a shared one like *.vercel.app: anyone could deploy there, and Outqube would run your test logins against their site.
Test only labelled pull requests
Turn on Only test pull requests with a label under More options to run tests only where you ask for them. The label starts as outqube; pick another from the repository's labels or type a name. Labels match in any case.
- A pull request without the label runs nothing and gets no check or comment.
- Adding the label starts a run right away. In the preview deploy mode it runs on the preview if one is already ready, otherwise when it is.
- Pushes to a labelled pull request run as usual.
- Removing the label stops a run still going, and later pushes run nothing.
Target branches and drafts still apply. If the repository has no such label yet, Create label on GitHub adds it. With Only when my CI runs the Outqube CLI, your CI decides which pull requests run and the label does not apply.
The check
Every run whose repository and full commit SHA match a linked repository of its app gets an Outqube check on that commit, whoever started it: GitHub, the CLI or the API. It goes from queued to in progress, then:
| Run | Check |
|---|---|
| Passed | Success |
| A test failed | Failure |
| None failed, but one could not finish | Neutral |
| Cancelled | Cancelled |
Details opens the run in Outqube. The check summary has the counts and the failure reports of up to five failed tests. When nothing could start, a neutral check says why: no suite, no tests, the budget is reached, or the URL is not on the environment's domains.
The comment
With the comment on, Outqube keeps one comment per pull request and rewrites it with the check's summary when the newest run is queued and when it ends. It never adds one per push.
Re-run and newer pushes
Re-run on the check runs the same tests on the same commit, environment and URL as a new run linked to the old one. With Cancel runs for older pushes on, a push stops the runs Outqube started for the pull request's earlier commits. Runs your CI started with the CLI are left to your CI.
Fork pull requests
Pull requests from forks never run on their own: the fork controls the preview's code, and a run would use your test logins against it. Their commit gets one neutral Skipped: fork PR check, for deployments and re-runs too. To test one, push the branch to the repository itself.
Use the pull request's changes (optional)
With this on, a failed test's report names up to three changed files that likely relate, each with a one-line reason. Outqube matches the changed files' paths and lines (at most 200 files, about 1 MB) against the failure's URLs, console errors, failing step and test. No AI model sees the code. The diff stays in memory, only paths and reasons are kept, and these runs never go into training data or any export.
Security
- Webhooks from GitHub are verified with its signature, and each delivery is handled once.
- Installation tokens live in memory for under an hour and are never stored or logged.
- An installation is linked only after GitHub confirms the installing user can reach it and Outqube confirms they are an admin of the organisation they started from.
Next steps
- Test every pull request with the GitHub App: install, link and require the check.
- CLI reference: start runs from your own pipeline and still get the pull request check.
- Event webhooks: get a signed request when a pull request run finishes.