Smoke tests

Boards change their markup without telling anyone. The smoke run opens a short list of real public application pages with the built extension, runs detection and a fill with a mock api and a synthetic profile, and reports what changed since the last time. It never submits anything.

It lives in apps/e2e/smoke/ and runs weekly in .github/workflows/smoke.yml. Pull requests never run it: the required checks only run the local tests, which use fixture pages on loopback.

Run itCopy Anchor Link

pnpm --filter extension build
pnpm --filter e2e smoke
pnpm --filter e2e smoke --only lever-spotify
pnpm --filter e2e smoke --update-baseline

The extension must be built first (or point E2E_EXTENSION_DIR at an unzipped build). Exit codes:

What a target checksCopy Anchor Link

For each target the runner loads the page, optionally clicks the apply button, waits for the extension's fill offer, accepts it and records:

• whether the offer toast showed up (detection)
• fields collected (sent to the mock api to be mapped, plus controls the extension filled without asking), filled and outlined (read from the result toast), and left (fields still empty afterwards)
• sensitive fields: questions about gender, race, veterans, disability, consent, signatures and the like must not be asked about and must keep their value. The runner reads every control before and after the fill
• console errors and the time taken

A target is drift when the offer does not appear, filled is below expect.minFilled or fell more than the threshold against the baseline, a field listed in expect.detect is not handled, a sensitive field was filled or asked about, or the click selector matched nothing. A changed field signature (hash of the labels and types) is only a note.

Add a targetCopy Anchor Link

Add an object to smoke/targets.json:

{
  "id": "lever-acme",
  "ats": "lever",
  "url": "https://jobs.lever.co/acme/00000000-0000-0000-0000-000000000000/apply",
  "click": "text=Apply now",
  "checked": "2026-11-12",
  "expect": { "minFilled": 4, "detect": ["email"], "untouched": ["pronouns"], "toast": true }
}

• id: lowercase letters, digits and dashes. Names the baseline file
• ats: greenhouse, lever, ashby, workable or other
• url: a public http or https page that needs no login. Loopback, private and credentialed addresses are refused when the file is read
• click: only for pages that open the form behind a button. It is the one click the runner makes on the page. Selectors that mention submit, send, pay, confirm or finish, or use xpath, are refused
• checked: the day you confirmed the page exists
• enabled: set to false to park a target without deleting it. workable-huggingface is parked: on the live page the first name, last name, email and phone stay empty after the fill, which is a real finding for the extension. Enable it again once that is fixed
• expect.minFilled: the least number of fields the fill must report
• expect.detect: words, each of which must appear in the label of a field the fill handled
• expect.untouched: extra words that mark fields which must come out as they went in. The built in list already covers the usual diversity and consent questions

Before adding a site, read its robots.txt and stay out of anything it disallows for *: the runner sends only GET requests and fetches what a browser visiting that page would, so a page that needs a disallowed path (an /api/ call, say) cannot be a target. Then run pnpm --filter e2e smoke --only <id> --update-baseline, look at summary.md, and commit the target and baselines/<id>.json (a test fails when a target has no baseline or a baseline has no target). A baseline holds counts and one hash of the field labels and types. It never holds page text, the page itself or any personal data. To drop a target, delete it from the list and delete its baseline.

Politeness and safetyCopy Anchor Link

• Every request goes through the snapshot tool's guard and fetcher: public hosts only, every redirect hop checked, the address vetted when the connection is made, size caps and a deadline
• Chromium cannot connect to anything on its own: it is started with a proxy that points at a closed local port, and only the mock api is let around it. A page that asks the browser to preconnect, prefetch or speculate gets nothing, and every real request is made by the guard
• Only GET and HEAD leave the machine. Every other request (fetch, beacon, xhr, ping, a form post) is aborted, with no exception. Nothing is submitted, uploaded or typed outside the fill engine. Popups are closed, and a hostile page that makes the route handler throw ends as an error for that target, never as a crashed run
• The api is the mock one on loopback and the profile is synthetic. No real data and no secrets are involved. The toast is only clicked where the extension drew it (its own closed shadow root), never on a button a page draws with the same attribute
• User agent skiffr-smoke/<version> with the repository link, no personal details
• One request per host at a time, 10 seconds between targets, at most 300 requests per target (the rest are aborted and the target is an error), images, fonts, media and known trackers and analytics hosts are not loaded
• A 429, or a 403 on the page or its api calls, from the target's own host counts as one strike for that host per target (third party hosts do not count). A second strike stops the whole run. The target is reported blocked, the rest skipped, and nothing is retried
• Targets are checked against robots.txt by hand when they are added. Checked 2026-11-13: job-boards.greenhouse.io allows everything, jobs.lever.co allows / for *, apply.workable.com has an empty Disallow. jobs.ashbyhq.com disallows /api/ for * and its postings only render through POST calls to it, so no Ashby page is a target
• Only the summary (ids, statuses and counts) is uploaded as a workflow artifact, for 3 days. Traces are opt in (--traces, or the traces input of a manual run) because they hold the third party page

Terms of serviceCopy Anchor Link

Only add public application pages of boards that you may load like any visitor, and keep the run at a low frequency (weekly). Read a site's terms before adding it and drop the target if it asks automated visitors to stay away. The run is a read only check of a public page, not a way to collect data: it keeps counts and a hash, not content.

The workflowCopy Anchor Link

smoke.yml runs on a weekly schedule and by hand. It does nothing until the repository variable SMOKE_ENABLED is true. It needs no secrets, builds the extension the same way the e2e job does, uploads summary.md and summary.json as an artifact, and keeps a single issue with the label smoke up to date: it opens it on drift (or on runner errors, with a title that says so), edits it in place on the next failing run and adds a comment when the set of failing targets changes (an edit notifies nobody), and comments and closes it when a run is clean. A run that verified nothing (exit code 3) leaves the issue alone and fails the job.