Drift reports
Turn the real-site smoke summary into a drift report, a tracking issue, observability records and a snapshot to learn from.
The smoke run (see Smoke tests) says whether the extension still works on a few real public application pages. The drift step reads its output and the committed baselines, decides how bad each change is, and hands the result to four places: a file, one GitHub issue, the self-hosted observability stack and a sanitized snapshot of every changed page. It lives in apps/drift and the optional exporters in packages/telemetry.
It does not depend on the smoke code. It reads three json formats, described below, so it can be merged and used on its own.
Run it
pnpm --filter drift drift report \
--summary apps/e2e/smoke/results/summary.json \
--baselines apps/e2e/smoke/baselines \
--targets apps/e2e/smoke/targets.json \
--out /tmp/drift| Flag | Meaning |
|---|---|
--summary <file> | the smoke summary.json (required) |
--baselines <dir> | the folder of <id>.json baselines (required, a missing file is fine) |
--out <dir> | where the report and snapshots go (required) |
--targets <file> | the smoke targets.json, needed for the page urls the snapshots use |
--previous <file> | the drift-report.json of an earlier run, to tell new drift from persisting drift |
--github | update the tracking issue, needs gh and a token with issues: write |
--repo owner/name | repository for --github, default is the one gh finds |
--export | send the report to the exporters whose env is set |
--snapshots | take a sanitized snapshot of each changed page |
--snapshot-cli <file> | the snapshot tool, default apps/e2e/tools/snapshot/cli.ts |
--delay <ms> | pause between snapshots, default 10000 |
Exit code 0 is clean, 1 means at least one target drifted, 2 is a usage or input error.
What it reads
summary.json is an array with one entry per target. Only these keys are read, anything else is ignored:
{
"id": "lever-acme",
"ats": "lever",
"status": "ok | drift | gone | blocked | error | skipped",
"reasons": ["filled fell from 12 to 6"],
"notes": ["field signature changed"],
"observation": {
"toast": true, "collected": 10, "filled": 8, "outlined": 0, "left": 1,
"sensitiveAsked": 0, "sensitiveFilled": 0, "consoleErrors": 0, "clickMissed": false
},
"ms": 12000
}observation is null for a page that was gone or blocked. A baseline is baselines/<id>.json: { "recorded": "2026-11-01", "collected": 10, "filled": 12, "outlined": 0, "signature": "<16 hex>" }. targets.json entries need id and url, and may have ats, click and enabled.
The report
drift-report.json has schemaVersion: 1, generatedAt, drifted, totals (counts by status and by severity), targets and resolved. Each target has its status, severity, reasons and notes, the numbers it filled and collected, deltas against the baseline, signatureChanged, and newSince or persisting against the previous report.
| Severity | When |
|---|---|
critical | a sensitive field (gender, race, consent, signature and the like) was filled or asked about. This counts as drift even when the smoke status is ok: the run exits non zero, the issue title says so and the issue is never closed while it is there |
high | detection stopped: no fill offer, or the apply button selector matched nothing |
medium | the run drifted for another reason, usually filled or collected fell |
low | only the field signature changed. Not drift, but it gets a snapshot |
none | nothing to see. gone, blocked, skipped and error are always none: they say nothing about the extension |
drift-report.md is the issue body: one line, one table, plain words. Every string that came from a page is escaped before it is written (no mention, link, image, heading, html or table break survives), each cell is cut to 200 characters and the body stays under 60000 characters with an "and N more rows" line. Target ids and board names must be plain lowercase words (^[a-z0-9-]+$) or the summary is refused, and the summary schema caps the length of every string and list.
Where it goes
- The tracking issue. One open issue with the label
drift, the exact titledrift: real ats pages changed, the marker the report ends with and an author ingithub-actions,app/github-actions,github-actions[bot]orDRIFT_ISSUE_AUTHORS(comma separated logins, for a run with a personal token). Drift edits that issue in place, or opens a new one (and creates the label if it is missing) when none matches. An issue that only carries the label is never touched. A clean run comments and closes it. Nothing else is posted to GitHub - OpenObserve. One record per target and one per run in the stream
skiffr_drift, plus gauge metrics (skiffr.drift.targets,skiffr.drift.drifted,skiffr.drift.severity) over OTLP/HTTP - Umami gets a
drift-detectedordrift-cleanevent. GlitchTip gets one error message when something iscritical
All exporters are no-ops while their variables are unset, never throw and never log a secret. A failing service is reported in the log and does not change the exit code.
| Variable | What it is |
|---|---|
OPENOBSERVE_URL | base url of OpenObserve |
OPENOBSERVE_AUTH | the full Authorization header value, for example Basic ... |
OPENOBSERVE_ORG | organisation, default default |
OPENOBSERVE_STREAM | log stream, default skiffr_drift |
UMAMI_URL, UMAMI_WEBSITE_ID | Umami base url and the id of the tracked site. UMAMI_HOSTNAME overrides the host name sent |
GLITCHTIP_DSN | Sentry compatible DSN of the GlitchTip project |
No value lives in the repo: set them as repository variables or secrets, or in Infisical.
Learn from it
With --snapshots, every target that drifted (and every target whose signature changed) is opened by the snapshot tool, sanitized and written to <out>/snapshots/<id>/, then the whole folder is checked with pnpm snapshot --check. The tool's own guard and politeness rules apply: public hosts only, one page at a time, a pause between pages, only GET. A page that cannot be snapshotted is a line in the log, not a failure.
The folder is uploaded as the drift-<run id> artifact. It is never committed. A person or an agent looks at it, and if the site really changed, turns it into a new fixture under apps/e2e/pages/adapters or apps/e2e/pages/snapshots, and updates the target baseline on purpose.
The workflow
.github/workflows/drift.yml runs when the smoke workflow completes (or by hand with a smoke run id). It does nothing until the repository variable DRIFT_ENABLED is true. It downloads the smoke artifact, reads the baselines from the checkout, runs the steps above, and uploads the report and snapshots. The run is red when something drifted.
What is reachable from where
Read from the keep (the-keep), no secrets:
| Service | Version | Address | Reach |
|---|---|---|---|
| OpenObserve | v1.1.2 | o2.findmalek.engineer | tailnet only |
| Infisical | v0.167.3 | secrets.findmalek.engineer | tailnet only |
| Umami | 3.4.1 | analytics.findmalek.engineer | public |
| GlitchTip | 6.2.6 | errors.findmalek.engineer | public |
The tailnet only routes sit behind a traefik ipAllowList of 100.64.0.0/10 (and the tailnet ipv6 range). A GitHub hosted runner is not on the tailnet, so it cannot send to OpenObserve or read Infisical. Set OPENOBSERVE_URL only on a runner that joined the tailnet, or put a public ingest proxy in front of it. Umami and GlitchTip work from any runner.
Who may start it
drift.yml reads the artifact of a smoke run with issues: write, so it only trusts a run that was started by the schedule or by hand, from this repository, on the default branch and from .github/workflows/smoke.yml. A pull request can add a workflow named smoke, and its artifact is never read. A manual dispatch names a run by id and gets the same checks through the api before anything is downloaded.
Order of merging
Drift is inert until the smoke job is merged: drift.yml reacts to the workflow smoke and reads apps/e2e/smoke/baselines and targets.json, which only exist on that branch. Merge smoke first. The repository variable DRIFT_ENABLED stays off until then.
The exporters are temporary
packages/telemetry is a stop gap with the same variable names as @skiffr/observability (GLITCHTIP_DSN, UMAMI_URL, UMAMI_WEBSITE_ID). It does not follow redirects, never throws (an object that cannot be serialized resolves as an error result), sends the error class and scalar extras only (never a message or a stack), and drops Umami events past 20 a minute. The swap to @skiffr/observability is tracked in #202: this package is deleted then and the playground and drift use it.