---
title: "Drift reports"
description: "Turn the real-site smoke summary into a drift report, a tracking issue, observability records and a snapshot to learn from."
url: https://docs.skiffr.findmalek.engineer/docs/drift
markdown: https://docs.skiffr.findmalek.engineer/docs/drift.md
---

# Drift reports

The smoke run (see Smoke tests) says whether the extension still works on a few real public application pages. The drift step reads its output and the committed baselines, decides how bad each change is, and hands the result to four places: a file, one GitHub issue, the self-hosted observability stack and a sanitized snapshot of every changed page. It lives in `apps/drift` and the optional exporters in `packages/telemetry`.

It does not depend on the smoke code. It reads three json formats, described below, so it can be merged and used on its own.

## Run it

```bash
pnpm --filter drift drift report \
  --summary apps/e2e/smoke/results/summary.json \
  --baselines apps/e2e/smoke/baselines \
  --targets apps/e2e/smoke/targets.json \
  --out /tmp/drift
```

| Flag | Meaning |
| --- | --- |
| `--summary <file>` | the smoke `summary.json` (required) |
| `--baselines <dir>` | the folder of `<id>.json` baselines (required, a missing file is fine) |
| `--out <dir>` | where the report and snapshots go (required) |
| `--targets <file>` | the smoke `targets.json`, needed for the page urls the snapshots use |
| `--previous <file>` | the `drift-report.json` of an earlier run, to tell new drift from persisting drift |
| `--github` | update the tracking issue, needs `gh` and a token with `issues: write` |
| `--repo owner/name` | repository for `--github`, default is the one `gh` finds |
| `--export` | send the report to the exporters whose env is set |
| `--snapshots` | take a sanitized snapshot of each changed page |
| `--snapshot-cli <file>` | the snapshot tool, default `apps/e2e/tools/snapshot/cli.ts` |
| `--delay <ms>` | pause between snapshots, default 10000 |

Exit code 0 is clean, 1 means at least one target drifted, 2 is a usage or input error.

## What it reads

`summary.json` is an array with one entry per target. Only these keys are read, anything else is ignored:

```json
{
  "id": "lever-acme",
  "ats": "lever",
  "status": "ok | drift | gone | blocked | error | skipped",
  "reasons": ["filled fell from 12 to 6"],
  "notes": ["field signature changed"],
  "observation": {
    "toast": true, "collected": 10, "filled": 8, "outlined": 0, "left": 1,
    "sensitiveAsked": 0, "sensitiveFilled": 0, "consoleErrors": 0, "clickMissed": false
  },
  "ms": 12000
}
```

`observation` is `null` for a page that was gone or blocked. A baseline is `baselines/<id>.json`: `{ "recorded": "2026-11-01", "collected": 10, "filled": 12, "outlined": 0, "signature": "<16 hex>" }`. `targets.json` entries need `id` and `url`, and may have `ats`, `click` and `enabled`.

## The report

`drift-report.json` has `schemaVersion: 1`, `generatedAt`, `drifted`, `totals` (counts by status and by severity), `targets` and `resolved`. Each target has its status, severity, reasons and notes, the numbers it filled and collected, `deltas` against the baseline, `signatureChanged`, and `newSince` or `persisting` against the previous report.

| Severity | When |
| --- | --- |
| `critical` | a sensitive field (gender, race, consent, signature and the like) was filled or asked about. This counts as drift even when the smoke status is `ok`: the run exits non zero, the issue title says so and the issue is never closed while it is there |
| `high` | detection stopped: no fill offer, or the apply button selector matched nothing |
| `medium` | the run drifted for another reason, usually `filled` or `collected` fell |
| `low` | only the field signature changed. Not drift, but it gets a snapshot |
| `none` | nothing to see. `gone`, `blocked`, `skipped` and `error` are always `none`: they say nothing about the extension |

`drift-report.md` is the issue body: one line, one table, plain words. Every string that came from a page is escaped before it is written (no mention, link, image, heading, html or table break survives), each cell is cut to 200 characters and the body stays under 60000 characters with an "and N more rows" line. Target ids and board names must be plain lowercase words (`^[a-z0-9-]+$`) or the summary is refused, and the summary schema caps the length of every string and list.

## Where it goes

1. The tracking issue. One open issue with the label `drift`, the exact title `drift: real ats pages changed`, the marker the report ends with and an author in `github-actions`, `app/github-actions`, `github-actions[bot]` or `DRIFT_ISSUE_AUTHORS` (comma separated logins, for a run with a personal token). Drift edits that issue in place, or opens a new one (and creates the label if it is missing) when none matches. An issue that only carries the label is never touched. A clean run comments and closes it. Nothing else is posted to GitHub
2. OpenObserve. One record per target and one per run in the stream `skiffr_drift`, plus gauge metrics (`skiffr.drift.targets`, `skiffr.drift.drifted`, `skiffr.drift.severity`) over OTLP/HTTP
3. Umami gets a `drift-detected` or `drift-clean` event. GlitchTip gets one error message when something is `critical`

All exporters are no-ops while their variables are unset, never throw and never log a secret. A failing service is reported in the log and does not change the exit code.

| Variable | What it is |
| --- | --- |
| `OPENOBSERVE_URL` | base url of OpenObserve |
| `OPENOBSERVE_AUTH` | the full `Authorization` header value, for example `Basic ...` |
| `OPENOBSERVE_ORG` | organisation, default `default` |
| `OPENOBSERVE_STREAM` | log stream, default `skiffr_drift` |
| `UMAMI_URL`, `UMAMI_WEBSITE_ID` | Umami base url and the id of the tracked site. `UMAMI_HOSTNAME` overrides the host name sent |
| `GLITCHTIP_DSN` | Sentry compatible DSN of the GlitchTip project |

No value lives in the repo: set them as repository variables or secrets, or in Infisical.

## Learn from it

With `--snapshots`, every target that drifted (and every target whose signature changed) is opened by the snapshot tool, sanitized and written to `<out>/snapshots/<id>/`, then the whole folder is checked with `pnpm snapshot --check`. The tool's own guard and politeness rules apply: public hosts only, one page at a time, a pause between pages, only GET. A page that cannot be snapshotted is a line in the log, not a failure.

The folder is uploaded as the `drift-<run id>` artifact. It is never committed. A person or an agent looks at it, and if the site really changed, turns it into a new fixture under `apps/e2e/pages/adapters` or `apps/e2e/pages/snapshots`, and updates the target baseline on purpose.

## The workflow

`.github/workflows/drift.yml` runs when the `smoke` workflow completes (or by hand with a smoke run id). It does nothing until the repository variable `DRIFT_ENABLED` is `true`. It downloads the smoke artifact, reads the baselines from the checkout, runs the steps above, and uploads the report and snapshots. The run is red when something drifted.

## What is reachable from where

Read from the keep (`the-keep`), no secrets:

| Service | Version | Address | Reach |
| --- | --- | --- | --- |
| OpenObserve | v1.1.2 | `o2.findmalek.engineer` | tailnet only |
| Infisical | v0.167.3 | `secrets.findmalek.engineer` | tailnet only |
| Umami | 3.4.1 | `analytics.findmalek.engineer` | public |
| GlitchTip | 6.2.6 | `errors.findmalek.engineer` | public |

The tailnet only routes sit behind a traefik `ipAllowList` of `100.64.0.0/10` (and the tailnet ipv6 range). A GitHub hosted runner is not on the tailnet, so it cannot send to OpenObserve or read Infisical. Set `OPENOBSERVE_URL` only on a runner that joined the tailnet, or put a public ingest proxy in front of it. Umami and GlitchTip work from any runner.

## Who may start it

`drift.yml` reads the artifact of a `smoke` run with `issues: write`, so it only trusts a run that was started by the schedule or by hand, from this repository, on the default branch and from `.github/workflows/smoke.yml`. A pull request can add a workflow named `smoke`, and its artifact is never read. A manual dispatch names a run by id and gets the same checks through the api before anything is downloaded.

## Order of merging

Drift is inert until the smoke job is merged: `drift.yml` reacts to the workflow `smoke` and reads `apps/e2e/smoke/baselines` and `targets.json`, which only exist on that branch. Merge smoke first. The repository variable `DRIFT_ENABLED` stays off until then.

## The exporters are temporary

`packages/telemetry` is a stop gap with the same variable names as `@skiffr/observability` (`GLITCHTIP_DSN`, `UMAMI_URL`, `UMAMI_WEBSITE_ID`). It does not follow redirects, never throws (an object that cannot be serialized resolves as an error result), sends the error class and scalar extras only (never a message or a stack), and drops Umami events past 20 a minute. The swap to `@skiffr/observability` is tracked in #202: this package is deleted then and the playground and drift use it.