Observability tour

Observability says what is collected and what never leaves the process. This page is the other half: where to click, what to type, and how to follow one request through all three tools. The menu paths are the same on any instance. The self hosted stack on the keep uses errors.findmalek.engineer for GlitchTip and analytics.findmalek.engineer for Umami, replace the host if yours differs. OpenObserve there is reachable on the tailnet only.

Where to lookCopy Anchor Link

Nothing sends a push notification yet. The alerts are created switched off, see alerts. Until you wire one, look at skiffr overview once a day.

OpenObserveCopy Anchor Link

DashboardsCopy Anchor Link

They live in the folder skiffr (menu Dashboards, folder skiffr). The default range is the last hour. Auto refresh is not stored in the dashboard, it is a url parameter, so open them with &refresh=30s and bookmark that:

• skiffr overview: requests per minute by status class, 5xx share, db up, p50 and p95 per route, scheduler freshness and failures, status mix, model calls and tokens, rate limited requests, top routes.
• skiffr errors and slow requests: 4xx and 5xx over time, auth failures and 429, tables of the latest 5xx, latest 4xx and requests over a second, slowest routes, failing routes. The tables have plain columns: time, severity, status, method, route, ms, request id, message.
• skiffr jobs and scheduler: runs by job and outcome, seconds since the last success, average run time, items handled, failed runs, sync failures by kind, rate limited requests.

https://<openobserve host>/web/dashboards/view?org_identifier=default&folder=<folder id>&refresh=30s&dashboard=<dashboard id>

The folder and dashboard ids are in the address bar when you open a dashboard from the list. They are different on every instance.

Notes on reading them:

• A passing /health is counted in the metrics but not written as a log line, because the container healthcheck hits it every few seconds. A failing one (status 400 or more) is logged, so a down probe shows in the error tables. The log panels also hide passing health lines from older data, and the metric panels for traffic and latency leave /health out.
• Panels filter on fields (severity, http_route, http_status_code, duration_ms), never on the text of body.
• Metric panels use PromQL (rate, increase, histogram_quantile) because the counters are cumulative. Log panels use SQL.
• The latency percentiles come from histogram buckets, so read them as a band. p95 of a route with three requests means little.
• Panels about model calls, sync failures and rate limits are empty until the first event of that kind. An empty panel there is good news. A flat empty requests per minute is not.
• Only the logs go through the 500 line queue, so above about 30 requests per second the log tables lose lines while the metric panels stay exact.

Saved viewsCopy Anchor Link

In Logs, open the stream skiffr, then the Saved views menu (the bookmark icon next to the query box). Pick one, set the time range, press the run button.

Every view shows the columns severity, http_status_code, http_method, http_route, duration_ms, request_id, body, and starts at the last hour. Change the columns with the field list on the left, the view itself stays as saved until you choose update view.

The stream also has these settings, so filters on them are fast: full text search on body, an index on severity, http_route, http_method, http_status_code, http_status_class, request_id and service_name, and a bloom filter on request_id. Retention is 30 days, set on the server.

Field glossaryCopy Anchor Link

Log stream skiffr:

Metric streams (each has service_name, deployment_environment_name, start_time which changes when the api restarts, so a restart shows up as a new series):

SQL cheat sheetCopy Anchor Link

Open Logs, pick the stream skiffr, switch on SQL mode, paste, set the range. Metric queries go in the Metrics page (SQL or PromQL). Counters are cumulative, so a plain max(value) is the total since the last restart. For rates use PromQL.

Errors, last hour:

select _timestamp, severity, http_status_code, http_route, duration_ms, request_id, body
from "skiffr"
where severity = 'ERROR' or http_status_code >= 500
order by _timestamp desc limit 100

Which routes fail, and how:

select http_route, http_status_code, count(*) as n
from "skiffr" where http_status_code >= 400
group by http_route, http_status_code order by n desc limit 20

Slow requests:

select _timestamp, http_method, http_route, duration_ms, request_id
from "skiffr" where duration_ms > 1000
order by duration_ms desc limit 50

Requests by route with latency:

select http_route, count(*) as requests, avg(duration_ms) as avg_ms,
  approx_percentile_cont(duration_ms, 0.95) as p95_ms, max(duration_ms) as max_ms
from "skiffr" group by http_route order by requests desc limit 20

Status class breakdown:

select http_status_class, count(*) as n from "skiffr"
group by http_status_class order by n desc

One request by id:

select _timestamp, severity, http_status_code, http_route, duration_ms, body
from "skiffr" where request_id = 'PASTE-REQUEST-ID-HERE' order by _timestamp

Who gets rate limited:

select http_route, http_method, count(*) as n
from "skiffr" where http_status_code = 429
group by http_route, http_method order by n desc

Who is refused (not signed in, not allowed):

select http_route, http_status_code, count(*) as n
from "skiffr" where http_status_code in (401, 403)
group by http_route, http_status_code order by n desc

Requests per minute:

select histogram(_timestamp, '1 minute') as minute, count(*) as requests
from "skiffr" group by minute order by minute

5xx share per 5 minutes:

select histogram(_timestamp, '5 minutes') as bucket, count(*) as total,
  sum(case when http_status_code >= 500 then 1 else 0 end) as errors
from "skiffr" group by bucket order by bucket

Db up gaps (a missing minute means nobody polled /health, a 0 means postgres was unreachable). Run it as a metrics query:

select histogram(_timestamp, '1 minute') as minute, min(value) as db_up
from "skiffr_db_up" group by minute order by minute

Scheduler runs that did not end ok since the last restart:

select job, outcome, max(value) as runs_since_start
from "skiffr_scheduler_runs" where outcome <> 'ok'
group by job, outcome order by job

Scheduler freshness (PromQL, seconds since the last good pass per job):

min by (job) (time() - skiffr_scheduler_last_success_seconds)

Sync failures and model budget (PromQL):

sum by (job, error_kind) (increase(skiffr_sync_failures[1h]))
sum(increase(skiffr_ai_calls{outcome="budget"}[1h]))

p95 per route (PromQL):

histogram_quantile(0.95, sum by (le, http_route) (rate(skiffr_http_duration_ms_bucket{http_route!="/health"}[5m])))

Time range and refreshCopy Anchor Link

• Start with the last 15 minutes in Logs, widen only when you need to. Queries over days of logs get slow.
• Dashboards default to 1 hour. Add &refresh=30s to the address to auto refresh, and bookmark that address. The minimum is 5 seconds.
• Times in the tables made with to_char are UTC. The log table and charts use your browser's timezone.
• Metrics and logs are flushed every 15 seconds, so the newest point is up to 15 seconds behind.

Import the JSONCopy Anchor Link

Everything above is in ops/openobserve/ as json, so another instance (or this one after a mishap) can get it back. You need curl and jq.

export OPENOBSERVE_URL=https://o2.example.net
export OPENOBSERVE_AUTH='Basic <base64 of email:password>'
sh ops/openobserve/import.sh

The script is safe to run again. It finds every object by name and updates it in place, never deletes, never touches anything not named skiffr, and passes the auth to curl on stdin so it is not printed or visible in the process list. Details are in ops/openobserve/README.md. To change a dashboard, edit it in the UI, export the json from the dashboard settings and replace the file.

AlertsCopy Anchor Link

The import creates three alerts in the alert folder skiffr, all switched off:

Alert names cannot contain spaces in this OpenObserve version, so they use underscores.

Two blind spots: if the api is down or crashed there is no sample and no log line, so nothing fires (the series just ends). And a job that is switched off (GMAIL_SCHEDULER=off) has no series, so no alert. Watch skiffr overview or add an uptime check from outside for the first case.

OpenObserve cannot create an alert without a destination, and the open source build here has no SMTP, so email destinations are refused. The alerts point at a placeholder webhook, skiffr_unset, aimed at https://example.com/skiffr-alert-placeholder, which receives nothing you own. To go live:

1. Alerts, Destinations, add a webhook (ntfy, Discord, Slack or any url you own) or, after setting ZO_SMTP_* on the OpenObserve container, an email destination.
2. Open each alert, replace skiffr_unset with the new destination, switch it on.
3. Running the import again is safe: it only creates the placeholder when it is missing and keeps the destinations and the on/off switch of an alert that already exists.

GlitchTipCopy Anchor Link

Org findmalek, team skiffr, projects skiffr-api, skiffr-web, skiffr-dashboard. Docs has no project because it reports nothing. Every project belongs to the team skiffr, so Issues and Projects list them for any member.

TriageCopy Anchor Link

1. Issues, pick the project (or all three), filter is:unresolved. Newest and most frequent first.
2. Open an issue: the stack (function, file, line), the tags (route, request_id, job, account_id, source_id, error_kind, model, and sometimes cause, constraint, table) and the environment. The exception type is the issue title. There is no message, request body, header or breadcrumb, on purpose.
3. Decide: fix it and mark resolved, or ignore it (for a known upstream blip), or leave it open and comment.
4. Regressions: if a resolved issue gets a new event, it reopens by itself and counts as a regression. Mark it resolved in the release that fixed it to keep that signal sharp.
5. Only errors in our own code reach GlitchTip: failed procedures, model errors and internal sync failures. Google 5xx, expired grants, rate limits and dead careers pages are metrics only.

Finding the event for a request idCopy Anchor Link

An unexpected error thrown inside a procedure carries skiffr's request id as a tag (request_id). In Issues type request_id:<id> in the search box. The same id is in the log line (request_id field) and in the x-request-id response header, which makes it the bridge between the two tools.

Model errors are different. The api reports them with the provider's request id in the request_id tag, then answers the caller with a 503, so skiffr's own id never reaches GlitchTip. For those, search by route and model instead, for example route:ai.draftAnswer model:<model id> (the route is the procedure path, shown on the log line's error row). Scheduler errors have no request at all, they carry job and account_id or source_id.

What is scrubbedCopy Anchor Link

• The api only sends allowlisted fields (see observability), so most of this is a second line of defence.
• On the server each project has data scrubbing on: values under keys such as password, token, secret, authorization, cookie, session, credit card numbers, private key blocks and email addresses in free text become [Filtered]. Tags like request_id and route are not touched.
• Ip addresses are anonymised (scrub_ip_addresses), per project and for the org.
• Retention is the server default of 90 days, set with GLITCHTIP_RETENTION_DAYS on the container, not per project.

A test event with an email in the message and a tag called authorization came back with both filtered and request_id, route and job intact. It was deleted afterwards.

AlertsCopy Anchor Link

Each project has one rule, skiffr new issue or regression (1 event within 1 minute). GlitchTip has no on/off switch for a rule, a rule with no recipients does nothing, and this install has EMAIL_URL=consolemail:// (email is only printed in the worker log), so the rules have no recipients yet. To turn them on: set a real EMAIL_URL (smtp) on the GlitchTip containers, then in each project, Settings, Alerts, open the rule and add the email recipient or a webhook. New issues and regressions are both covered by that rule.

Settings that are fixed by the serverCopy Anchor Link

• Environment filters: the apps send DEPLOY_ENV as the environment, so after the first event production shows up in the environment dropdown. Pick it there, GlitchTip remembers the choice per browser. GlitchTip creates the environment on its own, so there is nothing to preset.
• Event limits: event_throttle_rate is 0 (nothing dropped). There is no per project size or count cap in this version.

UmamiCopy Anchor Link

Nothing to configure here. The numbers live at analytics.findmalek.engineer (sign in with the Umami admin), one site per app: skiffr web, skiffr docs, skiffr dashboard. The api has no site, so the ai_budget_exhausted event is not recorded until UMAMI_API_WEBSITE_ID is set. Page views are counted without cookies, without query strings and not at all with Do Not Track on. Umami says nothing about errors or latency, use the two tools above for those.

A request went wrong: follow the trailCopy Anchor Link

Say a user reports "saving an answer failed a minute ago".

1. OpenObserve, Logs, saved view skiffr errors, range last 15 minutes. If you know the route, add and http_route = '/rpc/answers/update'. Pick the line with a 5xx. If nothing is there, try skiffr warnings and errors: a 4xx is the caller's input, a 429 is the limiter.
2. Read the line: body has the method, route, status and time, duration_ms tells slow from broken. Copy request_id from the row.
3. Open skiffr by request id, paste the id between the quotes, run. You get every line carrying that id.
4. GlitchTip, project skiffr-api, Issues, search request_id:<id>. If an issue comes up, the stack and tags say where it threw. If nothing comes up, check the route: a 503 on a model route is a model provider error, and GlitchTip has it under route and model, not under this id (the log line error provider: AiModelError next to it holds the same story). If it is neither, the failure was not an error in our code (bad input, a refusal, an upstream answer), and the log line is the whole story.
5. Check the neighbourhood on skiffr overview: is it one request, or did the error rate and p95 move at the same time, or db up drop? If the db or a scheduler is involved, skiffr jobs and scheduler shows it.
6. Fix, ship, and mark the GlitchTip issue resolved. A regression will reopen it.

What is never loggedCopy Anchor Link

These have no key to travel under, see the allowlist in observability:

• request or response bodies, headers, cookies, query strings and the typed url (only the router's route template)
• field labels and values, answers, resume and job text, emails, names, tokens, passwords
• user ids (only mailbox and source row ids appear, as account_id and source_id)
• model prompts and answers, unless AI_TRACE_CAPTURE_CONTENT=true
• error messages other than the fixed sentences written in this repo, causes and stack variables

The dashboards and views cannot show what is not stored. If a panel seems to be missing a detail, that is by design. Do not add a field to the allowlist to fill a dashboard without a privacy review.