Job sources

skiffr can watch a careers page or a feed and put new postings in your queue. You add the address under Sources in the dashboard, give it a name, and the api fetches it about once an hour.

What you can watchCopy Anchor Link

A page that builds its list with JavaScript, or sits behind a login, gives nothing: skiffr reads the html the server sends and does not run scripts. Find the page's feed or its board address instead.

Postings are queued with the source watch:<name> (the queue shows "Watched · name") and deduped by the same id the dashboard, the extension and Gmail use, so a job you also added by hand stays one row. skiffr remembers what it queued from a source: a job you delete does not come back on the next fetch.

At most 100 new jobs are queued per fetch. A long board fills up over the next fetches. The first fetch of a source with many postings queues them all (up to that limit), so expect a burst.

Removing a source stops the fetching. Jobs it already queued stay.

LimitsCopy Anchor Link

EnvironmentCopy Anchor Link

With several api replicas, one of them (chosen with a Postgres advisory lock, like Gmail) runs the schedule, and the source being fetched is leased in the database so a click and the schedule never fetch it at once. DATABASE_URL must be a direct Postgres connection, not a transaction pooler.

Security modelCopy Anchor Link

The server fetches addresses that users type in, so this is the part to read.

• Only public web pages. The address must be http or https on port 80 or 443 and must not carry a username or password. Names such as localhost, *.local, *.internal and single-label hosts are refused. Address literals are read the way a browser reads them, so 2130706433, 0x7f000001, 0177.0.0.1 and 127.1 are all 127.0.0.1, and refused.
• The name is resolved by skiffr, and every answer must be public. A name that resolves to any private, loopback, link-local (169.254.169.254, the cloud metadata address), carrier-grade NAT (100.64.0.0/10), multicast, reserved or documentation range of IPv4, or outside global unicast in IPv6 (unique local fc00::/7, link-local, IPv4-mapped, NAT64, 6to4, Teredo), is refused as a whole, even if another answer is public.
• The connection goes to the address that was checked. There is no second lookup between the check and the connect, so a name that answers differently the next time (dns rebinding) cannot send the request somewhere else.
• Every redirect starts over. At most 3 hops, and each target goes through the same checks, resolution included.
• Bounded. 2 MB on the wire and after decompression (a gzip bomb stops at the same number), a deadline for the whole fetch and an idle timeout (a slow server is cut), and only html, xml, rss, atom and json content types are accepted. Only identity and gzip encodings are asked for.
• Nothing identifying is sent. One fixed user agent, skiffr-watcher/1.0, no cookies (none are sent, kept or read), no referrer, no authorization.
• Job links are stored, never opened. Links found on a page must be plain http(s) links without credentials or internal hosts, and are only shown to you to click.
• Page text is untrusted. Titles and company names lose all markup, entities, control and direction characters and are cut to 200 characters. They are shown as text, never as html.
• Parsing is bounded. Pages are read with a streaming parser (htmlparser2) that builds no tree, so deep nesting costs nothing, at most 500 postings are kept, and JSON-LD blocks are capped in size, count and depth. A DOCTYPE with entities is never expanded.
• Per-user and per-host limits and the source cap above keep one account from using the server to hammer a site.
• Session only. The extension token can't call these routes.
• Logs carry the source id, the step and the kind of failure, never an address, a header or any page content.

What this does not cover: skiffr cannot stop a public site from serving a misleading posting, and anything reachable from the api's own network position through a public address (for example a public host that proxies into your network) is as reachable as it is to anyone. Run the api in a network where it can reach what it must and nothing more.

Why a source shows a problemCopy Anchor Link