Our crawler

Observes fetches public web pages to verify companies' own stated web presence. This page is the standing description of how that crawler behaves, and it changes in the same commit as the crawler itself — the string below is imported from the crawler's configuration, so this page and the wire cannot disagree.

Why we fetch — the observatory

Observes observes the UK commercial internet as a census: for each registered domain it records technical and factual state — name-server, mail, certificate, availability and redirect state, and the statutory disclosures a site publishes about the company operating it — so that the history of that state exists from now, whoever the domain later turns out to belong to.

• A census of the UK commercial internet: technical and factual data only.
• Identity evidence is read from sites' own disclosure pages — the company number, legal name and registered address a site publishes about itself — and nothing else.
• Page bodies are not retained: we keep the facts extracted, a receipt of the fetch and a fingerprint of the page, never the page.
• Contact and opt-out are as stated below.

The wording above is the stated purpose recorded in the project's decision record for the observatory and changes only with it.

User-Agent

observes-stream-tap/0.0.0 (+https://observes.co.uk)

What we do and don't do

• Public, unauthenticated pages only. Never behind a login, paywall or session.
• robots.txt is honoured. So are 403 and 429: a block is an answer, and we record it as one — we never rotate addresses, disguise the agent, or solve CAPTCHAs to get past it.
• Conservative rate limits per host; HEAD preferred to GET.
• Company-level technical and factual data only. If a page unexpectedly returns personal data, we stop and flag rather than store.

Ask us to stop

Block the agent in robots.txt (we honour it), return 403 (we treat it as final), or write to crawler@observes.co.uk and we will exclude your host.