Our crawler

Observes fetches public web pages to verify companies' own stated web presence. This page is the standing description of how that crawler behaves, and it changes in the same commit as the crawler itself — the string below is imported from the crawler's configuration, so this page and the wire cannot disagree.

User-Agent

observes-stream-tap/0.0.0 (+https://observes.co.uk)

What we do and don't do

• Public, unauthenticated pages only. Never behind a login, paywall or session.
• robots.txt is honoured. So are 403 and 429: a block is an answer, and we record it as one — we never rotate addresses, disguise the agent, or solve CAPTCHAs to get past it.
• Conservative rate limits per host; HEAD preferred to GET.
• Company-level technical and factual data only. If a page unexpectedly returns personal data, we stop and flag rather than store.

Ask us to stop

Block the agent in robots.txt (we honour it), return 403 (we treat it as final), or write to crawler@observes.co.uk and we will exclude your host.