Our crawler

OrleaBot

OrleaBot visits public web pages on behalf of Orlea customers to notice when those pages change. It identifies itself with the user agent OrleaBot/1.0 (+https://orlea.com/bot; competitive monitoring).

How it behaves

  • It only fetches publicly accessible pages. It never logs in, submits forms or bypasses paywalls.
  • It honours robots.txt, including rules addressed to OrleaBot specifically.
  • Each page is fetched on a schedule at most hourly, and usually far less often, however many customers track it; a change is confirmed with one extra read.
  • It keeps the readable text of a page to detect changes. When a page is first tracked, and when a change on it is confirmed, a screenshot of it is taken through a browser service as evidence of what it showed; that visit loads the page as a browser would, images and scripts included.
  • When a site turns automated visitors away, a public page may be read through a fetch service instead. The same robots.txt rules, frequency limits and public-pages-only rule apply.

Opting out

Add this to your robots.txt and OrleaBot stops fetching your pages within six hours:

User-agent: OrleaBot
Disallow: /

For anything else, write to [email protected].