OutreachEmailFinderBot

What the Outreach Email Finder crawler does when it reaches your site, and how to keep it out.

User agent

Every request identifies itself with this string, so it can be found in your logs and matched by a rule:

OutreachEmailFinderBot/1.0 (+https://outreachemailfinder.com/bot)

What it fetches

The crawler is driven by a person who names one or more domains. It fetches pages of those sites: the page it starts from and the pages a link builder would open by hand, which is the home, contact, about, team, author, advertising, press and legal pages a site links to. It reads the addresses those pages publish and nothing else. It does not search the web for domains, and it does not guess addresses that are not on a page.

One URL submitted on its own is still a crawl of that site, not a single fetch. A single-URL check fetches only the page you give it when that page links to no other contact page of the site, and otherwise follows the candidate pages on the same host, up to the page cap below. It never leaves the host named in the request.

How often it requests

• One request at a time per host, with at least 1 second between two requests to the same host.
• If your robots.txt sets a Crawl-delay, the crawler waits that long up to 10 seconds. A longer value is capped there, so one site cannot hold a crawl open forever.

How much it reads

• At most 25 pages per domain: the start page, then the pages most likely to publish a contact address.
• It reads no page larger than 2 MB, and stops waiting on a page that has not answered within 10 seconds.
• It follows at most 5 redirects, and every hop goes through the same safety check that keeps requests off private and loopback addresses.

robots.txt

The crawler obeys robots.txt for every host it visits. It reads the file before the first page, and applies it to every URL it considers on that host, so a disallowed page is not fetched. When the file keeps the crawler off the page it would have started at, the result says that robots.txt kept it off, rather than showing an empty result with no reason.

How to block it

Add these two lines to your robots.txt:

User-agent: OutreachEmailFinderBot
Disallow: /

With those lines in place the crawler reads your robots.txt to find the rule, and fetches nothing else on your site. You can also refuse the user agent string at your server or firewall, and the opt-out list keeps a single address out of the results even when the page it sits on is public.

Bot walls and protected addresses

The crawler never works around protection. If a site answers with a bot challenge or a block page, the crawl stops and the result says the site blocked it. There is no headless trick, no different user agent and no retry with a changed address to get past that answer.

The same applies to an address a site has chosen to hide. When a page carries an address behind email protection, or writes it in a scrambled form that only a script can rebuild, the crawler leaves it alone: we do not decode it, and it does not appear in a result. Hiding an address is read as a decision and respected.

Contact

Questions about a crawl, or to tell us the crawler did something on your site that this page does not describe: [email protected].