Extension:CrawlerProtection
 Release status: stableCategory:Stable extensions |
|
|---|---|
| Implementation | HookCategory:Hook extensions |
| Description | Anti-crawler suite for MediaWiki |
| Author(s) | Jeffrey Wang (MyWikis-JeffreyWangtalk) |
| MediaWiki | 1.35-1.39Category:Extensions with manual MediaWiki version |
| License | MIT License |
| Download | Download master snapshot Note: |
The CrawlerProtection extension blocks anonymous users from performing actions and accessing special pages which are most frequently abused by AI crawler bots. The aim of this extension is to stop AI crawler bots from accessing pages which are expensive to render, thereby stymying excessive resource drain.
CrawlerProtection was introduced at MediaWiki Users and Developers Workshop Spring 2025. The video recording of the introduction can be found on YouTube.
Installation
- Download, extract and place the file(s) in a directory called
CrawlerProtectionin yourextensions/folder. - Add the following code at the bottom of your LocalSettings.php file:
wfLoadExtension( 'CrawlerProtection' );
Done – Navigate to Special:Version on your wiki to verify that the extension is successfully installed.
Configuration parameters
This extension doesn’t require any configuration. At the moment, the extension is plug-and-play, but also all-or-nothing.
Handled features
The following wiki features are disabled for anonymous users by CrawlerProtection:
- Page diffs
?type=revision?action=history?diff=1234?oldid=1234
- Special:RecentChangesLinked
- Special:WhatLinksHere
Frequently asked questions
- Does this extension harm SEO or otherwise prevent Google from crawling my wiki?
- This extension in no way prevents traditional legitimate robots.txt-respecting crawlers (such as those from Google) from accessing content pages of wikis with CrawlerProtection installed.
- Does hiding the history and diff pages from users violate the Creative Commons license?
- There is no requirement in any Creative Commons license to also show the history of a page. Therefore, this extension wouldn't violate the terms of any Creative Commons license. (This is not legal advice.)
Future plans
CrawlerProtection will continue to have features added to it with the community's support.
MediaWiki 1.43 support is currently being added.