MediaWiki Limit Crawler Resource Usage
A default MediaWiki installation exposes an effectively unlimited number of expensive, uncacheable URLs. Search engines, AI crawlers and commercial scrapers will find them and request them continuously. On shared hosting this exhausts the account's resource limits and takes the site offline for real visitors.
This is one of the most common causes of resource-limit errors on MediaWiki sites, and every fix below is a documented MediaWiki recommendation rather than a workaround.
Symptoms
- Visitors see 508 Resource Limit Is Reached or 503 Service Unavailable, often intermittently.
- The site is slow or unreachable while the server itself is idle.
- Errors arrive in bursts lasting a few minutes, at no consistent time of day.
- Access logs show large numbers of requests to
/w/index.phpwithtitle=Special:...in the query string. - Requests come from many different IP addresses, each sending only one or two requests.
The burst pattern is the key signal. Ordinary traffic growth is gradual. Crawler-induced exhaustion is sudden, brief, and repeats at unpredictable intervals.
Why MediaWiki is affected more than most applications
MediaWiki serves content from two separate URL spaces:
- The article path —
/wiki/Page_Name. Real content. Cacheable. A finite number of pages. - The script path —
/w/index.php?.... Everything else: editing, history, logs, search, user pages, login. Generated live from the database on every request.
Nearly all crawler damage happens in the second space, for three reasons that compound.
Special pages are generated, not stored
Pages in the Special: namespace are database queries rendered on demand. Special:Contributions, Special:ListFiles, Special:PrefixIndex, Special:Log and Special:RecentChanges are excluded from the parser cache by design, because their output changes constantly.
Every request means a full PHP startup plus live SQL. There is no cheap path.
Every page links to them
The sidebar, footer and toolbox on every article link into the Special: namespace. A crawler that discovers a single article discovers the entire machinery within one hop.
The URL space is unbounded
MediaWiki's login link records where the visitor came from, so it can return them there afterwards. That means there is not one login URL — there is one for every page a visitor could have been on, multiplied by every query variant that page could have carried:
/w/index.php?title=Special:UserLogin&returnto=Special:Log/block&returntoquery=page=User:203.0.113.9 /w/index.php?title=Special:UserLogin&returnto=Special:PrefixIndex/User:198.51.100.4/&returntoquery=printable=yes /w/index.php?title=Special:ListFiles&user=203.0.113.44 /w/index.php?printable=yes&title=User_talk:198.51.100.7
Modifiers such as printable=yes, action=history, oldid= and diff= multiply the space again.
The result is a small wiki presenting a mathematically infinite set of distinct URLs, almost none of which are worth indexing. A crawler cannot detect this. It sees new links indefinitely and keeps requesting them.
| Factor | Effect |
|---|---|
| Special pages bypass the parser cache | Every request costs full PHP + SQL |
| Sidebar and footer link to Special pages | Crawlers reach them from any article |
returnto encodes the previous page |
One login URL per page, per query variant |
printable, action, oldid, diff |
Multiplies every URL above |
Fixes
Apply these in order. The first two take minutes and carry no risk. The last one is the only step that can break a working site.
1. robots.txt
This is the single highest-value change and is recommended in MediaWiki's own manual. Most installations never do it.
Place the following at the document root of the domain — /robots.txt, not inside /w/:
User-agent: * Disallow: /w/ Allow: /w/load.php
Content under /wiki/ remains fully indexable. The machinery under /w/ stops being crawled.
The load.php exception matters: that endpoint serves the site's CSS and JavaScript. Search engines penalise pages they cannot render, so it must stay reachable.
To exclude a specific crawler entirely — for example if the site's content should not be used for AI training — add a matching block. Well-behaved crawlers publish their user-agent string and honour these directives:
User-agent: GPTBot Disallow: /
Whether to do this is an editorial decision about the content, not a technical one.
2. Enable miser mode
In LocalSettings.php:
$wgMiserMode = true;
This setting exists specifically for this situation. It disables the most expensive special pages and serves cached results for the remainder instead of recomputing them on each request. Wikipedia runs with it enabled. On a small or medium wiki there is no meaningful downside.
3. Enable caching
This is the underlying weakness, and fixing it is what makes the site resilient rather than merely less attractive to crawlers.
By default MediaWiki regenerates every page from the database on every request. For logged-in users that is unavoidable. For anonymous visitors — which includes all crawler traffic — it is pure waste.
$wgMainCacheType = CACHE_ACCEL; $wgUseFileCache = true; $wgFileCacheDirectory = "$IP/cache";
File cache is the important one. It writes rendered HTML to disk and serves anonymous page views directly from it, skipping the parser and most database work. Crawler load typically falls by an order of magnitude.
If the hosting account has Redis or Memcached available, point $wgMainCacheType at it instead of CACHE_ACCEL. Check with the hosting provider before assuming either is present.
Confirm the installed MediaWiki version at Special:Version before applying cache settings — some options have been renamed or removed across releases, and the current manual for that specific version is authoritative.
4. Reject the worst URLs at the web server
The first three steps handle every crawler that follows the rules. They do nothing about scrapers that rotate user-agent strings and route through residential proxy networks, which ignore robots.txt entirely.
Those requests have to be refused before PHP starts. In .htaccess at the document root:
RewriteCond %{HTTP_COOKIE} !(UserID|_session) [NC]
RewriteCond %{QUERY_STRING} (^|&)returnto= [NC,OR]
RewriteCond %{QUERY_STRING} title=Special(%3A|:)(ListFiles|Contributions|PrefixIndex|Log) [NC]
RewriteRule ^/?w/index\.php$ - [F,L]
The cookie condition is what makes this safe. Visitors who are logged in or hold an active session still reach these pages normally; only anonymous requests to the combinatorial URLs receive a 403.
A 403 returned by mod_rewrite costs microseconds. A rendered Special: page costs a full application slot.
Verifying the fix
After deploying, confirm the change in the raw access logs rather than in a dashboard. Substitute the correct log path for the hosting account.
Count requests to the script path over a recent window:
awk '$7 ~ /index\.php/' access_log | wc -l
Break down response codes, looking for the disappearance of 508 and 503:
awk '{print $9}' access_log | sort | uniq -c | sort -rn
Identify which clients are still reaching the script path:
awk -F'"' '$0 ~ /index\.php/ {print $6}' access_log | sort | uniq -c | sort -rn | head
Expect the total to fall sharply within a day, as crawlers re-read robots.txt on their own schedule rather than immediately.
Reading the user-agent results
- One user-agent, high count, consistent format. A declared crawler. Verify it before blocking — most major operators publish their address ranges, and a forged user-agent is common. Reverse DNS is not a reliable check on its own.
- Several near-identical browser strings with similar counts, then a sharp drop-off. Automation. Real browser populations do not distribute evenly across a handful of versions.
- Many distinct IP addresses sending one request each. A residential proxy pool. Blocking individual addresses will not work; the source rotates on every burst.
Common misconceptions
"The pages already send noindex, so crawlers leave them alone."
Recent MediaWiki versions do send noindex,nofollow on Special: pages. That instructs a crawler not to index the result — but the request has already been made and the resources already spent by the time the header is read. Only robots.txt prevents the request itself.
"We will block the offending IP addresses."
Effective against a single misbehaving crawler, useless against proxy pools. Each burst typically arrives from a different source, so blocking is permanently one step behind.
"The server needs more memory."
Occasionally true, usually not. A properly cached wiki serves this traffic comfortably within standard shared-hosting limits. Raising limits without fixing the cause postpones the outage rather than preventing it.
"This is an attack."
Almost never. The overwhelming majority of these incidents are ordinary crawlers encountering a site that offers infinite expensive URLs. The request volume is usually modest — a few thousand requests spread over hours. What makes it fatal is cost per request, not request rate.