Acenet Knowledgebase - Recent changes [en]

MediaWiki Limit Crawler Resource Usage

Docs admin — Wed, 02 Sep 2026 16:49:27 GMT

wrong syntax

← Older revision		Revision as of 12:49, 2 September 2026
Line 94:		Line 94:
	In <code>LocalSettings.php</code>:		In <code>LocalSettings.php</code>:

	<~~syntaxhighlight lang="php"~~>		<pre>
	$wgMiserMode = true;		$wgMiserMode = true;
	</~~syntaxhighlight~~>		</pre>

	This setting exists specifically for this situation. It disables the most expensive special pages and serves cached results for the remainder instead of recomputing them on each request. Wikipedia runs with it enabled. On a small or medium wiki there is no meaningful downside.		This setting exists specifically for this situation. It disables the most expensive special pages and serves cached results for the remainder instead of recomputing them on each request. Wikipedia runs with it enabled. On a small or medium wiki there is no meaningful downside.
Line 106:		Line 106:
	By default MediaWiki regenerates every page from the database on every request. For logged-in users that is unavoidable. For anonymous visitors — which includes all crawler traffic — it is pure waste.		By default MediaWiki regenerates every page from the database on every request. For logged-in users that is unavoidable. For anonymous visitors — which includes all crawler traffic — it is pure waste.

	<~~syntaxhighlight lang="php"~~>		<pre>
	$wgMainCacheType = CACHE_ACCEL;		$wgMainCacheType = CACHE_ACCEL;
	$wgUseFileCache = true;		$wgUseFileCache = true;
	$wgFileCacheDirectory = "$IP/cache";		$wgFileCacheDirectory = "$IP/cache";
	</~~syntaxhighlight~~>		</pre>

	File cache is the important one. It writes rendered HTML to disk and serves anonymous page views directly from it, skipping the parser and most database work. Crawler load typically falls by an order of magnitude.		File cache is the important one. It writes rendered HTML to disk and serves anonymous page views directly from it, skipping the parser and most database work. Crawler load typically falls by an order of magnitude.
Line 124:		Line 124:
	Those requests have to be refused before PHP starts. In <code>.htaccess</code> at the document root:		Those requests have to be refused before PHP starts. In <code>.htaccess</code> at the document root:

	<~~syntaxhighlight lang="apache"~~>		<pre>
	RewriteCond %{HTTP_COOKIE} !(UserID\|_session) [NC]		RewriteCond %{HTTP_COOKIE} !(UserID\|_session) [NC]
	RewriteCond %{QUERY_STRING} (^\|&)returnto= [NC,OR]		RewriteCond %{QUERY_STRING} (^\|&)returnto= [NC,OR]
	RewriteCond %{QUERY_STRING} title=Special(%3A\|:)(ListFiles\|Contributions\|PrefixIndex\|Log) [NC]		RewriteCond %{QUERY_STRING} title=Special(%3A\|:)(ListFiles\|Contributions\|PrefixIndex\|Log) [NC]
	RewriteRule ^/?w/index\.php$ - [F,L]		RewriteRule ^/?w/index\.php$ - [F,L]
	</~~syntaxhighlight~~>		</pre>

	The cookie condition is what makes this safe. Visitors who are logged in or hold an active session still reach these pages normally; only anonymous requests to the combinatorial URLs receive a 403.		The cookie condition is what makes this safe. Visitors who are logged in or hold an active session still reach these pages normally; only anonymous requests to the combinatorial URLs receive a 403.

W/index.php

Docs admin — Wed, 02 Sep 2026 16:48:24 GMT

Docs admin moved page W/index.php to MediaWiki Limit Crawler Resource Usage without leaving a redirect Misspelled title

New page

__TOC__

A default MediaWiki installation exposes an effectively unlimited number of expensive, uncacheable URLs. Search engines, AI crawlers and commercial scrapers will find them and request them continuously. On shared hosting this exhausts the account's resource limits and takes the site offline for real visitors.

This is one of the most common causes of resource-limit errors on MediaWiki sites, and every fix below is a documented MediaWiki recommendation rather than a workaround.

== Symptoms ==

* Visitors see '''508 Resource Limit Is Reached''' or '''503 Service Unavailable''', often intermittently.
* The site is slow or unreachable while the server itself is idle.
* Errors arrive in bursts lasting a few minutes, at no consistent time of day.
* Access logs show large numbers of requests to <code>/w/index.php</code> with <code>title=Special:...</code> in the query string.
* Requests come from many different IP addresses, each sending only one or two requests.

The burst pattern is the key signal. Ordinary traffic growth is gradual. Crawler-induced exhaustion is sudden, brief, and repeats at unpredictable intervals.

== Why MediaWiki is affected more than most applications ==

MediaWiki serves content from two separate URL spaces:

* '''The article path''' — <code>/wiki/Page_Name</code>. Real content. Cacheable. A finite number of pages.
* '''The script path''' — <code>/w/index.php?...</code>. Everything else: editing, history, logs, search, user pages, login. Generated live from the database on every request.

Nearly all crawler damage happens in the second space, for three reasons that compound.

=== Special pages are generated, not stored ===

Pages in the <code>Special:</code> namespace are database queries rendered on demand. <code>Special:Contributions</code>, <code>Special:ListFiles</code>, <code>Special:PrefixIndex</code>, <code>Special:Log</code> and <code>Special:RecentChanges</code> are excluded from the parser cache by design, because their output changes constantly.

Every request means a full PHP startup plus live SQL. There is no cheap path.

=== Every page links to them ===

The sidebar, footer and toolbox on every article link into the <code>Special:</code> namespace. A crawler that discovers a single article discovers the entire machinery within one hop.

=== The URL space is unbounded ===

MediaWiki's login link records where the visitor came from, so it can return them there afterwards. That means there is not one login URL — there is one for every page a visitor could have been on, multiplied by every query variant that page could have carried:

<pre>
/w/index.php?title=Special:UserLogin&returnto=Special:Log/block&returntoquery=page=User:203.0.113.9
/w/index.php?title=Special:UserLogin&returnto=Special:PrefixIndex/User:198.51.100.4/&returntoquery=printable=yes
/w/index.php?title=Special:ListFiles&user=203.0.113.44
/w/index.php?printable=yes&title=User_talk:198.51.100.7
</pre>

Modifiers such as <code>printable=yes</code>, <code>action=history</code>, <code>oldid=</code> and <code>diff=</code> multiply the space again.

The result is a small wiki presenting a mathematically infinite set of distinct URLs, almost none of which are worth indexing. A crawler cannot detect this. It sees new links indefinitely and keeps requesting them.

{| class="wikitable"
! Factor !! Effect
|-
| Special pages bypass the parser cache || Every request costs full PHP + SQL
|-
| Sidebar and footer link to Special pages || Crawlers reach them from any article
|-
| <code>returnto</code> encodes the previous page || One login URL per page, per query variant
|-
| <code>printable</code>, <code>action</code>, <code>oldid</code>, <code>diff</code> || Multiplies every URL above
|}

== Fixes ==

Apply these in order. The first two take minutes and carry no risk. The last one is the only step that can break a working site.

=== 1. robots.txt ===

This is the single highest-value change and is recommended in MediaWiki's own manual. Most installations never do it.

Place the following at the '''document root''' of the domain — <code>/robots.txt</code>, not inside <code>/w/</code>:

<pre>
User-agent: *
Disallow: /w/
Allow: /w/load.php
</pre>

Content under <code>/wiki/</code> remains fully indexable. The machinery under <code>/w/</code> stops being crawled.

The <code>load.php</code> exception matters: that endpoint serves the site's CSS and JavaScript. Search engines penalise pages they cannot render, so it must stay reachable.

To exclude a specific crawler entirely — for example if the site's content should not be used for AI training — add a matching block. Well-behaved crawlers publish their user-agent string and honour these directives:

<pre>
User-agent: GPTBot
Disallow: /
</pre>

Whether to do this is an editorial decision about the content, not a technical one.

=== 2. Enable miser mode ===

In <code>LocalSettings.php</code>:

<syntaxhighlight lang="php">
$wgMiserMode = true;
</syntaxhighlight>

This setting exists specifically for this situation. It disables the most expensive special pages and serves cached results for the remainder instead of recomputing them on each request. Wikipedia runs with it enabled. On a small or medium wiki there is no meaningful downside.

=== 3. Enable caching ===

This is the underlying weakness, and fixing it is what makes the site resilient rather than merely less attractive to crawlers.

By default MediaWiki regenerates every page from the database on every request. For logged-in users that is unavoidable. For anonymous visitors — which includes all crawler traffic — it is pure waste.

<syntaxhighlight lang="php">
$wgMainCacheType = CACHE_ACCEL;
$wgUseFileCache = true;
$wgFileCacheDirectory = "$IP/cache";
</syntaxhighlight>

File cache is the important one. It writes rendered HTML to disk and serves anonymous page views directly from it, skipping the parser and most database work. Crawler load typically falls by an order of magnitude.

If the hosting account has Redis or Memcached available, point <code>$wgMainCacheType</code> at it instead of <code>CACHE_ACCEL</code>. Check with the hosting provider before assuming either is present.

Confirm the installed MediaWiki version at <code>Special:Version</code> before applying cache settings — some options have been renamed or removed across releases, and the current manual for that specific version is authoritative.

=== 4. Reject the worst URLs at the web server ===

The first three steps handle every crawler that follows the rules. They do nothing about scrapers that rotate user-agent strings and route through residential proxy networks, which ignore <code>robots.txt</code> entirely.

Those requests have to be refused before PHP starts. In <code>.htaccess</code> at the document root:

<syntaxhighlight lang="apache">
RewriteCond %{HTTP_COOKIE} !(UserID|_session) [NC]
RewriteCond %{QUERY_STRING} (^|&)returnto= [NC,OR]
RewriteCond %{QUERY_STRING} title=Special(%3A|:)(ListFiles|Contributions|PrefixIndex|Log) [NC]
RewriteRule ^/?w/index\.php$ - [F,L]
</syntaxhighlight>

The cookie condition is what makes this safe. Visitors who are logged in or hold an active session still reach these pages normally; only anonymous requests to the combinatorial URLs receive a 403.

A 403 returned by mod_rewrite costs microseconds. A rendered <code>Special:</code> page costs a full application slot.

{{Warning|Test this rule before relying on it. An error in the cookie condition will lock legitimate users out of logging in. Verify that you can log in, edit, and view page history with the rule active.}}

== Verifying the fix ==

After deploying, confirm the change in the raw access logs rather than in a dashboard. Substitute the correct log path for the hosting account.

Count requests to the script path over a recent window:

<pre>
awk '$7 ~ /index\.php/' access_log | wc -l
</pre>

Break down response codes, looking for the disappearance of 508 and 503:

<pre>
awk '{print $9}' access_log | sort | uniq -c | sort -rn
</pre>

Identify which clients are still reaching the script path:

<pre>
awk -F'"' '$0 ~ /index\.php/ {print $6}' access_log | sort | uniq -c | sort -rn | head
</pre>

Expect the total to fall sharply within a day, as crawlers re-read <code>robots.txt</code> on their own schedule rather than immediately.

=== Reading the user-agent results ===

* '''One user-agent, high count, consistent format.''' A declared crawler. Verify it before blocking — most major operators publish their address ranges, and a forged user-agent is common. Reverse DNS is not a reliable check on its own.
* '''Several near-identical browser strings with similar counts, then a sharp drop-off.''' Automation. Real browser populations do not distribute evenly across a handful of versions.
* '''Many distinct IP addresses sending one request each.''' A residential proxy pool. Blocking individual addresses will not work; the source rotates on every burst.

== Common misconceptions ==

'''"The pages already send <code>noindex</code>, so crawlers leave them alone."'''

Recent MediaWiki versions do send <code>noindex,nofollow</code> on <code>Special:</code> pages. That instructs a crawler not to index the result — but the request has already been made and the resources already spent by the time the header is read. Only <code>robots.txt</code> prevents the request itself.

'''"We will block the offending IP addresses."'''

Effective against a single misbehaving crawler, useless against proxy pools. Each burst typically arrives from a different source, so blocking is permanently one step behind.

'''"The server needs more memory."'''

Occasionally true, usually not. A properly cached wiki serves this traffic comfortably within standard shared-hosting limits. Raising limits without fixing the cause postpones the outage rather than preventing it.

'''"This is an attack."'''

Almost never. The overwhelming majority of these incidents are ordinary crawlers encountering a site that offers infinite expensive URLs. The request volume is usually modest — a few thousand requests spread over hours. What makes it fatal is cost per request, not request rate.

W/index.php

Docs admin — Wed, 02 Sep 2026 15:11:38 GMT

Created page with "__TOC__ A default MediaWiki installation exposes an effectively unlimited number of expensive, uncacheable URLs. Search engines, AI crawlers and commercial scrapers will find them and request them continuously. On shared hosting this exhausts the account's resource limits and takes the site offline for real visitors. This is one of the most common causes of resource-limit errors on MediaWiki sites, and every fix below is a documented MediaWiki recommendation rather tha..."

New page