Robots.txt - Revision history

Docs admin: Docs admin moved page What is a robots.txt file? to Robots.txt

2012-12-27T16:23:09Z

Docs admin moved page What is a robots.txt file? to Robots.txt

← Older revision	Revision as of 12:23, 27 December 2012
(No difference)

Docs admin at 16:19, 27 December 2012

2012-12-27T16:19:50Z

← Older revision		Revision as of 12:19, 27 December 2012
Line 55:		Line 55:

	[[Category:Technical Support FAQ]]		[[Category:Technical Support FAQ]]
			[[Category:SEO]]

Docs admin at 21:51, 9 October 2012

2012-10-09T21:51:06Z

← Older revision		Revision as of 17:51, 9 October 2012
Line 3:		Line 3:
	This article will explain how a robots.txt file within your account can help you manage the bots that crawl your site. The first things to understand are the options you can put into your robots.txt file:		This article will explain how a robots.txt file within your account can help you manage the bots that crawl your site. The first things to understand are the options you can put into your robots.txt file:

	==User-agent==		==robots.txt directives==

			===User-agent===

	<syntaxhighlight lang="bash">User-agent:</syntaxhighlight>		<syntaxhighlight lang="bash">User-agent:</syntaxhighlight>
Line 9:		Line 11:
	This specifies the bot you want to pass instructions to. This should be the first line in your robots.txt file. You can either specify all bots using the wildcard symbol * or you can name them individually (bing, yahoo, etc).		This specifies the bot you want to pass instructions to. This should be the first line in your robots.txt file. You can either specify all bots using the wildcard symbol * or you can name them individually (bing, yahoo, etc).

	==Disallow==		===Disallow===

	<syntaxhighlight lang="bash">Disallow:</syntaxhighlight>		<syntaxhighlight lang="bash">Disallow:</syntaxhighlight>
Line 15:		Line 17:
	This is the line you will most commonly follow the previous option with. Following this option, you will designate a folder or file that you do not want to be 'crawled'. For example, you can specify simply / to prevent any of your files from being 'crawled' or you can get more specific and designate something like /images/image.jpg to stop that specific image file from being 'crawled'.		This is the line you will most commonly follow the previous option with. Following this option, you will designate a folder or file that you do not want to be 'crawled'. For example, you can specify simply / to prevent any of your files from being 'crawled' or you can get more specific and designate something like /images/image.jpg to stop that specific image file from being 'crawled'.

	==Allow==		===Allow===

	<syntaxhighlight lang="bash">Allow:</syntaxhighlight>		<syntaxhighlight lang="bash">Allow:</syntaxhighlight>
Line 21:		Line 23:
	This line will specify any exceptions to the disallow rule that you want certain bots to follow. Note that only the major search bots follow the allow option overide, so Bing, Yahoo, and etc.		This line will specify any exceptions to the disallow rule that you want certain bots to follow. Note that only the major search bots follow the allow option overide, so Bing, Yahoo, and etc.

	==Crawl-delay==		===Crawl-delay===

	<syntaxhighlight lang="bash">Crawl-delay:</syntaxhighlight>		<syntaxhighlight lang="bash">Crawl-delay:</syntaxhighlight>

	This is the most commonly used option in a robots.txt file as it slows down the rate at which bots hit your site. This will prevent bots from consuming all the resources available for your site as they continue to index it. After the option, you will place the amount of seconds that you want to delay the bots by.		This is the most commonly used option in a robots.txt file as it slows down the rate at which bots hit your site. This will prevent bots from consuming all the resources available for your site as they continue to index it. After the option, you will place the amount of seconds that you want to delay the bots by.

			==Example robots.txt==

	So, lets say we configured a robots.txt file with the following options:		So, lets say we configured a robots.txt file with the following options:

Docs admin at 21:50, 9 October 2012

2012-10-09T21:50:12Z

@@ Line 1: / Line 1: @@
-<html> Search engine optimization (SEO) is an important thing in this day of  information technology. Most search engines utilize site crawlers,  spiders, or simply just bots to ensure their searches find the proper  results. However, not all bots are beneficial and sometimes even the  ones that are can cause issues on your site.<br /> <br /> This article will explain how a robots.txt file within your account can  help you manage the bots that crawl your site. The first things to  understand are the options you can put into your robots.txt file:<br /> <br /> <em><span style="text-decoration: underline;">User-agent:</span></em> - This specifies the bot you want to pass  instructions to. This should be the first line in your robots.txt file.  You can either specify all bots using the wildcard symbol * or you can  name them individually (bing, yahoo, etc). <br /> <br /> <span style="text-decoration: underline;"><em>Disallow:</em></span> - This is the line you will most commonly follow  the previous option with. Following this option, you will designate a  folder or file that you do not want to be 'crawled'. For example, you  can specify simply / to prevent any of your files from being 'crawled'  or you can get more specific and designate something like  /images/image.jpg to stop that specific image file from being 'crawled'.<br /> <br /> <span style="text-decoration: underline;"><em>Allow:</em></span> - This line will specify any exceptions to the  disallow rule that you want certain bots to follow. Note that only the  major search bots follow the allow option overide, so Bing, Yahoo, and  etc. <br /> <br /> <span style="text-decoration: underline;"><em>Crawl-delay:</em></span> - This is the most commonly used option in a  robots.txt file as it slows down the rate at which bots hit your site.  This will prevent bots from consuming all the resources available for  your site as they continue to index it. After the option, you will place  the amount of seconds that you want to delay the bots by.<br /> <br /> So, lets say we configured a robots.txt file with the following options:<br /> <br /> User-Agent: *<br /> Allow: /<br /> Disallow: /administrator<br /> Disallow: /cgi-bin<br /> <br /> That robots.txt file would allow all folders but the administrator and  cgi-bin folders to be crawled and indexed by all search engines. <br /> <br /> These options will help you to control the traffic of bots that access  your site.
+Search engine optimization (SEO) is an important thing in this day of  information technology. Most search engines utilize site crawlers,  spiders, or simply just bots to ensure their searches find the proper  results. However, not all bots are beneficial and sometimes even the  ones that are can cause issues on your site.
- Googlebot is also controlled by "Google Webmaster Tools" which you can  sign up/access here:<br /> <br /> <a href="http://www.google.com/webmasters/tools/" target="_blank">http://www.google.com/webmasters/tools/</a><br /> <br /> It is also important to know that malicious bots do not adhere to the  robots.txt file. You would need to block access from any malicious bots. If you have access to cPanel, you can block the malicious bot's IP(s) from accessing your  site using cPanel's 'IP Deny Manager'. You can view our knowledgebase  article on how to use cPanel's 'IP Deny Manager' here:
- <a title="Blocking an IP from your Website" href="https://esupport.acenet-inc.net/index.php?/Knowledgebase/Article/View/126/4/blocking-an-ip-from-your-website">Blocking an IP from your Website</a>
+This article will explain how a robots.txt file within your account can  help you manage the bots that crawl your site. The first things to  understand are the options you can put into your robots.txt file:
   You can view more information on robots.txt files here:
- <a title="Robots exclusion standard" href="http://en.wikipedia.org/wiki/Robots_exclusion_standard">http://en.wikipedia.org/wiki/Robots_exclusion_standard</a> </html> [[Category:Technical Support FAQ]]
+==User-agent==

Docs admin: test

2012-09-28T19:35:49Z

test

New page

<html> Search engine optimization (SEO) is an important thing in this day of information technology. Most search engines utilize site crawlers, spiders, or simply just bots to ensure their searches find the proper results. However, not all bots are beneficial and sometimes even the ones that are can cause issues on your site. This article will explain how a robots.txt file within your account can help you manage the bots that crawl your site. The first things to understand are the options you can put into your robots.txt file: User-agent: - This specifies the bot you want to pass instructions to. This should be the first line in your robots.txt file. You can either specify all bots using the wildcard symbol * or you can name them individually (bing, yahoo, etc). Disallow: - This is the line you will most commonly follow the previous option with. Following this option, you will designate a folder or file that you do not want to be 'crawled'. For example, you can specify simply / to prevent any of your files from being 'crawled' or you can get more specific and designate something like /images/image.jpg to stop that specific image file from being 'crawled'. Allow: - This line will specify any exceptions to the disallow rule that you want certain bots to follow. Note that only the major search bots follow the allow option overide, so Bing, Yahoo, and etc. Crawl-delay: - This is the most commonly used option in a robots.txt file as it slows down the rate at which bots hit your site. This will prevent bots from consuming all the resources available for your site as they continue to index it. After the option, you will place the amount of seconds that you want to delay the bots by. So, lets say we configured a robots.txt file with the following options: User-Agent: * Allow: / Disallow: /administrator Disallow: /cgi-bin That robots.txt file would allow all folders but the administrator and cgi-bin folders to be crawled and indexed by all search engines. These options will help you to control the traffic of bots that access your site.
Googlebot is also controlled by "Google Webmaster Tools" which you can sign up/access here: <a href="http://www.google.com/webmasters/tools/" target="_blank">http://www.google.com/webmasters/tools/</a> It is also important to know that malicious bots do not adhere to the robots.txt file. You would need to block access from any malicious bots. If you have access to cPanel, you can block the malicious bot's IP(s) from accessing your site using cPanel's 'IP Deny Manager'. You can view our knowledgebase article on how to use cPanel's 'IP Deny Manager' here:
<a title="Blocking an IP from your Website" href="https://esupport.acenet-inc.net/index.php?/Knowledgebase/Article/View/126/4/blocking-an-ip-from-your-website">Blocking an IP from your Website</a>
You can view more information on robots.txt files here:
<a title="Robots exclusion standard" href="http://en.wikipedia.org/wiki/Robots_exclusion_standard">http://en.wikipedia.org/wiki/Robots_exclusion_standard</a> </html> [[Category:Technical Support FAQ]]