As a blogger you may have heard about robot.txt at least once and till now you aren't sure about what is the use of robot.txt ? Then you are at the right place. First of all , Robot.txt is a protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers like , Google bot and Bing bot.Website owners can use the /robots.txt file to give instructions about their site to web robots. These instructions includes sitemap of your site or blog and Directories or pages you don't want to include in search result. Search engine robots uses robot.txt as a permission to crawl your site data which can they index later to search results.
Step 1: Go to blogger > Setting > Search Preferences.
Step 2: Edit or Enable Custom robot.txt from Crawler and Indexing Section.
Step 3: Copy and paste below codes , add your sitemap and pages you want to disallow and save changes.
Change red bold text with required data and save changes.
- How to use robot.txt in blogger ?
Step 1: Go to blogger > Setting > Search Preferences.
Step 2: Edit or Enable Custom robot.txt from Crawler and Indexing Section.
Step 3: Copy and paste below codes , add your sitemap and pages you want to disallow and save changes.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: Page URL you don't want to appear in search
Disallow: Page URL you don't want to appear in search
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml
Disallow:
User-agent: *
Disallow: Page URL you don't want to appear in search
Disallow: Page URL you don't want to appear in search
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml
Change red bold text with required data and save changes.
- How to use robot.txt in wordpress ?
Step 1: Open any of your favorite text editing software, You can also use notepad (if using windows).
Step 2: Copy and paste below codes to text editor.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: Page URL you don't want to appear in search
Disallow: Page URL you don't want to appear in search
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml
User-agent: *
Disallow: Page URL you don't want to appear in search
Disallow: Page URL you don't want to appear in search
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml
Step 3: Change red bold text with your sitemap and pages you don't want crawlers to access.
Step 4: Save this file as robot.txt and upload to your root directory.
Thanks for taking the time to share this article here about the robots.txt file. Your article is very informative and useful to know more about the use of the robots.txt file in blogger and websites. website design Portland Maine provides the best web designing solutions at the affordable price.
ReplyDeleteVery informative article, Which you have shared here about the robots.txt files. After reading your article I got very much information about the use of robots.txt file in websites and It resolved many of my doubts. Thanks for sharing this article here. website development miami
ReplyDeleteThis information is meaningful and magnificent, which you have shared here about robot.txt files. I am impressed by the details that you have shared in this post and It reveals how nicely you understand this subject. If anyone looking for the Web Design in Ireland, webie is the best for you.
ReplyDeleteYou made some decent points there. I looked on the internet for the issue and found most individuals will go along with with your website.
ReplyDeleteYou have shared a nice article here about the robots.txt files. Your article is very informative and I will share it with my other friends as the information is really very useful. Keep sharing your excellent work. Singapore web designer
ReplyDelete