How to use robot.txt in blogger/wordpress

As a blogger you may have heard about robot.txt at least once and till now you aren't sure about what is the use of robot.txt ? Then you are at the right place. First of all , Robot.txt is a protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers like , Google bot and Bing bot.Website owners can use the /robots.txt file to give instructions about their site to web robots. These instructions includes sitemap of your site or blog and Directories or pages you don't want to include in search result. Search engine robots uses robot.txt as a permission to crawl your site data which can they index later to search results.


  • How to use robot.txt in blogger ?

Step 1: Go to blogger > Setting > Search Preferences.

Step 2: Edit or Enable Custom robot.txt from Crawler and Indexing Section.


Step 3: Copy and paste below codes , add your sitemap and pages you want to disallow and save changes.

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: Page URL you don't want to appear in search 
Disallow: Page URL you don't want to appear in search 
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml


Change red bold text with required data and save changes.

  • How to use robot.txt in wordpress ?
Step 1: Open any of your favorite text editing software, You can also use notepad (if using windows).

Step 2: Copy and paste below codes to text editor.


User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: Page URL you don't want to appear in search 
Disallow: Page URL you don't want to appear in search 
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml


Step 3: Change red bold text with your sitemap and pages you don't want crawlers to access.

Step 4: Save this file as robot.txt and upload to your root directory.

Post a comment

13 Comments

  1. Thanks for taking the time to share this article here about the robots.txt file. Your article is very informative and useful to know more about the use of the robots.txt file in blogger and websites. website design Portland Maine provides the best web designing solutions at the affordable price.

    ReplyDelete
  2. Very informative article, Which you have shared here about the robots.txt files. After reading your article I got very much information about the use of robots.txt file in websites and It resolved many of my doubts. Thanks for sharing this article here. website development miami

    ReplyDelete
  3. This information is meaningful and magnificent, which you have shared here about robot.txt files. I am impressed by the details that you have shared in this post and It reveals how nicely you understand this subject. If anyone looking for the Web Design in Ireland, webie is the best for you.

    ReplyDelete
  4. You made some decent points there. I looked on the internet for the issue and found most individuals will go along with with your website.

    ReplyDelete
  5. You have shared a nice article here about the robots.txt files. Your article is very informative and I will share it with my other friends as the information is really very useful. Keep sharing your excellent work. Singapore web designer

    ReplyDelete
  6. I had an amazing experience with personally Affiliate marketing services. 12MinuteAffiliate of whether we enable you to determine that the best choice target audience is really otherwise you pick the industry and concentrate on information -- we are able to give you the information, Great function!

    ReplyDelete
  7. You have really selected the suitable topic; this is one of my favorite blogs.

    ReplyDelete
  8. I was searching for many blogs site and now finally I have got this cool place with lots of information.
    web design company san francisco

    ReplyDelete
  9. It’s a classic great for me to go to this blog site, it offers helpful suggestions
    UX design services

    ReplyDelete
  10. I read your post and got it quite informative. I couldn't find any knowledge on this matter prior to. I would like to thanks for sharing this article here.

    San Antonio website design

    ReplyDelete
  11. Great article, very useful and explanation. Your post is extremely incredible. we provide Mobile App Development Company in Dubai at affordable charges. to know more visit our website.

    ReplyDelete
  12. Just wanted to say I love reading your blog. we provide Design Agency for Medical Companies Orange County at affordable charges. to know more visit our website.

    ReplyDelete
  13. The team’s wide range of expertise and dedication to communicating brand agency San Francisco efficiently make them very valuable

    ReplyDelete