How to use robot.txt in blogger/wordpress

Livethreads 04:30
As a blogger you may have heard about robot.txt at least once and till now you aren't sure about what is the use of robot.txt ? Then you are at the right place. First of all , Robot.txt is a protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers like , Google bot and Bing bot.Website owners can use the /robots.txt file to give instructions about their site to web robots. These instructions includes sitemap of your site or blog and Directories or pages you don't want to include in search result. Search engine robots uses robot.txt as a permission to crawl your site data which can they index later to search results.


  • How to use robot.txt in blogger ?

Step 1: Go to blogger > Setting > Search Preferences.

 Step 2: Edit or Enable Custom robot.txt from Crawler and Indexing Section.


Step 3: Copy and paste below codes , add your sitemap and pages you want to disallow and save changes.

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: Page URL you don't want to appear in search 
Disallow: Page URL you don't want to appear in search 
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml


Change red bold text with required data and save changes.

  • How to use robot.txt in wordpress ?
Step 1: Open any of your favorite text editing software, You can also use notepad (if using windows).

Step 2: Copy and paste below codes to text editor.


User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: Page URL you don't want to appear in search 
Disallow: Page URL you don't want to appear in search 
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml


Step 3: Change red bold text with your sitemap and pages you don't want crawlers to access.

Step 4: Save this file as robot.txt and upload to your root directory.
Tags:

Post a comment

10 Comments

  1. theportwebdesign22 April 2019 at 14:46

    Thanks for taking the time to share this article here about the robots.txt file. Your article is very informative and useful to know more about the use of the robots.txt file in blogger and websites. website design Portland Maine provides the best web designing solutions at the affordable price.

    ReplyDelete
  2. First Image Consulting24 April 2019 at 18:15

    Very informative article, Which you have shared here about the robots.txt files. After reading your article I got very much information about the use of robots.txt file in websites and It resolved many of my doubts. Thanks for sharing this article here. website development miami

    ReplyDelete
  3. David  aarons8 May 2019 at 17:31

    This information is meaningful and magnificent, which you have shared here about robot.txt files. I am impressed by the details that you have shared in this post and It reveals how nicely you understand this subject. If anyone looking for the Web Design in Ireland, webie is the best for you.

    ReplyDelete
  4. Thomas Belmont12 August 2019 at 04:08

    You made some decent points there. I looked on the internet for the issue and found most individuals will go along with with your website.

    ReplyDelete
  5. Laura Bush5 September 2019 at 17:53

    You have shared a nice article here about the robots.txt files. Your article is very informative and I will share it with my other friends as the information is really very useful. Keep sharing your excellent work. Singapore web designer

    ReplyDelete
  6. Gavin13 June 2020 at 00:57

    I had an amazing experience with personally Affiliate marketing services. 12MinuteAffiliate of whether we enable you to determine that the best choice target audience is really otherwise you pick the industry and concentrate on information -- we are able to give you the information, Great function!

    ReplyDelete
  7. 100kblueprint4.org12 July 2020 at 03:24

    You have really selected the suitable topic; this is one of my favorite blogs.

    ReplyDelete
  8. Japness27 July 2020 at 04:28

    I was searching for many blogs site and now finally I have got this cool place with lots of information.
    web design company san francisco

    ReplyDelete
  9. Cynthia Boyd1 September 2020 at 03:23

    It’s a classic great for me to go to this blog site, it offers helpful suggestions
    UX design services

    ReplyDelete
  10. funnelboostmedia14 September 2020 at 05:36

    I read your post and got it quite informative. I couldn't find any knowledge on this matter prior to. I would like to thanks for sharing this article here.

    San Antonio website design

    ReplyDelete