As a blogger you may have heard about robot.txt at least once and till now you aren't sure about what is the use of robot.txt ? Then you are at the right place. First of all , Robot.txt is a protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers like , Google bot and Bing bot.Website owners can use the /robots.txt file to give instructions about their site to web robots. These instructions includes sitemap of your site or blog and Directories or pages you don't want to include in search result. Search engine robots uses robot.txt as a permission to crawl your site data which can they index later to search results.
Step 1: Go to blogger > Setting > Search Preferences.
Step 2: Edit or Enable Custom robot.txt from Crawler and Indexing Section.
Step 3: Copy and paste below codes , add your sitemap and pages you want to disallow and save changes.
Change red bold text with required data and save changes.
- How to use robot.txt in blogger ?
Step 1: Go to blogger > Setting > Search Preferences.
Step 2: Edit or Enable Custom robot.txt from Crawler and Indexing Section.
Step 3: Copy and paste below codes , add your sitemap and pages you want to disallow and save changes.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: Page URL you don't want to appear in search
Disallow: Page URL you don't want to appear in search
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml
Disallow:
User-agent: *
Disallow: Page URL you don't want to appear in search
Disallow: Page URL you don't want to appear in search
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml
Change red bold text with required data and save changes.
- How to use robot.txt in wordpress ?
Step 1: Open any of your favorite text editing software, You can also use notepad (if using windows).
Step 2: Copy and paste below codes to text editor.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: Page URL you don't want to appear in search
Disallow: Page URL you don't want to appear in search
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml
User-agent: *
Disallow: Page URL you don't want to appear in search
Disallow: Page URL you don't want to appear in search
Allow: /
Sitemap: http://www.Yourdomain.com/sitemap.xml
Step 3: Change red bold text with your sitemap and pages you don't want crawlers to access.
Step 4: Save this file as robot.txt and upload to your root directory.
Thanks for taking the time to share this article here about the robots.txt file. Your article is very informative and useful to know more about the use of the robots.txt file in blogger and websites. website design Portland Maine provides the best web designing solutions at the affordable price.
ReplyDeleteVery informative article, Which you have shared here about the robots.txt files. After reading your article I got very much information about the use of robots.txt file in websites and It resolved many of my doubts. Thanks for sharing this article here. website development miami
ReplyDeleteThis information is meaningful and magnificent, which you have shared here about robot.txt files. I am impressed by the details that you have shared in this post and It reveals how nicely you understand this subject. If anyone looking for the Web Design in Ireland, webie is the best for you.
ReplyDeleteYou made some decent points there. I looked on the internet for the issue and found most individuals will go along with with your website.
ReplyDeleteYou have shared a nice article here about the robots.txt files. Your article is very informative and I will share it with my other friends as the information is really very useful. Keep sharing your excellent work. Singapore web designer
ReplyDeleteI had an amazing experience with personally Affiliate marketing services. 12MinuteAffiliate of whether we enable you to determine that the best choice target audience is really otherwise you pick the industry and concentrate on information -- we are able to give you the information, Great function!
ReplyDeleteYou have really selected the suitable topic; this is one of my favorite blogs.
ReplyDeleteI was searching for many blogs site and now finally I have got this cool place with lots of information.
ReplyDeleteweb design company san francisco
It’s a classic great for me to go to this blog site, it offers helpful suggestions
ReplyDeleteUX design services
I read your post and got it quite informative. I couldn't find any knowledge on this matter prior to. I would like to thanks for sharing this article here.
ReplyDeleteSan Antonio website design
Great article, very useful and explanation. Your post is extremely incredible. we provide Mobile App Development Company in Dubai at affordable charges. to know more visit our website.
ReplyDeleteJust wanted to say I love reading your blog. we provide Design Agency for Medical Companies Orange County at affordable charges. to know more visit our website.
ReplyDeleteThe team’s wide range of expertise and dedication to communicating brand agency San Francisco efficiently make them very valuable
ReplyDelete