AdSense Friendly Custom Robots.txt for Blogger

Bloggers who have SEO Knowledge, should know about robots.txt and its use. But it is quite unfamiliar to the new bloggers. But some of them might be interested in Blogger's search preferences and they must ask about what is Custom robots.txt? Well this is not so easy to explain. To the new bloggers, the explanation might be confusing. In order to explain the term robots.txt, I've to tell about search bots or search crawlers. But I will try to explain the term clearly. 


Suppose, you've opened a blog in blogger. continuously you're publishing new posts. If everything is okay, your post will appear in Google Search within few hours. If your post is matched with someone's search topic, your site will appear in the search results. How it's possible? 


Blogger's Default robots.txt:

Search engines scan your site for new posts continuously. This is known as crawling. Search bots crawl your site regularly and index your posts in their directory. By default, blogger comes with a robots.txt for your site that helps search engines crawl your site. 

But as we know, the default or general settings never guarantee the accurate result. So we need to customize the settings as our requirement. Same rule is applicable for blogger's robots.txt. 


Custom Robots.txt for Blogger Blog:

Today you will learn how to customize the robots.txt of blogger. There is a rumor that the default robots.txt can't index the old posts properly. If you have a proper custom robots.txt, your posts will be indexed properly. In my last post, I discussed about creating XML sitemap for blogger blog. There I gave example of XML sitemap. You can use that sitemap as your custom robots.txt content. 

Look at the above image. This is how you can customize your robots.txt element. Or use the instruction below:

Sign in to Blogger Dashboard > Settings > Search Preferences > Crawler and Indexing > Custom robots.txt > Enable? > Yes > Now add the code below > Save Changes. 

# Blogger Sitemap generated on 2013.11.06
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://munnamark.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

Here you just need to change the site URL. Use the word allin1stuff4pc with yours. And for sure, this will help search engines for better crawling of your site.

AdSense Friendly Custom Robots.txt for Blogger:

This custom robots.txt might be less useful for AdSense enabled blog. If you use this type of custom robots.txt in your blog, AdSense will not be able to show relevant ads on your site. Consequently your ads impression you will affected seriously. And ultimately, your revenue will fall.

But a little change in that could save you all the day. If your site is AdSense enabled, then use the following code: 

User-agent: Mediapartners-Google
Disallow:

# Blogger Sitemap generated on 2013.11.04
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://munnamark.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

Look, only the blue part has been added with the previous code. If you're working with AdSense then consider using this code as your robots.txt element. Actually the blue part is used by Google AdSense only. 

If you use this code (User-agent: Mediapartners-Google Disallow:), AdSense will be able to crawl your posts properly. Then it will show relevant ads on your site. And the other part of the code will be used by the search engines for indexing your contents.