Standard robots.txt file for latest Blogger 2021

The standard robots.txt file for Blogger controls how search engines crawl a website. That means it plays an important role in the search engine optimization of Blogger blogs. In this article, we will understand the best implementation of the standard robots.txt File for the Latest Blogger 2021.

What is the robots.txt file?

The robots.txt file informs search engines about what pages should and should not be crawled. Thus, it allows google to control the activity of search engine bots.

In robots.txt file, we declare user-agent, allow, disallow, sitemap functions for search engines like Google, Bing, Yandex, etc. Let’s understand what all these terms mean.

Normally, we use the robots meta tag for all search engines to crawl bots to index blog posts and pages across the entire web. But if you want to save crawl budget, block search engine bots in some parts of your website, you must understand robots.txt file for Blogger blog.

Analyze Blogger Blog’s Default Robots.txt File

To create the perfect custom robots.txt file for Blogger’s BlogSpot blog. First, we have to understand the structure of the Blogger blog and analyze the default robots.txt file.

By default, this file looks like:

User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: /search

Allow: /

Sitemap: https://www.example.com/sitemap.xml

  • The first line of this file declares the bot type  . This is Google AdSense, no use allowed. That means AdSense ads can appear on the entire site.
  • The next user agent is *  , which means all search engine bots   are  not allowed to  enter  search  pages   . That means disallowed for all search and label pages (same URL structure).
  • And the allow tag  specifies that all pages other than the disallow section will be allowed to crawl.
  • The next line  contains the post sitemap  for the Blogger blog.

This is an almost perfect file for controlling search engine bots and providing instructions for pages to crawl or not to crawl. Please note, here, what is allowed to crawl will not guarantee that the pages will be indexed.

But this file allows archive pages to be indexed, which can cause duplicate content issues. That means it will create garbage for the Blogger blog.

Create standard robots.txt file for latest Blogspot 2021

We have understood how to set the default robots.txt File to perform its function for Blogger blogs. Please optimize it for best SEO.

The default robots.txt file allows the repository to index the cause of the duplicate content problem. We can prevent duplicate content issues by preventing bots from crawling the archive. For this we have to apply the Disallow/20* rule to the robots.txt file. But this rule will stop crawling of pages. So to avoid this, we have to apply a new Allow rule to the /*.html section that allows bots to crawl posts and pages.

The default sitemap includes posts, not pages. So you have to add sitemaps for pages located in  https://domain.blogspot.com/sitemap-pages.xml  or  https://www.domain.com/sitemap-pages.xml  for custom domain .

So the standard robots.txt file for blogger would look like this.

User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: /search*

Disallow: /20*

Allow: /*.html

Sitemap: https://bblogger.net/sitemap.xml

Sitemap: https://www.bblogger.net/sitemap-pages.xml

You must replace  https://bblogger.net  with your Blogger domain or custom domain. For example, assuming your custom domain is www.abcd.com, the sitemap will be at https://www.abcd.com/sitemap.xml. Also you can check the current robots.txt at https://www. doanhaiblog.com/robots.txt

The file above, settings is best practice for robots.txt as well as for SEO. This will save your site’s crawl budget and will help your Blogger blog appear in search results. Along with that you have to write SEO friendly content to appear in search results.

For the best possible settings for robots.txt and the robots meta tag, try the advanced robots meta tag and the robots.txt file. This combination is one of the best practices for boosting Blogger blog SEO.

How to edit Blogspot Robots.txt file?

The Robots.txt file is always located at the base level of any web page. But in Blospot, there is no root access, so how to edit this robots.txt file?

Blogspot provides all the original file settings in its settings like robots.txt and ads.txt files. You must log in to your Blogger account and edit the robots.txt file.

Standard robots.txt file for latest Blogger 2021

  1. Go to Blogger Dashboard and click on settings option,
  2. Scroll down to crawlers and indexing section,
  3. Enable custom robots.txt with toggle button.
  4. Click on  custom robots.txt  , a window will open, paste the robots.txt file and update.

After updating the custom robots.txt file, test it by going to https://www.doanhaiblog.com/robots.txt, where www.doanhaiblog.com will be replaced with the address of your domain .

Conclusion

We have understood the functionality of the robots.txt file. Blogspot blog users can set the robots.txt file above for best results.

In the default robots.txt file, the archive is also allowed to be crawled, which causes duplicate content problems for search engines. And so search engines get confused about what to show in search results and don’t consider your pages for search results.

It means that the Robots tag is essential for the SEO of a website. You might consider combining both the robots.txt file and the robots meta tag in your Blogger blog for best results.

I hope you enjoyed this article. If you have any difficulty or question regarding Blospot or WordPress SEO, you can comment below.

Click to rate this post!
[Total: 0 Average: 0]
51
Subscribe
Notify of
guest
0 Góp ý
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x