Setting up Robots.txt

What Is Robots.txt? Robots.txt is a text file used to instruct search engine bots (also known as crawlers, robots, or spiders) how to crawl and index website pages. Ideally, a robots.txt file is placed in the top-level directory of your website so that robots can access its instructions right away. Why Is Robots.txt Important? Correct robots.txt operation ensures that search engine bots are routed to required pages, disallowing content duplicates that lead to a fall in position. For that reason, you should make sure your site has a thoughtfully created robot.txt file. If a robots.txt file is set up incorrectly, it can cause multiple indexing mistakes. So, every time you start a new SEO campaign, check your robots.txt file with Google’s robots texting tool. Do not forget: If everything is correctly set up, a robots.txt file will speed up the indexing process. Robots.txt on the Web Yet, do not forget that any robots.txt file is publicly available on the web. To access a robots.txt file, simply type: www.websiteexample. com/robots.txt. 

 BEST PRACTICES FOR SETTING UP META ROBOTS TAGS & ROBOTS.TXT 4 This availability means that: You cannot secure or hide any data within it. Bad robots and malicious crawlers can take advantage of a robots. txt file, using it as a detailed map to navigate your most valuable web pages. Also, keep in mind that robots.txt commands are actually directives. This means that search bots can crawl and index your site, even if you instruct them not to. The good news is that most search engines (like Google, Bing, Yahoo, and Yandex) honor robots.txt directives. Robots.txt files definitely have drawbacks. Nonetheless, I strongly recommend you make them an integral part of every SEO campaign. Google recognizes and honors robots.txt directives and, in most cases, having Google under your belt is more than enough. Robots.txt Basics The robots.txt file should: Contain the usual text in the UTF-8 encoding, which consists of records (lines), divided by symbols. Be situated at the root of the website host to which it applies. 

 BEST PRACTICES FOR SETTING UP META ROBOTS TAGS & ROBOTS.TXT 4 Be unique. Contain not more than 1,024 rules. Be under 500KB. Google bots find all the content available for indexing if: There is no robots.txt file. A robots.txt file isn’t shown in the text format. They do not receive the 200 OK response. Note: You can, but are not allowed to, mention the byte order mark (BOM) at the beginning of the robots.txt file, as it will be ignored by bots. The standard recommends the use of a newline before each Useragent directive. If your encoding contains symbols beyond the UTF-8, bots may analyze the file incorrectly. They will execute the valid entry only, ignoring the rest of your content without notifying you about the mistake. 

 BEST PRACTICES FOR SETTING UP META ROBOTS TAGS & ROBOTS.TXT 4 Robots.txt Structure Robots.txt File consists of: One or several User-agent directives, meant for robots of various search engines. Disallow and Allow directives that allow or restrict indexing. Sitemap directives. Disallow directives forbid indexing, Allow directives allow indexing. Each record consists of the directory field (allow, disallow, host or user-agent), two-spot and a value. Empty spaces are not required, but recommended for better readability. You can place comments anywhere in the file and mark them with the # symbol. “#” is the symbol meant for comment descriptions. Google bots do not count everything mentioned between the # symbol and the next newline. The general format is: :<#comment (optional)>. Empty spaces at the beginning and the end will be ignored. Letter case for element does not matter. Letter case might be important for the element, depending on the element. 

 BEST PRACTICES FOR SETTING UP META ROBOTS TAGS & ROBOTS.TXT 4 What to Hide with Robots.txt Obviously, you do not want to show search engines your private technical page, customers’ personal data, and duplicate content. Robots.txt files can be used to exclude certain directories, categories, and pages from search. To that end, use the “disallow” directive. Here are some pages you should hide using a robots.txt file: Pages with duplicate content Pagination pages On-site search pages Dynamic product and service pages Account pages Admin pages Shopping cart Chats Thank-you pages 

 BEST PRACTICES FOR SETTING UP META ROBOTS TAGS & ROBOTS.TXT 4 BEST PRACTICES FOR SETTING UP META ROBOTS TAGS & ROBOTS.TXT 4 How to Use Robots.txt Robots.txt files are pretty flexible and can be used in many ways. Their main benefit, however, is that they enable SEO experts to “allow” or “disallow” multiple pages at once without having to access the code of page by page. Here is an example of how I instruct Googlebot to avoid crawling and indexing all pages related to user accounts, cart, and multiple dynamic pages that are generated when users look for products in the search bar or sort them by price, and so on.