Robots.txt file plays vital role in SEO. It instructs search engine robots whether website/webpage should be crawled or not. This file should present in root path.
If a search engine robot wants to visit the web page (say, www.domain.com/about-me/), before it does so, it will visit the page www.domain.com/robots.txt to check is there any instruction to crawl the webpage www.domain.com/about-me/
Continue reading...
