Robots.txt is a file that exists on the basis Listing of every Web-site and can be employed to instruct search engines like google and yahoo on which directories/data files of the website they can crawl and include things like within their index. Although crawling a page, they identify its matter. https://news.rafeeg.ae