What Is Robots.txt? A robots.txt file is a text file stored in a website’s root directory that gives web crawlers directions regarding which pages, folders… How to Use Robots.txt ?