Following a defined procedure known as the robot exclusion protocol allows search engines to locate the “robots.txt” file that is hosted on a website. In… Understanding the Robot Exclusion Protocol