What is a WordPress robots.txt file and do I need to worry about it? A robots.txt file is a file on your site that allows you to deny search engines access to certain files and folders. You can use it to block Google’s (and other search engines) bots from crawling certain pages on your site. … Continue reading How to Optimize WordPress Robots.txt File for Better SEO