When you install WP it comes with an autogenerated robots.txt file which is rather incomplete. From experience I find this is the most appropriate robots.txt file (at a minimum), for a WP site installed right in the root folder: User-agent: * Disallow: /feed Disallow: /*/feed Disallow: /xmlrpc Disallow: /wp- Disallow: /?p= Disallow: /*trackback Allow: /wp-admin/admin-ajax.php … Continue reading The robots.txt file for a WordPress Site and Other Considerations