# www.robotstxt.org/ # Allow crawling of all content User-agent: * Disallow: /includes/ Disallow: /websites/ Disallow: /owners/ Disallow: /shops/