# www.robotstxt.org/ # Allow crawling of all content User-agent: * Crawl-delay: 1 Disallow: /add/ Disallow: /admin/ Disallow: /reviews/add/ Disallow: /company/edit/ User-agent: GPTBot Crawl-delay: 1 Disallow: /add/ Disallow: /admin/ Disallow: /reviews/add/ Disallow: /company/edit/ User-agent: SemrushBot Disallow: / User-agent: DotBot Disallow: /