User-agent: * Disallow: /cart Disallow: /account Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /*/collections/*filter*&*filter* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /search Allow: /search/ Disallow: /search/?* Sitemap: https://ampdlifestyle.com/sitemap.xml # Google adsbot ignores robots.txt unless specifically named! User-agent: adsbot-google Disallow: /cart Disallow: /account Disallow: /search Allow: /search/ Disallow: /search/?* User-agent: Nutch Disallow: / # AhrefsBot builds Ahrefs' public backlink index. A delay is reasonable # politeness for a third-party crawler, but 10 seconds a page was excessive for # a store of this size — it stretched a full crawl into minutes for no benefit. User-agent: AhrefsBot Crawl-delay: 2 Disallow: /cart Disallow: /account Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /*/collections/*filter*&*filter* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /search Allow: /search/ Disallow: /search/?* Sitemap: https://ampdlifestyle.com/sitemap.xml # AhrefsSiteAudit is OUR OWN tool — it only crawls sites whose ownership has been # verified in Ahrefs. It was inheriting both a 10-second delay and the full # disallow list, which meant our own SEO audits crawled at one page per ten # seconds and were blind to the very pages we would want audited. An audit that # cannot see a page cannot report a problem on it, so the tool was quietly # reporting a clean site by omission. # # No delay, and only the two paths nothing should ever crawl: /cart is # session-specific and /account is private. Everything else is fair game — that # is the point of running an audit. User-agent: AhrefsSiteAudit Disallow: /cart Disallow: /account Sitemap: https://ampdlifestyle.com/sitemap.xml User-agent: MJ12bot Crawl-Delay: 10 User-agent: Pinterest Crawl-delay: 1