Crawler

CombupBot

CombupBot is the crawler behind Combup, a fashion search service that helps shoppers compare products across stores. It reads public product information so people can find an item and then buy it on the store's own website.

How it identifies itself

Mozilla/5.0 (compatible; CombupBot/0.1; +https://combup.com/bot)

On some stores we read pages with a real browser, which can complete automated browser checks such as Cloudflare's. We never sign in or read anything behind a login.

What it reads

How fast it crawls

We keep the load on each site low: requests are rate-limited per site, and we slow down automatically when a site responds slowly or returns errors. A Crawl-delay written for CombupBot is honored.

How to limit or block it

CombupBot obeys robots.txt rules written for CombupBot by name (Robots Exclusion Protocol, RFC 9309). Rules addressed to all crawlers (*) are not applied. To block it completely:

User-agent: CombupBot
Disallow: /

To ask it to slow down:

User-agent: CombupBot
Crawl-delay: 10

How products appear on Combup

Each product links to the store's own page, where the purchase happens. Combup shows the store's name as text and doesn't use store logos.

Remove your store from Combup

To have your products removed from Combup, email [email protected] from an address at your store's domain. We apply the request within 48 hours, and CombupBot stops visiting your site. Blocking CombupBot in robots.txt also works; the change is picked up the next time it reads your robots.txt, at most one day later.

Contact

Questions or corrections: [email protected].