Crawler
CombupBot
CombupBot is the crawler behind Combup, a fashion search service that helps shoppers compare products across stores. It reads public product information so people can find an item and then buy it on the store's own website.
How it identifies itself
Mozilla/5.0 (compatible; CombupBot/0.1; +https://combup.com/bot)
On some stores we read pages with a real browser, which can complete automated browser checks such as Cloudflare's. We never sign in or read anything behind a login.
What it reads
- Your
robots.txt, before anything else, at most once a day. - Public product data your storefront already publishes: name, price, available sizes and colors, product images and the product page address.
- Nothing behind a login: no customer data, no reviews and no checkout pages.
How fast it crawls
We keep the load on each site low: requests are rate-limited per site, and we slow down automatically when a site responds slowly or returns errors. A Crawl-delay written for CombupBot is honored.
How to limit or block it
CombupBot obeys robots.txt rules written for CombupBot by name (Robots Exclusion Protocol, RFC 9309). Rules addressed to all crawlers (*) are not applied. To block it completely:
User-agent: CombupBot Disallow: /
To ask it to slow down:
User-agent: CombupBot Crawl-delay: 10
How products appear on Combup
Each product links to the store's own page, where the purchase happens. Combup shows the store's name as text and doesn't use store logos.
Remove your store from Combup
To have your products removed from Combup, email [email protected] from an address at your store's domain. We apply the request within 48 hours, and CombupBot stops visiting your site. Blocking CombupBot in robots.txt also works; the change is picked up the next time it reads your robots.txt, at most one day later.
Contact
Questions or corrections: [email protected].