Crawler
CombupBot
CombupBot is the crawler behind Combup, a fashion search service that helps shoppers compare products across stores. It reads public product information so people can find an item and then buy it on the store's own website.
How it identifies itself
Mozilla/5.0 (compatible; CombupBot/0.1; +https://combup.com/bot)
We don't imitate a web browser, and we don't try to get around bot protection, CAPTCHAs or access controls. If a site challenges or blocks our requests, we stop.
What it reads
- Your
robots.txt, before anything else, at most once a day. - Public product data your storefront already publishes: name, price, available sizes and colors, product images and the product page address.
- Nothing behind a login: no customer data, no reviews and no checkout pages.
How fast it crawls
At most about one request per second per site, usually far less. We honor Crawl-delay and slow down automatically when a site responds slowly or returns errors.
How to limit or block it
CombupBot follows the Robots Exclusion Protocol (RFC 9309). It obeys rules written for CombupBot, and otherwise the rules for *. To block it completely:
User-agent: CombupBot Disallow: /
To ask it to slow down:
User-agent: CombupBot Crawl-delay: 10
How products appear on Combup
Each product links to the store's own page, where the purchase happens. Combup shows the store's name as text and doesn't use store logos.
Remove your store from Combup
To have your products removed from Combup, email [email protected] from an address at your store's domain. We apply the request within 48 hours, and CombupBot stops visiting your site. Blocking CombupBot in robots.txt also works; the change is picked up the next time it reads your robots.txt, at most one day later.
Contact
Questions or corrections: [email protected].