Crawler
BidTrollerBot
BidTrollerBot reads public bid and procurement listings so that contractors can find work they are qualified to bid. If you run a site it visits, here is what to expect.
How to recognize it
Every request carries this user agent:
BidTrollerBot/1.0 (+https://bidtroller.com/bot)How it behaves
- Respects robots.txt. It reads and follows each site's robots.txt before crawling.
- Rate-limited. It limits requests per site and backs off when a site is slow or returns errors.
- Public pages only. It does not log in to paid services, and it does not try to get around paywalls, CAPTCHAs, or other access controls.
- Identifies itself. It never disguises itself as a browser or another crawler.
How to limit or block it
Add a rule for BidTrollerBot to your robots.txt. For example, to block it entirely:
User-agent: BidTrollerBot
Disallow: /Changes take effect the next time it checks your robots.txt.
Contact
Questions or problems: info@bidtroller.com.