Overview
A limited crawler for development.
This alternate version of rrainn Spider is used exclusively to test new functionality. A site may see a small number of development requests during that work.
Key differences
Do not identify it by reverse DNS.
Development crawler IP addresses do not have the reverse-DNS records used by the production crawler. Contact [email protected] if you need to verify whether a development request is authentic.
Development requests may originate from a developer’s personal internet connection. Blocking the address could affect unrelated, legitimate visitors who later receive the same address.
The development User-Agent starts with rrainnSpiderDevelopment instead of rrainnSpider. Because this crawler tests unfinished behavior, small differences not documented here may occur.
Shared behavior
The same site controls still apply.
The development crawler uses the same robots.txt user agent as production: rrainnSpider. It also follows the production crawler’s link-handling and indexing controls.
Read the production crawler documentation for complete behavior and configuration details.
Blocking guidance
Choose the scope you actually need.
To block only development traffic, contact [email protected] so we can help identify and stop the relevant requests without affecting ordinary visitors.
To block both production and development crawling, use the controls described in the production access-control section.
Support
Need help with a request?
Send the source IP, timestamp, and user agent so we can investigate efficiently.