rrainn Spider

Overview

A crawler for rrainn projects.

rrainn Spider gathers public information for rrainn projects, including contextual search experiences and web search tools. You may see requests from it while it discovers and evaluates public pages on your site.

Identification

Verify both the agent and the network.

Requests include a User-Agent header similar to rrainnSpider/1.0.0 (+https://rrainn.com/spider). The version segment changes as the crawler evolves.

Every production crawler IP also has reverse DNS ending in spider.rrainn.com. Verify the reverse record first:

$ host 146.190.58.17
17.58.190.146.in-addr.arpa domain name pointer 146-190-58-17.spider.rrainn.com.

Then perform a forward lookup to confirm the hostname points back to the same IP address:

$ host 146-190-58-17.spider.rrainn.com
146-190-58-17.spider.rrainn.com has address 146.190.58.17

If either side of that check fails, the request was not made by the production rrainn Spider.

Content discovery

Make public content easier to understand.

rrainn Spider reads structured data in script[type=application/ld+json] elements. Using a Schema.org context helps the crawler understand the purpose and relationships of your content.

Publishing a sitemap and referencing it from robots.txt also gives the crawler a reliable inventory of public URLs.

Request frequency

Set a pace that works for your infrastructure.

rrainn Spider takes steps to avoid overloading servers. You can further limit its frequency with the Crawl-Delay directive in robots.txt. The crawler looks for the rrainnSpider user agent first, then falls back to *.

The crawler honors HTTP 429 responses. If a response does not include Retry-After, rrainn Spider waits 24 hours before trying the site again.

Access control

Control what the crawler can use.

rrainn Spider respects Allow and Disallow directives in robots.txt, except that it may retrieve the robots.txt file itself. Rules for rrainnSpider take priority over rules for *.

Links marked with rel=nofollow are not added to the crawl queue from that page. The same URL may still be discovered independently elsewhere.

You can also use meta[name=rrainnSpider] or meta[name=robots]. A rrainn-specific tag takes priority when both are present.

all

No indexing or link-following restrictions.

nofollow

Do not follow links found on the page.

noindex

Do not index or suggest the page in rrainn products.

none

Equivalent to nofollow, noindex.

Operations

What site operators should know.

• Report concerning or unexpected behavior to [email protected] so we can investigate it.
• robots.txt may be cached for up to 60 minutes. Contact support if an urgent rule change needs to be reloaded sooner.
• If robots.txt cannot be retrieved because of an error, timeout, or network failure, the crawler treats the site as unrestricted. Keep the file publicly reachable when it contains required rules.