TheSpreadBot

Your public RSS or Atom feed. It never requests article pages or images. Besides your feed it fetches only your robots.txt (usually at most once a day, more often while it's failing) and, for WordPress feeds, the feed's second page.

From each feed item it keeps:

• the headline
• a snippet of up to 200 characters, which is all we ever display. For AI summaries we also keep up to 1,000 characters of the feed's description internally; it's never shown on the site.
• the link to the article, with utm_* tracking parameters removed
• the image URL listed in the feed, if there is one (the image itself is not downloaded)
• the publish time

Readers who want the story click through to your site.

We use Anthropic's Claude to write short summaries of stories that two or more newsrooms reported, from the headlines and descriptions above. Full articles are never sent.

Blocking Anthropic's crawlers keeps your outlet's text out of them, by design. If your robots.txt disallows ClaudeBot or anthropic-ai (or Claude-User, Claude-SearchBot, or every bot with User-agent: *), your headlines and descriptions are never sent to Claude and summaries don't name your outlet. Your stories still appear on the site as usual. We re-read robots.txt daily, and a change also clears any summary already built from your text.

Text from outlets that opt out of AI is never sent to Claude, so it's never summarized or used for titles. All headlines and short descriptions, including theirs, pass through OpenAI's embedding API to group stories; OpenAI doesn't train on it. We're moving that step to a model we run ourselves after the election.

Or email hello@thespread.ca and we'll exclude you by hand.

• Only for what's changed. Each request sends back the ETag or Last-Modified your server gave us last time (as If-None-Match / If-Modified-Since), so an unchanged feed costs you a 304 with no body.
• Feeds are requested one at a time, never in parallel. Per site, each run (every 15 minutes) asks for your feed, with the occasional robots.txt check and, when the newest page of a WordPress feed no longer reaches back to our previous visit, its second page (?paged=2), so stories aren't missed.
• Each feed has a minimum gap between fetches: at least 10 seconds, and longer if your robots.txt sets a Crawl-delay or you ask us.
• We honour Retry-After on a 429 or 503. Other errors back off: from the second failure in a row we wait 15 minutes, doubling each time, up to 6 hours.
• If a request times out it is retried once, then left until the next run.

We follow robots.txt: if it disallows TheSpreadBot (or all crawlers), we don't fetch, and we never crawl faster than its Crawl-delay.

If a site refuses us (a 401 or 403, or a Cloudflare challenge), we back off and show the outlet as “feed blocked”. We never disguise the crawler or look for a way round a block.

Email hello@thespread.ca and we will stop crawling your feed and remove your items within 24 hours.