Give it a list of domains. It maps each site without crawling a single page: what robots.txt allows for your user agent, every sitemap (including sitemap indexes and gzipped .xml.gz files), RSS/Atom/JSON feeds, and whether the site publishes llms.txt / llms-full.txt for AI agents. You can also get a clean, filtered URL list with lastmod, changefreq and priority.
It is built for teams that feed websites to LLMs, RAG pipelines, AI agents and SEO tools. You find out what to fetch, what has changed since your last run, and what you are allowed to fetch before you spend money on a crawler.
Use cases
- Crawler and RAG seeding: get the complete URL list from sitemaps before you crawl or embed.
- SEO audits: find missing or broken sitemaps, sitemap size and lastmod freshness.
- AI-bot policy research: see which sites block GPTBot, ClaudeBot or CCBot in robots.txt, and who publishes llms.txt.
- Content monitoring: pull only URLs with lastmod after a date to see what a site published this week.
- Feed discovery: find RSS/Atom/JSON feeds for news and blog monitoring.
What you get per domain
- robots.txt summary for the user agent you choose (for example
GPTBot,Googlebotor your own bot). It shows the matched group, the allow and disallow rules, crawl-delay, whether the whole site is blocked for you, and which well-known AI bots (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot…) are blocked. Rules are matched per RFC 9309: longest match wins, with*and$wildcards. - Sitemaps listed in robots.txt. When robots.txt lists none, the common paths are tried. Sitemap indexes are followed, newest first, and gzip is handled.
- Feeds from
<link rel="alternate">on the homepage, or from common paths such as/feed,/rss.xmland/atom.xml. Each feed shows its format, title, item count and latest item date. - llms.txt and llms-full.txt: whether each is present, its size, and an excerpt.
- URL counts: URLs scanned, URLs that matched your filters, URLs filtered out, URLs disallowed by robots.txt, and URLs output.
There is also an optional per-URL record: url, lastmod, changefreq, priority, sourceSitemap and allowedByRobots.
Use it from AI agents, the API or no-code tools
Add Apify's hosted MCP server with just this tool:
{
"mcpServers": {
"site-discovery": { "url": "https://mcp.apify.com?tools=steadyline/site-discovery" }
}
}Sign in with Apify (OAuth) on first connect, or send an Authorization: Bearer <APIFY_TOKEN> header. Then ask your agent, for example:
Use steadyline/site-discovery to list all blog URLs changed in the last 30 days on vercel.com and tell me whether the site blocks GPTBot.
See MCP setup for Claude, Cursor and VS Code or add all tools at once.
REST API
Run it and get results in one call:
curl -X POST "https://api.apify.com/v2/acts/steadyline~site-discovery/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"domains": ["apify.com", "vercel.com", "blog.cloudflare.com"], "maxUrls": 100, "userAgent": "GPTBot"}'No-code
Use the official Apify app in Zapier ("Run Actor"), Make ("Run an Actor") or n8n (Apify node), choose steadyline/site-discovery, then send the dataset items to Google Sheets, Slack, a CRM or a database.
Pricing
Pay per event on Apify, no subscription. Prices are read from the public Apify Store listing; the Store page is authoritative.
| Event | Price | Per 1,000 |
|---|---|---|
| Domain summary Charged per domain scanned (summary record). | $0.002 | $2.00 |
| URL discovered Charged per unique URL discovered. | $0.0003 | $0.30 |
| Actor start (Apify standard) Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event). | $0.00005 | – |
Limitations
- Only one origin is analyzed per input.
wordpress.org/newsanalyzeswordpress.org, so enter subdomains such asblog.example.comseparately. - Sites that block datacenter IPs may return 403 for sitemaps. Those sitemaps are listed with their error.
- When
lastmodSinceis set and a site has no lastmod values (common), every URL is filtered out. CheckurlCounts.filteredOut.
Works well with
- Website → RAG knowledge base: list URLs here, convert PDFs with PDF to Markdown and HTML pages with HTTP Request Runner (
outputMarkdown: true). About $0.61 for 500 URLs. - Map, profile, contact: this Actor → Tech Stack Detector → Website Contact Extractor.
- Related: PDF & Document to Markdown, Bulk HTTP Request Runner, Tech Stack Detector.
FAQ
Does it crawl the site?
No. It reads robots.txt, sitemaps (including indexes and .gz files), feeds and llms.txt only, so it is fast and light on the target site.
Can I check whether a site blocks AI crawlers?
Yes. Set userAgent (e.g. GPTBot, ClaudeBot, CCBot) and the summary shows the robots.txt rules that apply, and each URL gets allowedByRobots.
What does it cost?
$0.002 per domain summary plus $0.0003 per URL you export. Set outputUrls: false for summaries only.
Can I run it on a schedule?
Yes. Create an Apify Schedule (or a saved Task with a schedule) and connect a webhook or integration to receive results.
Need a feature or found a bug?
Open an issue on the Issues tab of the tool's Apify Store page or email [email protected]. Built and maintained by Steadyline Data.
Acceptable use
This Actor reads only public, machine-readable files that sites publish for robots (robots.txt, sitemaps, feeds, llms.txt), plus the homepage. It identifies itself with a descriptive user agent. Respecting robots.txt is on by default. Use the results in line with each site's terms and robots.txt, and don't use them to overload or attack sites. You are responsible for what you do with the discovered URLs.
Made by Steadyline Data.