Sitemaps, llms.txt & Search Console
The discovery files 3dCMS generates so search engines and AI crawlers can find and read every page.
Open in the interactive manual →The interactive version has full-text search, a table of contents, and light/dark themes.
Publishing builds more than pages. It builds everything crawlers need to find and read them.
| File | What it's for |
|---|---|
| sitemap.xml | Every page in the site, with sensible priority and change-frequency hints. |
| robots.txt | Allows the crawl, points to the sitemap, and keeps bots out of app-only URLs. |
| llms.txt | An llmstxt.org-style summary of the site and its pages, for AI assistants. |
| A plain-text page | A clean, low-noise /text/… version of every page for AI reading. |
| sitemap-index.xml | A domain-wide index of all your sites' sitemaps: the single URL to hand to Search Console. |
What "unlisted" does to a crawler
The Show on Explore & your profile toggle isn't only a gallery setting: it reaches the crawlable surface too. While a site is unlisted:
- Its pages serve
noindex, followinstead ofindex, follow, on both the live pages and the static bundle a crawler actually fetches. Every page is written twice, once as the 3D page and once as its plain-text/text/…twin, and both heads now answer from the same rule. That matters more than it sounds: llms.txt tells every AI crawler in plain prose where the text version lives, so the twin is the copy an agent is most likely to fetch. - Its non-HTML surfaces answer with an
X-Robots-Tag: noindexheader: the site's own sitemap.xml and robots.txt, plus llms.txt, scene.json, and the social-card image. Those are the files an AI crawler and a link unfurler reach for directly, and robots.txt was never going to cover them. The sitemap matters most of the set: it enumerates every page URL at top priority, and it is the first file a crawler reaches for. - Its robots.txt still allows crawling, deliberately. Blocking the fetch outright would stop a crawler ever reading the
noindexon the page, which is the standard way to make a URL linger in an index rather than leave it. - It's left out of sitemap-index.xml, so it's never submitted for indexing in the first place.
- Re-publishing it sends no IndexNow ping: pointing a crawler at a page you've just asked it not to index would be backwards.
Turning listing off tells the crawlers. At the moment a listed site becomes unlisted, 3dCMS sends one IndexNow notification for its pages. That's not an invitation: IndexNow is the protocol for changed and removed URLs, and the page a crawler now fetches says noindex, so the ping is what makes a page that's already in an index get dropped promptly instead of waiting months for an organic recrawl. It fires on that transition only, and it can never fail the toggle: if the notification doesn't get through, your site is unlisted regardless.
Flipping the toggle takes effect on its own, without a republish. It's an opt-out from discovery, not a lock: the URL still works for anyone you hand it to, and it isn't a substitute for taking a site down.
Getting indexed faster
On every publish, 3dCMS sends an IndexNow ping so search engines like Bing learn about new and changed pages right away. For Google, submit your sitemap-index.xml in Search Console once, and it'll keep finding your pages as you publish.
/sitemap-index.xml once: it lists every site you publish, so new sites are discovered automatically without another trip to Search Console.