Coming soon
Find a website's sitemap
Find Sitemap is a planned feature. It will look up where a website publishes its sitemap, the file in which a site lists its own pages. It is not available in the API yet.
What it will do
- What you will send
- The address of the website.
- What you will get back
- Where the sitemap the website publishes is, when it has one. What else the result will include will be documented when the feature ships.
Where it will help
- Seeing a site's own list of pages before you plan requests against it.
- Checking whether a site publishes a sitemap at all.
- Starting a batch of requests from the URLs a site lists itself.
Why it will help
- Sitemaps are not always at the same address. Finding one by hand means checking robots.txt and the usual locations, site by site.
- Find Sitemap will do that lookup for you.
Other planned features
Planned features. None of them is available in the API yet.
Crawl
Coming soonPlanned: fetch the pages of a website from one starting URL, without a list of page URLs first.
Read the plan, CrawlExtract DESIGN.md
Coming soonPlanned: analyze a reference website and produce a DESIGN.md design brief your coding agents can follow.
Read the plan, Extract DESIGN.md
Hear when Find Sitemap ships
Released features are announced in the changelog. The features on the features page work today.