A sitemap is a file or page that lists the accessible URLs on a website. It comes in two main forms: the XML sitemap, intended for search engine crawlers, and the HTML sitemap, designed for human visitors. Using this document as a starting point makes it possible to quickly map all published content without relying solely on conventional menu navigation.
Orphan pages: what a sitemap reveals about internal linking
A complete sitemap lists every URL a site owner wants to make visible. But this list does not guarantee that every page is actually accessible through the site’s navigation.
Comparing the URLs in the sitemap with those reachable by following internal links from the homepage highlights orphan pages. These pages appear in the sitemap, but no internal links lead to them through a natural browsing path.
An audit guide published by Screaming Frog recommends comparing the results of a crawl starting from the homepage with the sitemap content, then checking Analytics and Search Console data. URLs that appear only in the sitemap need better internal linking; otherwise, they remain difficult for both visitors and crawlers to find.
The Wiki Dark sitemap is a good example of this comprehensive mapping approach: every section is listed, making it easier to spot areas that may be poorly connected to the rest of the site structure.
Crawling from the homepage or reading the sitemap: two complementary approaches
Starting a crawl from the homepage simulates the journey of a visitor or crawler following links one by one. This method reveals the site’s actual depth and navigation paths.
Reading the sitemap, on the other hand, provides the complete list of URLs declared by the site owner. These two approaches do not produce the same results.
URLs found only by crawling but missing from the sitemap indicate an omission in the sitemap. URLs listed in the sitemap but not found by the crawl point to an internal linking issue. Comparing both datasets is the most reliable way to create a realistic inventory of the site’s content.
Additional sources to cross-check
Screaming Frog’s audit guide recommends adding other sources to this comparison:
- Pages listed in Google Analytics, which show which URLs actually receive traffic, including those that the crawl or sitemap may have missed
- Google Search Console data, which lists indexed URLs and those with coverage errors
- The CMS itself, whose database may contain draft or archived pages that do not appear anywhere else
- Server logs (when available), which record actual crawler requests and reveal the URLs requested by Googlebot or Bingbot
None of these sources provides a complete picture on its own. Systematically cross-checking them helps identify inconsistencies before they affect search rankings.
XML sitemap and HTML sitemap: choose the right format for your goal
An XML sitemap is a technical file structured with tags. It contains URLs and may include metadata such as the last modified date or estimated update frequency. Search engines read it automatically during their regular visits.
An HTML sitemap is a standard web page that people can read. It presents links to different sections as a list or clickable hierarchy. Its primary purpose is to help visitors quickly find a specific page without using the menus.
The two formats serve different needs. A site with only an XML sitemap makes things easier for crawlers but does not help its visitors. A site with only an HTML sitemap deprives search engines of a structured file they can process more quickly.
Maintaining both formats covers automated crawling and human navigation. On common CMS platforms such as WordPress, plugins generate the XML sitemap automatically. The HTML sitemap often needs to be created manually or semi-automatically to keep it readable and relevant.
Reporting updates: the role of the sitemap compared with IndexNow for Bing
The sitemap and the IndexNow protocol serve different purposes, even though both relate to communicating with search engines.
The sitemap acts as a reference list of existing URLs. It informs search engines about all available pages. IndexNow, by contrast, is used to report new, updated, or deleted content in real time. Bing Webmaster Tools clearly distinguishes between these two mechanisms.
For a site whose content changes frequently, combining the two approaches speeds up the processing of changes. The sitemap ensures that the overall inventory stays up to date. IndexNow notifies search engines of individual changes without waiting for the crawler’s next visit.
When this distinction matters
For a static site updated a few times a year, the sitemap alone is more than sufficient. For an editorial or e-commerce site publishing content daily, IndexNow reduces the time between publication and indexing. The sitemap remains the foundation, while active notifications complement the setup for urgent content.
A complete sitemap remains the foundation of any methodical crawl, whether for a technical audit, an internal linking overhaul, or simply to help visitors find a page buried in the site hierarchy. The sitemap’s real limitation is not its format, but how diligently it is kept up to date with the content actually published.
