weird sitemap crawl results after urlscanner update
Installation and Configuration
soaringeagle:
hard to say but i remember last i used it (its been a long time) it listed a few thousand pages while the inspyder listed over a million
But which one is accurate? If you actually DO have a million pages on your site then that's fine, but there's no reason to list them all in your sitemap. The Sitemap file is used by crawlers to identify the
starting point for a crawl. There is no need to list ALL PAGES in a sitemap file, since Google will automatically follow any URLs that are found on a page.
So for example - in your site map file you only need the URL to a profile. You do not need to list every single page in the profile. Google/Bing/etc will easily find all those sub pages without being "told" they are there.
updated by @brian: 01/27/17 08:50:25AM