How to Find Every Page on a Competitor Website

You've looked at a competitor's menu.

You've clicked through their blog.

You think you've seen the site.

You haven't.

The pages that explain how a competitor wins often sit deep in the site.

A calculator in a resources folder.

Location pages linked only from the footer.

A guide from four years ago that still pulls visits.

In this tutorial I'll show you how I build a full page list for any competitor, with Python snippets you can run yourself.

It's the first stage of every competitor analysis I do.

What "All Pages" Actually Means

There is no single list of every page on someone else's site.

There are four partial lists, and each misses something:

The method below combines the first three.

The fourth is a sanity check.

Step 1: Find the Sitemap in robots.txt

Open https://competitor.com/robots.txt in your browser.

Look for a line that starts with Sitemap:.

Google's sitemap documentation asks site owners to add exactly that line, so most sites do.

No line? Try the usual locations:

• /sitemap.xml
• /sitemap_index.xml
• /wp-sitemap.xml on WordPress sites without an SEO plugin

Shopify stores normally serve theirs at /sitemap.xml too.

Write down every sitemap URL you find.

Some sites list several in robots.txt: one for pages, one for products, one per language.

Step 2: Expand Every Child Sitemap

Small sites have one sitemap file. Large ones don't.

Google limits a single sitemap to 50,000 URLs or 50MB uncompressed.

So big sites publish a sitemap index: a file that lists other sitemap files, which can be compressed as .gz.

You need to follow every branch.

Here's a short Python function that does it:

import gzip
import requests
import xml.etree.ElementTree as ET

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
HEADERS = {"User-Agent": "sitemap-research-script (contact: you@example.com)"}

def fetch_xml(url):
    r = requests.get(url, headers=HEADERS, timeout=30)
    r.raise_for_status()
    data = r.content
    if data[:2] == b"\x1f\x8b":  # gzip magic bytes
        data = gzip.decompress(data)
    return ET.fromstring(data)

def collect_urls(sitemap_url, seen=None):
    seen = set() if seen is None else seen
    if sitemap_url in seen:
        return []
    seen.add(sitemap_url)
    root = fetch_xml(sitemap_url)
    if root.tag.endswith("sitemapindex"):
        urls = []
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            urls += collect_urls(loc.text.strip(), seen)
        return urls
    return [loc.text.strip() for loc in root.findall("sm:url/sm:loc", NS)]

pages = collect_urls("https://competitor.com/sitemap_index.xml")
print(len(pages), "URLs")

It checks the first two bytes rather than the file extension, because some servers decompress .gz files on the way out and some don't.

Save the raw list with today's date.

You'll want it when you repeat the research in six months.

Step 3: Clean and Deduplicate the URLs

Raw sitemap lists are messy.

The same page can appear several ways:

• With and without www
• With and without a trailing slash
• With tracking parameters attached
• Once per language folder, such as /en-gb/ and /en-us/

If you don't clean them, one page becomes three rows.

Worse, nothing will match when you join the list with ranking data later.

Here's the cleaning function I use as a starting point:

from urllib.parse import urlsplit

LOCALES = {"en", "en-gb", "en-us", "fr", "de", "es", "it", "nl"}

def clean_url(url):
    parts = urlsplit(url.strip())
    host = parts.netloc.lower().removeprefix("www.")
    segments = [s for s in parts.path.split("/") if s]
    if segments and segments[0].lower() in LOCALES:
        segments = segments[1:]
    return host + "/" + "/".join(segments).lower()

clean = sorted({clean_url(u) for u in pages})
print(len(clean), "unique pages")

It drops the scheme, query strings and fragments, and strips one leading locale folder.

Add the locale codes your competitor actually uses.

Note: Lowercasing the path can merge two URLs that differ only by case. For matching lists against each other that's what you want. For anything else, keep the original URL in a second column.

If the competitor runs separate country sites, decide whether you want one market or all of them before you strip anything.

My guide to hreflang explains how those language versions are meant to connect.

Step 4: Add the Pages That Rank but Aren't Listed

Sitemaps only list what the owner remembers to include.

Old guides, retired campaign pages and pages from a previous CMS often drop out of the sitemap while still ranking.

Those are worth finding, because they're earning visits with nobody looking after them.

Export the competitor's top pages from a rankings tool.

I pull mine through the Ahrefs API; a CSV export from Ahrefs or Semrush works just as well for a one-off.

Run every exported URL through the same clean_url() function. Then compare:

listed = set(clean)
ranking = {clean_url(u) for u in ranking_urls}  # from your rankings export

missing_from_sitemap = sorted(ranking - listed)
never_ranking = sorted(listed - ranking)

missing_from_sitemap is your list of forgotten pages. (Run the same comparison on your own site, too. Ranking pages missing from your sitemap are a classic item on any technical SEO checklist.) Open a few.

You'll often find the competitor's best-performing content from years ago.

Step 5: Crawl the Site for Linked Pages

A crawler starts at the homepage and follows every internal link it finds.

It catches pages that are linked but not listed in the sitemap: tag archives, filter pages, odd landing pages.

Desktop crawlers such as Screaming Frog do this well.

The free version has a crawl limit, which is fine for small competitors.

Crawl politely:

• Set a slow crawl speed. You're a guest on their server.
• Respect robots.txt.
• Use an honest user agent.
• Never try to get behind a login.

Once the crawl finishes, export the URLs that returned a 200 status and clean them like the others.

If the crawl finds thousands of filter or parameter URLs, that tells you something too.

Sites that let crawlers wander through every filter combination can waste a lot of crawling on low-value pages.

I explain why in what is crawl budget.

Step 6: Group the Pages by Folder

You now have one clean list.

Next, give it a shape.

Count pages by their first folder:

from collections import Counter

folders = Counter(u.split("/")[1] or "(home)" for u in clean)
for folder, count in folders.most_common(20):
    print(f"{count:>6}  /{folder}/")

The output reads like a table of contents for their strategy:

From here, the list is ready for real analysis: which sections rank, which content types win, where they're thin.

That's the job of SEO competitor analysis, and this page list is its foundation.

The same list is also the raw material for finding topics you haven't covered. My guide to content gap analysis picks up from here.

Spot-Check With a site: Search

Finally, sanity-check your list against Google.

Search site:competitor.com/blog/ and compare a handful of results with your list.

If Google shows pages you don't have, a source is missing something.

Don't use site: to count pages, though.

Google's site: operator documentation says the list it returns is not always exhaustive and that bigger sites shouldn't expect to see all their URLs.

Build Your First Competitor Page List

Pick one competitor and run Steps 1 to 3 today.

On most sites, that takes minutes and shows you far more than an afternoon of clicking.

Then add rankings and a crawl when you want the full picture.

If you'd rather have this done across your whole market, my competitor analysis automation runs the full process for each rival and joins it with ranking data.

You get one sheet per competitor with every page they have.

And if you're still deciding which competitors to include, that's where the automated competitor research starts: from who actually ranks, not who you assume does.

You can also compare your own site's structure with theirs as part of a technical SEO review.