SEO Automation
SEO Competitor Analysis Automation That Maps Every Page Your Rivals Have
I pull each competitor's XML sitemaps to see which pages exist, join them with Ahrefs ranking data to see which pages rank, and roll the result up into a research sheet: site structure, informational content by type, and domains that feature your rivals but not you.
- years in SEO
- 6+
- client websites worked on
- 60+
- person doing the work
- 1
What SEO competitor analysis automation gives you
SEO competitor analysis automation, as I run it, produces a page-by-page map of each rival's site that shows what they have published, which of those pages rank, and how the whole site is organised.
Manual competitor reviews tend to stop at the homepage, the main menu and whatever ranks for a handful of head terms. That misses most of the evidence. The pages that explain how a competitor wins are often deep in the site - a calculator in a resources folder, a set of location pages linked only from the footer, a buying guide that pulls visits for hundreds of long-tail queries. Nobody finds those by clicking around.
So I let code read the whole site. Each competitor's XML sitemaps list the pages they want indexed. The Ahrefs API returns the pages it sees ranking and the queries behind them. My pipeline joins the two on cleaned-up URLs and produces one row per page: does it exist, does it rank, for what, and what kind of page is it.
This is part of my SEO automation services, and like everything else there, the output is checked by me before it reaches you. The pipeline is fast; the reading is where the value is.
Key takeaways
- A sitemap tells you what a competitor has built; ranking data tells you what is working. The useful view comes from joining the two.
- URLs are normalised and stripped of locale folders before the join, so the same page is not counted several times under different spellings.
- Structure matters as much as content: collections, menu depth, product counts and the words competitors put in their URLs all show how they organise demand.
- You receive raw data per competitor that you can filter and check, not a summary deck with the evidence removed.
- API costs are estimated before large pulls, so the size of the research is a decision rather than a surprise.
What's included in my SEO competitor analysis automation
Six parts, each producing its own tab or set of columns in the research sheet.
Sitemap harvesting that lists every page each competitor publishes
The pipeline reads the sitemap index and every child sitemap for each competitor, collecting the full list of URLs they submit for indexing. Split sitemaps are followed through the index, and the raw list is kept so you can see exactly what was collected and when.
- Sitemap index and child sitemaps
- Full URL list per competitor
- Collection date recorded
- Raw list kept for checking
Ahrefs ranking data joined to sitemap URLs on normalised addresses
Organic pages and their keywords come from the Ahrefs API. Before joining, both lists are cleaned so that differences in how the same address is written - trailing slashes, query strings, locale folders - do not split one page into several rows. The result marks every URL as listed only, ranking only, or both.
- Organic pages and keywords per competitor
- URL normalisation and locale stripping
- Listed-only, ranking-only and matched flags
- API cost estimated before large pulls
Site structure roll-up: collections, menu levels, product counts and URL words
Rows are grouped into the sections that make up each competitor's site. For shops that means collections and the number of products in each; for service businesses, service and location sections. Menu depth shows how far from the homepage each page type sits, and recurring URL words show the vocabulary competitors organise around.
- Sections and collections per competitor
- Product counts per collection
- Menu level for each page type
- Most frequent URL keywords
Informational URLs typed as tools, lead magnets, guides and blog posts
Every informational page is given a type, so you can compare formats instead of simply counting articles. A calculator, a downloadable checklist and a long buying guide do very different jobs; lumping them together as content hides what is working. Each typed page keeps its ranking data alongside it.
- Tool, lead magnet, guide, blog and other types
- Ranking data kept per typed page
- Format counts by competitor
- Examples worth opening flagged
Competitor backlink gap checks for listicles, roundups and directories
The pipeline looks for referring domains that link to several competitors but not to you. These are often best-of listicles, comparison posts and directories - sites that already write about your category and might include you. I check the shortlist by hand before it goes anywhere.
- Domains linking to several competitors
- Your own referring domains excluded
- Page type noted, such as listicle or directory
- Manual check before outreach
A research sheet per competitor that you can filter and reuse
Everything lands in one workbook: a tab per competitor with the URL map, a structure summary, the typed content list and the backlink gap rows. Columns stay consistent across competitors, so you can compare them side by side, and the sheet is ready to feed keyword or content planning.
- One tab per competitor
- Consistent columns across sites
- Structure and content summaries
- My notes on the patterns worth acting on
Experience
SEO Competitor Analysis Automation from someone who has done it many times
You work with me from the first call to the last report. No account managers, no hand-offs to a junior team.
Book a free call- years in SEO
- 6+
- client websites worked on
- 60+
- documented case studies
- 35
What the URL map records for every competitor page
One row per URL, built from two sources that rarely agree on their own.
| Field | Where it comes from | What it tells you |
|---|---|---|
| Page exists | Competitor XML sitemaps | What the competitor has chosen to publish and submit for indexing |
| Page ranks | Ahrefs API organic data | Which of those pages earn visibility, and which sit unseen |
| Top queries and estimated traffic | Ahrefs API | The demand each page actually captures, not the demand its title implies |
| Ranks but missing from the sitemap | The join itself | Forgotten pages that still pull visits - often older content worth studying |
| Section and menu level | URL path and site navigation | How deep the competitor buries each type of page |
| Page type | URL patterns and titles | Whether the page is a product, collection, service, tool, lead magnet, guide or blog post |
Traffic figures from any third-party tool are estimates. I use them to compare pages with each other, not as a forecast of what you would earn.
Why the sitemap and ranking join beats either source alone
Each source has blind spots. Put together, they cover most of each other's.
Sitemaps alone overstate
A sitemap lists everything a competitor hopes will rank, including thin pages, old campaigns and tag archives. Treating it as a map of what works sends you chasing pages that earn nothing.
Rankings alone understate
Ranking tools only show pages they have seen ranking. New pages, pages ranking for queries outside the tool's database and pages that convert well but rank modestly can disappear from view.
URL spelling breaks naive joins
The same page can turn up with and without a trailing slash, with tracking parameters, or under a country folder. I normalise URLs and strip locale before joining, so one page stays one row.
The mismatch is the finding
Pages that rank without being in the sitemap, and sections in the sitemap with nothing ranking, are often the most interesting rows in the sheet. They show where a competitor's plan and its results have drifted apart.
Free review
Get a free SEO Competitor Analysis Automation review
Send me your site. Before the call I look at your rankings and pages, then tell you what I would fix first and what it is likely worth.
How a competitor analysis run works
From a list of domains to a research sheet you can sort.
- 1
Agree the competitor set
You and I decide which domains count - the ones you lose deals to, the ones that outrank you, or both. Search competitors are often not the businesses you think of as rivals, so I suggest additions from the results pages.
- 2
Estimate the data cost
Before pulling ranking data for large sites, I estimate the API cost of the run and agree it with you. Big catalogue sites can have very large sitemaps, so this step decides how deep the pull goes.
- 3
Harvest sitemaps
The pipeline reads each competitor's sitemap index and child sitemaps, collecting every listed URL with the date it was gathered.
- 4
Pull ranking data and join
Organic pages and keywords come from the Ahrefs API. Both lists are normalised, locale-stripped and joined into the URL map.
- 5
Roll up and type
Pages are grouped into sections, collections and menu levels, product counts are tallied, URL keywords extracted, and informational URLs typed as tools, lead magnets, guides, blog posts and so on.
- 6
Review and hand over
I check samples against the live sites, correct misclassified pages and add notes where a pattern deserves your attention. Then the sheet is yours.
A competitor review deck versus raw competitor data
Both have a place. Only one lets you check the conclusion.
Summary deck
- Twenty slides of screenshots and takeaways
- Conclusions you have to take on trust
- Out of date the moment the competitor publishes
- Hard to reuse for keyword research or content planning
- Covers the pages the analyst happened to open
Raw data per competitor
- A sheet per competitor with one row per URL
- Every conclusion traceable to the rows behind it
- Rerunnable on fresh data when the market moves
- Feeds straight into keyword and content gap work
- Covers every URL the sitemaps and ranking data expose
Reading competitor site structure, not just their content
The roll-up is where the data starts to explain strategy. For each competitor I count the collections or service sections, how many menu levels sit between the homepage and a product or service page, how many products each collection holds, and which words recur in their URLs. A rival with deep collections and short product lists is organising for a different kind of search than one with a flat catalogue.
Informational content gets its own pass. Every non-commercial URL is typed - interactive tool or calculator, lead magnet, guide, blog post, glossary entry, comparison - and set against what it ranks for. That shows which formats are pulling their weight in your market. If the competitors who outrank you all run a cost calculator and you do not, that is a decision worth making on purpose.
None of this tells you to copy a competitor. It tells you what the market is rewarding, so that your own keyword research and site structure start from evidence rather than a hunch.
Backlink gaps
Domains that feature your competitors but not you
Alongside the page map, the pipeline checks which referring domains link to several competitors but not to you - listicles of the best providers, roundups, industry directories. A domain that already mentions three of your rivals writes about your category. Those rows go to link prospecting automation or straight onto an outreach list, after I have confirmed the pages are real and relevant.
Where competitor data goes next
A competitor map is rarely the end product. Most often it feeds the keyword research automation pipeline, which mines competitor rankings as one of its stages, and the content gap automation pipeline, which uses competitor titles to propose topics you have not covered.
It also shapes decisions outside automation. If the map shows competitors winning with depth you lack, that becomes a content strategy question. If it shows their structure is cleaner than yours, it belongs in an SEO audit or a technical SEO review. The data stays the same; who acts on it changes.
If you just want the research and will act on it in-house, that works too. The sheet is designed to be used without me in the room.
Growth from real projects
Learn more about SEO Competitor Analysis Automation
Often needed alongside this
SEO Competitor Analysis Automation FAQs
What does SEO competitor analysis automation include that a manual review does not?
Coverage. A manual review usually looks at the menu, a few top pages and some head terms. The automated version reads every URL in each competitor's sitemaps, joins them with ranking data and types every informational page. You still get my reading of the patterns, but it rests on the whole site rather than a sample.
Which competitors should I include?
Usually a mix: the businesses you lose work to and the sites that outrank you for your main services. They are often different. I suggest search competitors from the results pages and agree the final list with you before any data is pulled, because each extra domain adds API cost.
Why do you need both sitemaps and ranking data?
Each misses things on its own. Sitemaps list pages a competitor wants indexed, including many that never rank. Ranking data only shows pages a tool has seen ranking. Joining them reveals forgotten pages that still get visits and whole sections that earn nothing, which are often the most useful findings.
How are data costs handled?
Ranking data comes from the Ahrefs API, which charges by usage. Before a large pull I estimate what the run will cost, based on the number of competitors and the size of their sites, and agree it with you. If the estimate is high, the competitor list or the sections pulled can be narrowed.
Can the analysis be repeated later?
Yes. The pipeline runs the same way each time, so a second run months later shows which pages competitors added, which started ranking and which disappeared. Because the method does not change, the differences reflect the competitors rather than a different person doing the research differently.
Do I get a presentation or the data?
The data, with notes. You receive a sheet per competitor with one row per URL and the summaries built on top of it. If you want a walkthrough call, I take you through the patterns I found, but the conclusions always point back to rows you can check yourself.
Start SEO competitor analysis automation on your market
Send me your domain and the three to five competitors you worry about most. I will check their sitemaps, estimate the data cost of a full run and tell you what the map is likely to show before you commit.