How to Automate Content Briefs From the Pages That Rank

Writing a content brief by hand means opening a stack of tabs, skimming each page and noting the headings.

It works for one keyword. By the tenth, you are skimming faster and missing more.

The fix is to automate content briefs at the evidence stage. A script gathers the pages that rank, keeps all their text and hands you one clean document to work from.

I built my own version in Python, and it runs behind my brief automation work. Here is how it works, step by step, so you can build or adapt your own.

What Gets Automated (and What Does Not)

Before you write a line of code, decide where the script stops and you start.

The script does the collecting. You do the deciding.

That split keeps briefs consistent at volume without turning them into a summary of what competitors already say.

Step 1: Collect the Ranking URLs

Start with the keyword and the pages ranking for it.

You can get the URLs from a SERP data API, from your keyword tool's export, or by searching and copying them.

At volume, an API is the only sensible option. If you also want each competitor's wider site, my guide to finding every page on a competitor website covers that.

Then filter the list. Remove results that are not comparable pages:

• Directories and marketplaces on a service query
• Forum threads and video results on a guide query
• Your own pages, which you will compare separately

Save the cleaned list with the brief.

Anyone checking your reasoning later can see exactly which pages it was built from.

Note: Pull results for the country your page targets. A UK service page briefed from US results will copy the wrong market.

Step 2: Fetch Each Page

Next, download each URL's HTML.

A plain HTTP request works for most server-rendered pages.

Some sites build their content with JavaScript in the browser, and a plain request returns a near-empty shell. For those, use a headless browser such as Playwright, which loads the page as a visitor would.

My approach is to try the plain request first and fall back to the browser when the response looks too thin.

Fetch politely:

• Space out your requests rather than hitting a site all at once
• Identify your script with a sensible user agent
• Respect robots.txt rules
• Log every failure so you can deal with it by hand

Expect some pages to fail. Bot protection, cookie walls and broken pages happen.

A flagged failure is fine. A silent gap in the evidence is not.

Step 3: Strip Only the Site Chrome

This is the step that makes or breaks automated briefs.

Most content extraction libraries are built to find the main article on a news or blog page.

They throw away anything that looks like layout.

On page-builder sites, much of the useful content looks like layout:

• Accordion FAQs, where the answers sit in collapsed panels
• Tabbed panels, where only the first tab shows by default
• Hero and banner sections, which often carry the clearest statement of what a business offers

On many page builders, collapsed and tabbed content is still in the HTML, just hidden from view.

An extraction library may discard it anyway.

So use a blunter rule. Remove only the parts that are clearly site furniture, the navigation menus and the footer, and keep everything in between.

You will read a little more noise.

You will also stop losing the sections that explain why a competitor ranks.

Pro tip: If a page loads accordion answers only when clicked, the text will not be in the HTML. The browser fallback from Step 2 can click them open, or you can flag the page for a manual check.

Step 4: Convert the Page to Markdown

Now turn the cleaned HTML into markdown.

A library such as markdownify does most of the work in Python. Check that it preserves:

• Heading levels, so you can see each page's H2 and H3 structure
• Lists and tables, as structure rather than run-on text
• Link anchor text, which shows what each competitor links to and how

Markdown is the right format for this job.

It opens in any editor, pastes cleanly into documents, and is compact enough to hand to a language model if you use one later.

Strip leftover clutter at this point too: cookie notices, empty headings and repeated calls to action.

Step 5: Assemble the Competitor Document

Put every converted page into one file, in ranking order, each under a heading with its URL.

Add a short summary at the top:

• The keyword and the date the results were collected
• Each page's URL and approximate word count
• The H2s that appear on several of the pages

This competitor document is the evidence base.

It is raw text, not a summary, so you or your writer can check any claim in the brief against the source.

It also tends to outlive the brief.

When the page needs a refresh later, rerun the collection and compare the two documents to see what changed. That comparison is a ready-made task list for on-page SEO updates.

Step 6: Draft the Brief From the Evidence

With the evidence in one place, draft the brief itself.

Use a fixed template so every brief has the same sections. My checklist of what to include in an SEO content brief covers the full set.

Some sections can be pre-filled from the document:

• Common sections, from the headings that recur across pages
• Questions to answer, from FAQ blocks and question-style headings
• A length range, from the word counts
• A first-pass outline, built from the common sections in a sensible order

You can generate these with simple rules, with a language model prompted on the competitor document, or by hand.

Whatever you use, never let the brief tell a writer to reword competitor copy.

Google's guidance is blunt on this: attribution or giving credit to other sources doesn't replace the need for original effort. The evidence shows what to match.

The brief has to say how to do better.

Step 7: Review Every Brief by Hand

This is where the script stops and you take over.

Read the draft brief against what you know about the business. I check:

• Fit. Cut sections that do not suit the services, area or customers.
• Angle. Decide what the page will say that the ranking pages do not.
• Proof. Add the examples, process detail and first-hand points only the business can make.
• Claims. List anything the writer must not promise.
• Boundaries. Name the existing pages this one must not compete with, and link to them instead. My guide to avoiding keyword cannibalisation explains why this matters.

The script handles the collecting, so you can spend your time on these five things.

That is the real gain from automation: more time on judgement, not less judgement.

If the draft will be written with AI help, read my take on whether AI can write SEO content before you hand the brief over.

Automate the Collection, Keep the Judgement

Automated content briefs work when the script does the slow, repetitive evidence gathering and a person makes every decision that depends on the business.

Your first action: take the next keyword you need to brief and run Steps 1 to 5 by hand once. You will see exactly which parts are worth scripting, and which hidden sections your usual skim has been missing.

Briefs work best when the keyword decision is already made upstream, which is what my guide on automating keyword research covers. Topics usually come out of a content gap analysis first.

If you would rather not build it yourself, my content brief automation runs this pipeline for you and delivers the competitor document plus a reviewed brief for each keyword. It sits alongside the rest of my SEO automation services, including drafting support with a human editor on every page.