How to Automate Keyword Research Without Losing Judgement

Manual keyword research has a ceiling.

You can only read so many rows before everything starts to look the same.

Automation removes the ceiling.

It also removes the brakes.

Run a keyword tool through a script with no rules, and you get thousands of rows sorted by volume.

Plenty of them are for services the business doesn't sell.

The trick is automating the right stages and keeping a person at the end.

This tutorial walks through the eight-stage pipeline I run for keyword research, with Python snippets for the parts that are easiest to get wrong.

What to Automate and What to Keep Human

Here's how I split the work:

Notice the pattern.

Machines handle volume and consistency. People handle context.

If you want the manual version first, start with my guide on keyword research for a service business.

Automation only makes sense once you know what good output looks like.

What You'll Need

You don't need a big stack to start. You need:

• Python, or any language you're comfortable scripting in.
• A keyword data API. I use the Ahrefs API. The Google Ads API's keyword planning service can also generate ideas from seed keywords, URLs or sites, with historical volume.
• A source of live search results for the intent check.
• A spreadsheet for the review at the end.

Paid APIs charge by usage, so estimate the cost of a run before you start one.

You'll be surprised how fast a big competitor's keyword list adds up.

Step 1: Turn the Business Into a Profile File

Every later stage checks keywords against what the business actually does.

So write that down in a form a script can read.

I start from a client questionnaire and turn the answers into a profile like this:

{
  "services": ["roof repair", "flat roofing", "gutter cleaning"],
  "areas": ["leeds", "wakefield", "harrogate"],
  "never": {
    "services": ["commercial roofing", "solar panels"],
    "areas": ["london", "manchester"],
    "words": ["diy", "cheapest"]
  }
}

The never block matters most.

These are the client's hard stops: services they don't offer, areas they won't travel to, words they can't or won't use.

Update the file whenever the business changes.

The pipeline is only as current as this profile.

Step 2: Agree the Page Inventory First

This is the step most keyword research skips.

It's also why so much research ends up in a folder.

Before collecting a single keyword, list the pages the site has or will have:

• Service pages
• Location pages
• Category pages
• Blog topic areas

Mark planned pages separately from live ones.

Then get the list approved.

Why first?

Because every keyword that survives the pipeline needs somewhere to go.

With an agreed inventory, a term that fits nowhere isn't lost.

It becomes a clear question: is this worth a new page?

Step 3: Mine Competitor Keywords

Competitors are often the richest single source of real queries.

Their rankings show what people actually type, not what you guess they type.

Pull the organic keywords for each competitor through your keyword API.

Keep the raw list, untouched, in its own file.

The quality of this step depends on picking the right competitors.

Use the sites that rank for your money terms, not just the businesses you think of as rivals.

I cover how to choose them in how to do SEO competitor analysis.

Step 4: Expand Your Seeds

Competitor terms show what already ranks.

Seeds fill the gaps competitors haven't covered.

Build seeds from two places: the services and areas in your profile, and the pages in your inventory.

"Flat roofing" plus "leeds" gives you a seed.

So does each blog topic area.

Then send the seeds to your keyword API for variations, questions and modifiers.

Expect your list to balloon. That's fine.

The next stages exist to shrink it.

Step 5: Cache Every API Call

Here's a stage that pays for itself on the first rerun.

You will rerun the pipeline.

A new never-rule, a changed page list, an extra competitor.

Without a cache, every rerun pays for the same data again.

Wrap every API call in a cache keyed on the request:

import hashlib
import json
from pathlib import Path

CACHE = Path("cache")
CACHE.mkdir(exist_ok=True)

def cached(endpoint, params, fetch):
    raw = json.dumps([endpoint, params], sort_keys=True)
    path = CACHE / (hashlib.sha256(raw.encode()).hexdigest() + ".json")
    if path.exists():
        return json.loads(path.read_text(encoding="utf-8"))
    data = fetch(endpoint, params)
    path.write_text(json.dumps(data), encoding="utf-8")
    return data

fetch is whatever function calls your API.

Same endpoint and same parameters means the answer comes from disk.

Delete the cache folder when you want fresh data, for example before a yearly refresh.

Step 6: Run the Relevance Gate

This is where automation earns its keep.

Before anyone looks at volume or difficulty, every keyword is checked against the profile.

Off-topic terms go, however big their numbers.

Here's a simple version of the gate:

import re

def gate(keyword, profile):
    kw = keyword.lower()
    for group, terms in profile["never"].items():
        for term in terms:
            if re.search(rf"\b{re.escape(term)}\b", kw):
                return "drop", f"never-rule ({group}): {term}"
    if any(re.search(rf"\b{re.escape(s)}\b", kw) for s in profile["services"]):
        return "keep", ""
    return "review", "no offered service named"

Three outcomes, not two.

Terms that break a rule are dropped with a reason.

Terms that name a service are kept.

Everything else goes to a review pile, because questions like "how long does a roof last" can still be useful blog terms.

Log every drop with its reason in a separate tab.

When someone asks why a term isn't in the plan, you can show them.

Step 7: Check Intent Against Live Results

Wording alone can't tell you intent. The search results can.

For each surviving keyword, fetch the current top results and label what type of page each one is: a service page, a guide, a directory, a product listing. Then count:

• Mostly providers and directories? It's a money page term.
• Mostly guides and explainers? It belongs in the blog plan.
• A mix? Flag it for a person to judge.

Note local packs and shopping results too.

They change what kind of page can compete at all.

If the idea of intent is new to you, read what is search intent before building this step.

It is the step that stops service pages turning into articles.

Step 8: Route Keywords and Review by Hand

Now every keyword gets one destination.

Money page terms go to the page in your inventory that matches their service and area.

Blog terms go to a topic in the blog plan.

Terms that fit nowhere are listed as candidate new pages.

One keyword, one page.

If two pages want the same term, pick one and log the other.

My guide to avoiding keyword cannibalisation explains why that rule matters.

To group close variants under one page, I use the methods in keyword clustering methods.

Then comes the part no script replaces.

Open the routed plan and read it.

You'll catch terms that passed every rule and are still wrong.

A phrase that sounds right but attracts the wrong customers.

A service the client mentioned once and doesn't want to push.

Fix those, then send the plan for approval.

Nothing goes to a client before I've reviewed it.

Mistakes That Break Automated Keyword Research

I've seen these sink keyword pipelines, including early versions of my own:

• Starting from the tool, not the business. The output fills with terms for services nobody sells.
• Filtering by volume first. Big off-topic terms survive; small, valuable ones vanish.
• No log of removals. Nobody can challenge a cut they can't see.
• No page inventory. The research produces a list, and the site stays the same.
• Shipping without review. Rules catch most problems. Not all of them.

Automate the Volume, Keep the Judgement

Start small.

Write the profile file and the relevance gate first, then run one competitor's keywords through them.

You'll see how much noise disappears before any human reads a row.

Add caching and the SERP check next, then routing.

If you'd rather not build and maintain it yourself, my keyword research automation runs this pipeline for you, with my review before anything reaches your desk.

The approved plan then feeds straight into briefs through content brief automation.

And if you want help deciding which searches to win in the first place, that strategic side is my keyword research service, with this automated pipeline doing the heavy lifting underneath.