Search Intent Classification at Scale: Labelling Thousands of Keywords

Labelling twenty keywords by intent is easy.

Labelling twenty thousand is a different job.

You can't read that many results pages, and you can't afford to pull them all.

I classify intent as one stage of my own keyword research pipeline. This tutorial shows the method I use, with Python snippets for the parts that are easiest to get wrong.

You'll finish with a label set tied to page types, a free rules pass, a results pass for the terms that matter and a review queue for the rest.

Before You Start

Search intent classification at scale needs four things:

• A cleaned keyword list with search volume or some other value score.
• A source of live search results for your country and device. Several providers sell results pages through an API, priced per request.
• Python or any language you can script in.
• A spreadsheet for the review queue.

If the idea of intent is new to you, read what is search intent first. This guide assumes you know the four classic types and how to read a results page by eye.

Results pulls cost money, so the method below is built to spend them only where they change a decision.

Step 1: Define Labels That Map to Page Types

Start with the labels, because everything else depends on them.

The textbook four (informational, commercial, transactional, navigational) are a fine start. They are too coarse to route keywords to pages.

I use labels named after the page that should rank:

A label you can't act on is wasted work.

Each one here points to a type of page, so the label becomes the routing decision.

Adapt the set to your business. A shop needs product and category split apart. A local trade can probably fold product into service, and if you are doing this by hand for a small firm, my guide to keyword research for a service business is the lighter version.

Step 2: Clean and Deduplicate the List

Messy input makes every later step more expensive.

Before classifying anything:

• Lowercase every keyword and collapse repeated spaces.
• Merge exact duplicates from different sources, keeping the highest volume.
• Remove terms for services, products or areas the business doesn't sell.
• Tag brand terms (yours and competitors') so they can skip the paid pass.

That third point saves the most money.

Every irrelevant term you remove now is a results pull you never pay for.

If your list came from rival rankings, my guide on how to find competitor keywords covers how to strip that noise first.

Step 3: Run a Modifier Rules Pass on Every Keyword

Now give every keyword a free first guess from its wording.

Certain words lean strongly one way. Here is a starting set:

import re

RULES = [
    ("local",   r"\bnear me\b|\bnearby\b|\bin (leeds|york|bradford)\b"),
    ("compare", r"\bbest\b|\bvs\b|\bversus\b|\breviews?\b|\balternatives?\b|\bcompare\b"),
    ("service", r"\bquote\b|\bhire\b|\bbook\b|\bcompany\b|\bservices?\b|\bemergency\b"),
    ("product", r"\bbuy\b|\bprice of\b|\bcheap\b|\bfor sale\b"),
    ("guide",   r"^(how|what|why|when|can|does|is|should)\b|\bguide\b|\bideas\b|\bexamples\b"),
]

def rule_label(keyword):
    for label, pattern in RULES:
        if re.search(pattern, keyword):
            return label, f"rule: {label}"
    return None, "no modifier"

Order matters.

The first match wins, so put the most decisive patterns first.

Replace the place names with the areas you actually serve.

This pass will label a good share of your long-tail keywords in seconds.

It will also be wrong sometimes. "Best way to fix a dripping tap" matches compare, but the results are guides.

That's fine. A rule label is a hypothesis, not a verdict.

Step 4: Pull Live Results for the Keywords That Matter

You can't afford a results pull for every keyword. You don't need one.

Pull results for:

• Every keyword above your value threshold (volume, or volume times a commercial weight).
• Every keyword the rules pass left unlabelled, if it has meaningful volume.
• Every keyword that might become the main term for a page.

For each pull, set the country, language and device.

For local businesses, set the location too, because results change from town to town.

Save the top ten organic results (URL and title) plus the features on the page: map pack, shopping results, People Also Ask, video.

Cache every response to disk, keyed on the keyword and settings.

You will rerun this stage when rules change, and you should never pay twice for the same page.

The same caching approach runs through my walkthrough on how to automate keyword research.

Step 5: Label Each Ranking Page by Type

The results page is a list of URLs. You need it as a list of page types.

Most pages give themselves away through their URL and title:

PAGE_TYPES = [
    ("forum",     r"reddit\.com|quora\.com|/forum|/community/|/threads?/"),
    ("video",     r"youtube\.com|vimeo\.com"),
    ("directory", r"yell\.com|checkatrade\.com|yelp\.|/directory/"),
    ("guide",     r"/blog/|/guides?/|/advice/|/learn/|\bhow to\b|\bwhat is\b|\bguide\b"),
    ("compare",   r"\bbest\b|\btop \d+\b|\bvs\b|\breview"),
    ("product",   r"/products?/|/shop/|/collections?/|/p/"),
    ("service",   r"/services?/|\bquote\b|\bbook\b|\bnear you\b"),
]

def page_type(url, title):
    text = f"{url} {title}".lower()
    for kind, pattern in PAGE_TYPES:
        if re.search(pattern, text):
            return kind
    return "unknown"

Build the directory list from the sites that actually appear in your market. The domains above are only examples.

Expect a pile of unknown pages on the first run.

Read twenty of them, add patterns, run again.

Note: Homepages need their own rule. A results page full of company homepages usually means service or brand intent, not a guide.

Step 6: Score Each Results Page and Assign a Label

Now turn ten page types into one label with a confidence score.

Map page types to labels, count them, and let features override where they are decisive:

from collections import Counter

TYPE_TO_LABEL = {
    "guide": "guide", "forum": "guide", "video": "guide",
    "compare": "compare", "directory": "service",
    "service": "service", "product": "product",
}

def serp_label(results, features):
    counts = Counter(TYPE_TO_LABEL.get(page_type(r["url"], r["title"]), "unknown") for r in results)
    known = sum(n for label, n in counts.items() if label != "unknown")
    if "map_pack" in features and counts["service"] >= 3:
        return "local", 0.8, "map pack with provider results"
    if not known:
        return "mixed", 0.0, "no recognisable pages"
    ranked = [(label, n) for label, n in counts.most_common() if label != "unknown"]
    label, top = ranked[0]
    share = top / known
    if share < 0.6:
        return "mixed", round(share, 2), f"top type only {top} of {known}"
    return label, round(share, 2), f"{top} of {known} pages are {label}"

The 0.6 threshold is my starting point, not a rule.

Raise it if too many labels look wrong in review. Lower it if the mixed pile grows unmanageable.

Shopping results deserve a similar override towards product.

People Also Ask boxes on their own prove little, because they appear across many intents.

Store three columns for every keyword: label, confidence and reason. Later, when a label turns into a page, those columns travel into the content brief so the writer knows what the results reward.

The reason column is what makes the output auditable. "7 of 9 pages are guide" is something anyone can check.

Step 7: Send Disagreements and Low Confidence to Review

You now have up to two labels per keyword: one from the rules, one from the results.

Combine them like this:

The review queue should be short.

If it isn't, your thresholds or page-type patterns need work.

For each queued term, you (or whoever knows the business) open the results page and pick a label.

Mixed results are not always a problem. Often they mean one page should serve the main intent and cover the secondary one lower down.

Labels then feed straight into grouping. Keywords with the same label and overlapping results usually belong on one page, which is the core of the SERP overlap method in keyword clustering.

Step 8: Audit a Sample and Tune the Rules

Never trust an automated label set you haven't spot-checked yourself.

After each full run:

1. Take a random sample from each label, around 30 to 50 keywords.
2. Search each one by hand and note whether you agree.
3. Look for patterns in the misses, such as a directory domain missing from your list or a modifier rule firing too broadly.
4. Fix the rule, rerun from cache, and check again.

Most of your gains will come from the page-type patterns, not the scoring maths.

Once a label set holds up, check your own pages against it. If Google already ranks your blog post for a term labelled service, the Search Console Performance report will show it, and that page probably needs a sibling.

Labels also expose clashes. Two of your pages targeting terms with one label and one set of results are likely to compete, which my guide on avoiding keyword cannibalisation explains how to fix.

Where Language Models Fit

A language model is useful in one narrow spot: labelling ranking pages your patterns can't place.

Give it the URL and title, a fixed list of page types, and ask for exactly one of them.

Then validate the answer against the list and drop anything else.

Don't ask a model to label intent from the keyword alone. It can't see the results page, so it falls back on wording, which your rules pass already covers for free.

Check a sample of its page-type answers by hand, exactly as in Step 8.

Label Your First 500 Keywords This Week

You don't need the whole pipeline on day one.

Take your 500 most valuable keywords. Run the rules pass, pull results for the top 100, and score them.

Compare the two labels. Where they disagree, you'll see exactly why wording alone isn't enough.

Then add the review queue and the sample audit before you scale up.

If you'd rather have intent labels produced as part of a full keyword pipeline, my keyword research automation runs this stage between the relevance gate and routing, with my review before anything reaches you.

For the strategic side, deciding which searches the business should win, see my keyword research consulting. Both rely on intent labels you can audit, not guesses.