Building a custom SERP pipeline

Manual SERP checks do not scale. This note covers the pipeline I use for controlled SEO experiments.

Why a custom pipeline

Third-party SERP APIs are useful, but for research I need:

• stable request fingerprints;
• reproducible snapshots;
• diffable structured exports.

Architecture

The flow is intentionally boring:

1. Queue a keyword set with locale and device profile.
2. Fetch results through a controlled worker.
3. Normalize titles, URLs, and positions.
4. Store JSON snapshots and compute diffs.

serp fetch --keywords keywords.txt --locale en-US --out ./snapshots
serp diff ./snapshots/2026-08-01.json ./snapshots/2026-08-13.json

What worked

• Treating SERP data as time-series, not one-off screenshots.
• Keeping parsers versioned next to the snapshots.

What did not

• Overfitting parsers to a single HTML shape without fallback selectors.

This is demo content for the lab site. The full tool notes live under Tools.