Building an evaluated taxonomy from 30.1M archived web pages

Scale and measurement validity

The research question required a longitudinal view of what problems startups said they were solving. Domains change owners, companies disappear, and present-day copy leaks future information into earlier years. The data layer had to recover each company website at the right historical snapshot before any model touched the text.

End-to-end architecture

The pipeline separates recovery, measurement, evaluation, representation learning, and analysis. Each layer writes a durable artifact with stable identifiers for the next stage.

Why Athena instead of running a Spark cluster?

The Common Crawl URL index already lives next to the data in object storage. The expensive operation was a selective scan and reduction, not an always-on distributed compute workload. Athena let the pipeline push domain filters into managed SQL, materialize a small pointer table, and pay only for scans. Spark would have added cluster provisioning, shuffle tuning, and idle capacity without eliminating the need to read the same index.

After pointer export, worker processes request only the byte interval that contains one WARC record. Results go directly to S3; EC2 disk is scratch. A URL cap prevents one company with a huge site from dominating the run, and checkpoint rows make a 120-snapshot job restartable.

def fetch_https(pointer: Pointer, timeout: float) -> bytes:
    end = pointer.offset + pointer.length - 1
    request = urllib.request.Request(
        f"https://data.commoncrawl.org/{pointer.warc_filename}",
        headers={
            "Range": f"bytes={pointer.offset}-{end}",
            "User-Agent": "problem-taxonomy-commoncrawl-fetcher/0.1",
        },
    )
    with urllib.request.urlopen(request, timeout=timeout) as response:
        return response.read()

Treat LLM extraction as a measurement instrument.

Website prose is noisy: it mixes the customer, the problem, the solution, marketing claims, and investor language. The extractor therefore returns fixed fields under a strict schema instead of free-form summaries. Inference runs at temperature 0, output parsing is deterministic, and the same record identifiers survive through caches and downstream flat files.

Before scaling the company-segment router, I ran a balanced 2,000-company audit against the licensed source taxonomy. Overall agreement was 85.45%, but the confusion matrix was asymmetric: text-derived B2B assignments often disagreed with source-side B2C labels when the described buyer was an institution or operator. That result became a routing diagnostic, not a claim that either label source was automatic ground truth.

The most important validation is external. In an ongoing program of 331 structured founder interviews, founders first describe their company’s problem in their own words and review the extracted components afterward. The order avoids anchoring them on the model’s framing. Only aggregate validation statistics are reported publicly; transcripts and responses are not exposed.

Human gold labels, DSPy, and a validation guard

For the customer-need classifiers, six coders iterated through six codebook rounds. Majority votes form the gold labels. Mean agreement is 91.5%, with weighted Fleiss κ = 0.57. Each construct uses separate train, validation, and test partitions; the test split is never available to optimization.

GEPA, implemented through DSPy, treats the classifier prompt as a fitted parameter. A stronger reasoning model reads production-model errors and proposes instruction revisions. The written codebook remains the seed prompt, so the search stays inside a human-defined construct. I tested instruction-only versus demonstrations, 0–16 few-shot examples, direct prediction versus chain-of-thought, optimizer budgets, and smaller production models.

A validation guard chooses among the codebook, optimized instructions, and optimized instructions plus demonstrations. More optimizer budget was not monotonically better, prompts did not transfer cleanly between model sizes, and one validation-selected program still landed slightly below baseline on the untouched test split.

Why sparse autoencoders instead of one hard clustering?

NMF and LDA provide familiar topic-word summaries, but their bag-of-words representation is sensitive to vocabulary and does not naturally express that one firm can address several problems. K-means is efficient and useful for local splits, yet it forces every firm into one centroid and produces unrelated partitions when the requested number of clusters changes.

The system trains top-K sparse autoencoders over 3,072-dimensional text embeddings at M ∈ {128, 256, 512, 1024}. M = 256 balances distinctness and interpretability; seed stability is checked by matching learned features with the Hungarian algorithm. A scripted review pass records splits, merges, reassignments, and exclusions, yielding 265 active clusters over 88,336 firms.

What did not work cleanly

• Current-web crawling created temporal errors. Historical reconstruction had to preserve snapshot identity rather than collapsing immediately to a year-level row.
• Fetching full WARC objects was the wrong unit. Pointer-first byte ranges reduced transfer and made retries addressable at the record level.
• Prompt optimization was not monotonic. Larger search budgets stayed in a narrow mean-F1 band, and optimized prompts were model-specific.
• One cluster resolution was not enough evidence. The full analysis was recomputed on a raw 512-feature partition to test sensitivity to both resolution and review decisions.
• LLM labels without coder disagreement hid construct problems. Codebook rounds and IRR exposed boundaries that no amount of prompt tweaking could define on its own.

My scope

I led the computational research pipeline and manuscript integration: historical web recovery, orchestration, structured LLM extraction, annotation tools, codebooks, gold-set construction, DSPy/GEPA evaluation, analysis artifacts, and reproducibility. The broader research—including interviews, construct development, and sparse-representation work—is collaborative.

Disclosure: licensed company data, raw web captures, and interview materials remain private. This case study uses aggregate metrics and implementation details only.