From 30.1M archived pages to 265 startup problem categories

Historical recovery and evaluation

The research question required a longitudinal view of what problems startups said they were solving. Domains change owners, companies disappear, and present-day copy leaks future information into earlier years. The data layer had to recover each company website at the right historical snapshot before any model touched the text.

End-to-end architecture

The pipeline separates recovery, measurement, evaluation, representation learning, and analysis. Each layer writes a durable artifact with stable identifiers for the next stage.

Why Athena instead of running a Spark cluster?

The Common Crawl URL index already lives next to the data in object storage. The expensive operation was a selective scan and reduction, not an always-on distributed compute workload. Athena let the pipeline push domain filters into managed SQL, materialize a small pointer table, and pay only for scans. Spark would have added cluster provisioning, shuffle tuning, and idle capacity without eliminating the need to read the same index.

After pointer export, worker processes request only the byte interval that contains one WARC record. Results go directly to S3; EC2 disk is scratch. A URL cap prevents one company with a huge site from dominating the run, and checkpoint rows make a 120-snapshot job restartable.

def fetch_https(pointer: Pointer, timeout: float) -> bytes:
    end = pointer.offset + pointer.length - 1
    request = urllib.request.Request(
        f"https://data.commoncrawl.org/{pointer.warc_filename}",
        headers={
            "Range": f"bytes={pointer.offset}-{end}",
            "User-Agent": "problem-taxonomy-commoncrawl-fetcher/0.1",
        },
    )
    with urllib.request.urlopen(request, timeout=timeout) as response:
        return response.read()

Structured extraction and external validation

Website prose is noisy: it mixes the customer, the problem, the solution, marketing claims, and investor language. The extractor therefore returns fixed fields under a strict schema instead of free-form summaries. Inference runs at temperature 0, output parsing is deterministic, and the same record identifiers survive through caches and downstream flat files.

Before scaling the company-segment router, I ran a balanced 2,000-company audit against the licensed source taxonomy. Overall agreement was 85.45%, but the confusion matrix was asymmetric: text-derived B2B assignments often disagreed with source-side B2C labels when the described buyer was an institution or operator. I used these cases to revise the routing rules.

External validation uses 331 structured founder interviews. Founders describe their company’s problem before reviewing the extracted components, which avoids anchoring their first response on the model’s framing.

Human gold labels, DSPy, and a validation guard

For the customer-need classifiers, six coders iterated through six codebook rounds. Majority votes form the gold labels. Mean agreement is 91.5%, with weighted Fleiss κ = 0.57. Each construct uses separate train, validation, and test partitions; the test split is never available to optimization.

GEPA, implemented through DSPy, treats the classifier prompt as a fitted parameter. A stronger reasoning model reads production-model errors and proposes instruction revisions. The written codebook remains the seed prompt, so the search stays inside a human-defined construct. I tested instruction-only versus demonstrations, 0–16 few-shot examples, direct prediction versus chain-of-thought, optimizer budgets, and smaller production models.

A validation guard chooses among the codebook, optimized instructions, and optimized instructions plus demonstrations. More optimizer budget was not monotonically better, prompts did not transfer cleanly between model sizes, and one validation-selected program still landed slightly below baseline on the untouched test split.

Representation comparison

NMF and LDA provide familiar topic-word summaries, but their bag-of-words representation is sensitive to vocabulary and does not naturally express that one firm can address several problems. K-means is efficient and useful for local splits, yet it forces every firm into one centroid and produces unrelated partitions when the requested number of clusters changes.

The system trains top-K sparse autoencoders over 3,072-dimensional text embeddings at M ∈ {128, 256, 512, 1024}. M = 256 balances distinctness and interpretability; seed stability is checked by matching learned features with the Hungarian algorithm. A scripted review pass records splits, merges, reassignments, and exclusions, yielding 265 active clusters over 88,336 firms.

Failures and sensitivity checks

• Current-site crawls leaked later information into earlier years. The final pipeline preserves snapshot identity until the longitudinal panel is assembled.
• Whole-WARC downloads transferred unnecessary data. Record-level byte ranges reduced transfer and made retries addressable.
• More optimization did not reliably improve held-out F1. Larger search budgets stayed in a narrow band, and optimized prompts were model-specific.
• Resolution sensitivity. We recomputed the analysis on a raw 512-feature partition.
• Coder disagreement exposed ambiguous construct boundaries. Six codebook rounds revised boundaries before prompt optimization.

My scope

I led the computational research pipeline and manuscript integration: historical web recovery, orchestration, structured LLM extraction, annotation tools, codebooks, gold-set construction, DSPy/GEPA evaluation, analysis artifacts, and reproducibility. The broader research—including interviews, construct development, and sparse-representation work—is collaborative.