Engineering a six-condition LLM experiment without breaking the experiment

The platform had to preserve causal validity.

An experiment platform must keep shared elements fixed while changing one mechanism by condition. Conditions 3, 5, and 6 use the same three preparation modules—what each side wants, where they agree and disagree, and how a deal could work—but vary whether the AI generates a sheet, extends a participant draft, or only pressure-tests reasoning.

System architecture

Grounding required semantic contracts.

A model can return a real quote that is still invalid for the intended use. A private note from one side cannot prove shared agreement; an unverified media report cannot establish a party’s red line; a passage about an intermediary does not authorize that actor to implement a deal.

The evidence handler therefore couples a JSON schema with deterministic checks. Each item has a verbatim quote, framework, slot, one or two visible cell IDs, importance, reason, and semantic-group label. A second layer validates source context and prohibited routes. Exact text matching confirms that the quoted span really exists in the document.

function sourceContextAllowsHighlight(framework, slot, sourceContext) {
  if (framework === "ioa" && slot.endsWith("-agree") &&
      ["role-private-document", "unverified-external-report",
       "third-party-media", "third-party-advocacy"]
        .includes(sourceContext)) return false;

  if (framework === "paths" && slot === "boundaries" &&
      !["unspecified", "role-private-document",
        "two-party-meeting-record"].includes(sourceContext)) return false;

  return true;
}

Cross-document deduplication also operates at the proposition level, not the keyword level. “Food distribution” can describe a constraint, a package term, or background context; those are not interchangeable evidence.

Model failure is part of the treatment environment.

Different conditions call different handlers and models, but failure behavior must not silently become another experimental manipulation. Structured-output requests use schema validation and bounded retries. Evidence ranking falls back to deterministic core/support ordering. Final scoring has a local rubric-based fallback so a participant still receives a result if the model provider is unavailable.

Log the path to each answer.

The platform records file opens, document clicks, saved evidence, AI requests, draft state, prompt edits, condition-specific artifacts, and timing. A monotonic interaction revision prevents a slower autosave from overwriting a newer snapshot. Each cohort also carries the case-material version and the deployed commit-derived system version, so analysis can separate participants who saw different code or content.

The dashboard supports launch and validity checks. Started versus completed counts, timeout rates, missing stages, fallback-scored rows, condition summaries, and interaction sequences surface broken data before a batch becomes unusable.

Failure modes that changed the design

• Per-file evidence labels looked globally meaningful. They were replaced with a separate cross-document clustering pass and migration logic for already-saved quotes.
• Topic overlap created false “similar evidence.” The clustering contract now requires the same decision-relevant proposition, actor, authority, and implication.
• Asynchronous snapshots could arrive out of order. Interaction saves now carry increasing revisions and reject stale state.
• Version labels were manually stale. The API derives the runtime version from the deployed commit and stores it with participant rows.
• Scoring coupled to completion created a fragile exit path. The final survey commits first; scoring saves later through a separate stage.

My scope

I implemented the experimental condition logic, document reader, evidence layer, LLM handlers, schemas, SSE path, Postgres persistence, behavioral instrumentation, dashboard, model benchmarks, tests, and deployment. Research materials shown in the public system are predetermined scenarios; participant records remain in the protected study database and are never embedded here.