A six-condition platform for human-AI negotiation research

Condition implementation

The platform implements distinct human-AI work configurations. Conditions 3, 5, and 6 use the same three preparation modules—what each side wants, where they agree and disagree, and how a deal could work—but differ in task structure, participant role, and AI role.

System architecture

Evidence validation by source and analytic role

A model can return a real quote that is still invalid for the intended use. A private note from one side cannot prove shared agreement; an unverified media report cannot establish a party’s red line; a passage about an intermediary does not authorize that actor to implement a deal.

The evidence handler therefore couples a JSON schema with deterministic checks. Each item has a verbatim quote, framework, slot, one or two visible cell IDs, importance, reason, and semantic-group label. A second layer validates source context and prohibited routes. Exact text matching confirms that the quoted span really exists in the document.

function sourceContextAllowsHighlight(framework, slot, sourceContext) {
  if (framework === "ioa" && slot.endsWith("-agree") &&
      ["role-private-document", "unverified-external-report",
       "third-party-media", "third-party-advocacy"]
        .includes(sourceContext)) return false;

  if (framework === "paths" && slot === "boundaries" &&
      !["unspecified", "role-private-document",
        "two-party-meeting-record"].includes(sourceContext)) return false;

  return true;
}

Cross-document deduplication also operates at the proposition level, not the keyword level. “Food distribution” can describe a constraint, a package term, or background context; those are not interchangeable evidence.

Matched fallback behavior

Different configurations call different handlers and models, so failure behavior can otherwise become an unintended difference between conditions. Structured-output requests use schema validation and bounded retries. Evidence ranking falls back to deterministic core/support ordering. Final scoring has a local rubric-based fallback if the model provider is unavailable.

Behavioral instrumentation and versioning

The platform records file opens, document clicks, saved evidence, AI requests, draft state, prompt edits, condition-specific artifacts, and timing. A monotonic interaction revision prevents a slower autosave from overwriting a newer snapshot. Each cohort also carries the case-material version and the deployed commit-derived system version, so analysis can separate participants who saw different code or content.

The dashboard supports launch and validity checks. Started versus completed counts, timeout rates, missing stages, fallback-scored rows, condition summaries, and interaction sequences surface broken data before a batch becomes unusable.

Failure modes that changed the design

• Per-file evidence labels appeared to apply across documents. A separate cross-document clustering pass now groups saved quotes.
• Topic overlap created false “similar evidence.” The clustering contract now requires the same decision-relevant proposition, actor, authority, and implication.
• Asynchronous snapshots could arrive out of order. Interaction saves now carry increasing revisions and reject stale state.
• Version labels were manually stale. The API derives the runtime version from the deployed commit and stores it with participant rows.
• Scoring coupled to completion created a fragile exit path. The final survey commits first; scoring saves later through a separate stage.

My scope

I implemented the experimental condition logic, document reader, evidence layer, LLM handlers, schemas, SSE path, Postgres persistence, behavioral instrumentation, dashboard, model benchmarks, tests, and deployment. Research materials shown in the public system are predetermined scenarios; participant records remain in the protected study database and are never embedded here.