Multi-LLM Qualitative Analysis Methodology

Unified codebook produced by GPT-5.2 reconciliation of themes independently discovered by all three models. Each theme lists which models independently identified it.

Development 10 themes

Design & Planning 10 themes

Quality & Risk 9 themes

Infrastructure & Ops 10 themes

Meta-Work 9 themes

Unified constraint codebook produced by GPT-5.2 reconciliation. Captures what developers do not want AI to handle.

Development 10 themes

Design & Planning 10 themes

Quality & Risk 10 themes

Infrastructure & Ops 10 themes

Meta-Work 10 themes

You are a qualitative research coder. Your task is to systematically code
each "WANT" response using ONLY the themes from the provided codebook.

Each response is shown alongside the same respondent's answer to a related
question about what they do NOT want AI to handle, for additional context.

CODEBOOK:
{codebook themes listed here}

ISSUE CODES (assign when a response has data quality issues):
- ISSUE_WRONG_FIELD: The respondent appears to have answered the other question
- ISSUE_BACK_REFERENCE: Response references a prior answer and is unintelligible
  on its own
- ISSUE_NON_RESPONSE: Terse non-answer with no analyzable content
- You may create other ISSUE_* codes if you encounter a different type of data
  quality problem

INSTRUCTIONS:
1. Read each response carefully
2. For each response, write a brief rationale
3. Then assign ALL applicable theme codes from the codebook
4. A response can have 0, 1, or multiple themes
5. Only use codes from the codebook or ISSUE codes
6. If no themes apply, return an empty array

RESPONSES TO CODE:
[Batches of 20 responses, each with context from opposite question]

OUTPUT FORMAT:
[
  {"pid": 8, "rationale": "...", "themes": ["theme_code_1", "theme_code_2"]},
  {"pid": 11, "rationale": "...", "themes": ["ISSUE_BACK_REFERENCE"]}
]

Return ONLY the JSON array, no other text.

Identical structure to the opportunity track prompt, but with:

• Constraint codebook themes replacing opportunity themes
• "NOT WANT" responses as the primary coding target
• "WANT" responses shown as cross-response context
• Same ISSUE code system applies

You are analyzing open-ended survey responses from software developers about
where they want AI assistance in their work. Your task is to identify themes
in these responses.

Guidelines for theme creation:
- Themes should be SPECIFIC and ACTIONABLE
  (e.g., "Automated test generation for edge cases" not just "Testing")
- Themes should be PROBLEM-FOCUSED (describe the pain point, not a solution)
- A response can belong to MULTIPLE themes
- Aim for 4-15 themes that capture the major patterns

For each response, return:
{
  "pid": <participant ID>,
  "themes": ["theme_code_1", "theme_code_2", ...]
}

Also provide a theme codebook:
{
  "themes": [
    {
      "code": "snake_case_theme_code",
      "name": "Human-Readable Theme Name",
      "description": "What this capability means and why developers want it",
      "pids": [list of PIDs expressing this]
    }
  ],
  "codings": [
    {"pid": 123, "themes": ["theme_code_1", "theme_code_2"]}
  ]
}

RESPONSES TO ANALYZE:
[All responses for the category]

You are a qualitative research analyst performing theme reconciliation.

CONTEXT: Three independent AI models analyzed survey responses from the
"[category]" category. Each identified opportunity themes. Your job is to
reconcile these into a unified codebook.

--- GPT-5.2 THEMES ---
[All GPT themes with names, descriptions, PIDs]

--- GEMINI THEMES ---
[All Gemini themes]

--- OPUS THEMES ---
[All Opus themes]

TASK: Create a unified codebook by:
1. Identifying themes that overlap across models (same concept, different names)
2. Merging overlapping themes into single unified themes
3. Keeping single-model themes IF substantive (≥3 PIDs)
4. Dropping themes that are too vague or have very few supporting responses
5. Aim for 5-10 unified themes per category

For each unified theme, provide:
- code: snake_case identifier
- name: Human-readable name
- description: Clear description of the desired capability
- source_models: which models identified it (["gpt", "gemini", "opus"])
- source_codes: the original codes from each model

Return ONLY valid JSON, no other text.

Theme Discovery Output

{
  "model": "gpt-5.2",
  "category": "design_planning",
  "category_name": "Design & Planning",
  "response_count": 223,
  "timestamp": "ISO-8601",
  "themes": [
    {
      "code": "string",
      "name": "string",
      "description": "string",
      "pids": [integer]
    }
  ],
  "codings": [
    { "pid": integer, "themes": ["string"] }
  ]
}

Consolidated Codebook

{
  "metadata": {
    "phase": "Opportunity Theme Reconciliation",
    "timestamp": "ISO-8601",
    "reconciliation_model": "gpt-5.2",
    "discovery_models": ["gpt-5.2", "gemini-3.1-pro-preview", "claude-opus-4-6"]
  },
  "categories": {
    "category_key": {
      "category": "string",
      "category_name": "string",
      "theme_count": integer,
      "models_reconciled": ["gpt", "gemini", "opus"],
      "themes": [
        {
          "code": "string",
          "name": "string",
          "description": "string",
          "source_models": ["gpt", "gemini", "opus"],
          "source_codes": {
            "gpt": ["string"],
            "gemini": ["string"],
            "opus": ["string"]
          }
        }
      ]
    }
  }
}

Systematic Codings (Phase 4)

{
  "category": "string",
  "phase": "Phase 4 - Systematic Coding",
  "timestamp": "ISO-8601",
  "models": ["gpt-5.2", "gemini-3.1-pro-preview", "claude-opus-4-6"],
  "codebook": [ { "code": "string", "name": "string", "description": "string" } ],
  "response_count": integer,
  "codings": {
    "gpt": [ { "pid": integer, "rationale": "string", "themes": ["string"] } ],
    "gemini": [ ... ],
    "opus": [ ... ]
  },
  "cost": { ... }
}

IRR Results (Phase 5)

{
  "phase": "Phase 5 - Inter-Rater Reliability",
  "methodology": {
    "raters": ["gpt-5.2", "gemini-3.1-pro-preview", "claude-opus-4-6"],
    "metrics": ["Krippendorff's Alpha", "Cohen's Kappa (pairwise)", "Percent Agreement"]
  },
  "overall_statistics": {
    "mean_krippendorff_alpha": float,
    "mean_percent_agreement": float,
    "interpretation": "string"
  },
  "category_results": {
    "category_key": {
      "krippendorff_alpha": { "theme_code": float },
      "percent_agreement": { "theme_code": float },
      "pairwise_kappa": {
        "gpt_vs_gemini": { "theme_code": float },
        "gpt_vs_opus": { "theme_code": float },
        "gemini_vs_opus": { "theme_code": float }
      },
      "code_frequencies": { "gpt": {}, "gemini": {}, "opus": {} }
    }
  }
}

Prevalence Results (Phase 6)

{
  "methodology": {
    "consensus_method": "majority_vote",
    "threshold": "2+ of 3 models must agree"
  },
  "category_results": {
    "category_key": {
      "theme_prevalence": [
        {
          "code": "string",
          "count": integer,
          "percentage": float,
          "pids": [integer]
        }
      ],
      "consensus_codings": { "pid": ["theme1", "theme2"] }
    }
  }
}

Rich Opportunity Card

{
  "rank": integer,
  "theme_code": "string",
  "category": "string",
  "title": "string",
  "problem_statement": "string",
  "proposed_capability": {
    "summary": "string",
    "context_sources_needed": ["string"],
    "capability_steps": ["string"]
  },
  "impact": {
    "description": "string",
    "evidence_quotes": [ { "pid": integer, "quote": "string" } ]
  },
  "success_definition": {
    "qualitative_measures": ["string"],
    "quantitative_measures": ["string"]
  },
  "constraints_and_guardrails": [
    {
      "constraint": "string",
      "supporting_quote": { "pid": integer, "quote": "string" }
    }
  ],
  "who_it_affects": {
    "prevalence_count": integer,
    "prevalence_percentage": float,
    "description": "string",
    "signals": ["string"]
  },
  "models_consulted": ["gpt-5.2", "gemini-3.1-pro-preview", "claude-opus-4-6"]
}

Why Krippendorff's Alpha?

• Designed for multi-rater reliability (3+ raters)
• Handles missing data gracefully (if one model fails on a batch)
• Supports nominal-level measurement (categorical theme codes)
• Does not assume a fixed rater set
• More conservative than simple percent agreement, adjusting for chance

How It's Calculated

For each theme code, a binary matrix is constructed:

# Row per model, column per response
#                PID_1  PID_2  PID_3  PID_4  ...
# GPT:          [  1,     0,     1,     0,   ...]
# Gemini:       [  1,     0,     1,     0,   ...]
# Opus:         [  1,     0,     0,     0,   ...]

alpha = krippendorff.alpha(
    reliability_data=matrix,
    level_of_measurement="nominal"
)

Reporting

• Per-theme α values identify which themes models agree/disagree on
• Themes with α < 0.67 are flagged for potential human adjudication
• Overall mean α provides a summary reliability score
• Pairwise κ identifies whether specific model pairs diverge
• Code frequency counts reveal systematic over/under-coding by individual models

What IRR Tells Us (and Doesn't)

High α means the three models consistently apply the same theme to the same responses—the codebook is operationally clear and the models "understand" it similarly. Low α on a specific theme may indicate the theme definition is ambiguous, the theme requires human judgment the models handle differently, or the theme captures a rare pattern where base-rate effects inflate disagreement.

IRR does not tell us whether the codes are correct—only that the coders agree. This is why the human review gate exists: to ensure the codebook itself captures meaningful, well-defined themes before reliability is measured.

Model Parameters

Token Pricing (per 1M tokens)

The CostTracker class in llm.py tracks input, output, and thinking tokens separately per API call, with phase-level summaries printed to console.