ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering

The scientific method has long guided empirical research in Software Engineering (SE), yet the complexity of modern software systems often hinders its systematic application. This paper introduces ECLAIR, a causally grounded AI framework that integrates Large Language Models (LLMs) into every stage of the scientific process, from hypothesis generation to analysis and interpretation. Rather than treating LLMs as passive instruments, ECLAIR positions them as active scientific agents operating under the principles of causal inference, ensuring transparency, validity, and reproducibility in empirical reasoning. We demonstrate the framework through a case study examining how prompt design influences code generation accuracy in two LLMs. Results show that, for both models, instruction-style, longer few-shot, and signature-augmented prompts yield small negative causal effects on accuracy. These results illustrate how causal reasoning provides a principled foundation for explaining why software phenomena occur, rather than merely describing them. This study presents the first structured methodology for embedding LLMs within the scientific method in SE, establishing a basis for scalable, interpretable, and causally rigorous AI-assisted research.

Quick start

• Python 3.10+

• Install dependencies:

  python -m venv .venv
  source .venv/bin/activate
  pip install -r requirements.txt

• Work in VS Code dev container (recommended) or run Jupyter locally:

  python -m jupyter lab

• Open URLs from the terminal using the host browser:

  "$BROWSER" https://example.com

Repository layout (high level)

• data/: Outcomes/results from each step of the pipeline
  • experiments/H1.json, H11.json, H13.json: Lists of incomplete function stubs, prompts, and metadata
  • step_*.json: Collected results across pipeline stages
• datax/: Large assets: cached models, logits, logs, and analysis-ready data
• notebooks/: End-to-end analysis pipeline (run in order)
  1. 1_observation_A.ipynb / 1_observation_B.ipynb
  2. 2_hyphothesis_generation.ipynb
  3. 3_experiment_design.ipynb
  4. 4_causal_analysis.ipynb
  5. 5_analysis_results.ipynb
• scripts/: Helpers for experiments and causal analyses
• semeru-datasets/: External datasets used in analyses

Workflow overview

1. Explore experiment specs in data/experiments/*.json (e.g., H1.json).
2. For each task entry: use provided docstrings/prompts to implement the target stub (preserve structure/style; avoid extra deps; add concise comments).
3. Where feasible, add or adjust minimal tests; run project tests and linting.
4. Aggregate results into data/step_*.json and analyze with notebooks.

ECLAIR notebooks and the scientific method

• Observation (Notebook 1: 1_observation_A.ipynb / 1_observation_B.ipynb)

  • Goal: Understand the problem space and data characteristics.
  • Inputs: data/experiments/H1.json (and H11 or H13), raw logs in datax/ as needed.
  • Outputs: Descriptive summaries, initial patterns, candidate research questions.
  • Checks: Data sanity, coverage of task types, baseline heuristics.
  • ECLAIR role: Use LLMs to summarize artifacts, surface patterns, and propose initial questions with provenance.

• Hypothesis generation (Notebook 2: 2_hyphothesis_generation.ipynb)

  • Goal: Formulate testable hypotheses derived from observations.
  • Inputs: Summaries from Notebook 1, prior work notes, domain constraints.
  • Outputs: Explicit hypotheses, expected effect directions, metrics, confounders.
  • Checks: Falsifiability, measurability, traceability to observations.
  • ECLAIR role: Guide LLM-assisted hypothesis drafting while enforcing causal identifiability and testability constraints.

• Experiment design (Notebook 3: 3_experiment_design.ipynb)

  • Goal: Specify treatments and controls, sampling, and evaluation protocol.
  • Inputs: Hypotheses from Notebook 2, task pool from data/experiments/*.json.
  • Outputs: A concrete plan, for example splits, parameters, and a run matrix in data/step_3.json.
  • Checks: Power and coverage considerations, leakage avoidance, reproducibility.
  • ECLAIR role: Structure prompts, treatments, and controls, and codify evaluation metrics and run matrices.

• Causal analysis (Notebook 4: 4_causal_analysis.ipynb)

  • Goal: Execute experiments and estimate effects, for example ATE or ATT, with robustness checks.
  • Inputs: Run artifacts or logits in datax/, plan from Notebook 3.
  • Outputs: Effect estimates and intermediate tables, for example datax/causal_analysis/causal_data.csv.
  • Checks: Balance diagnostics, sensitivity or ablation analyses, multiple comparisons when applicable.
  • ECLAIR role: Orchestrate runs, compute effect estimates, and document assumptions and diagnostics.

• Results and conclusions (Notebook 5: 5_analysis_results.ipynb)

  • Goal: Summarize findings, interpret results, and derive conclusions or limitations.
  • Inputs: Outputs from Notebook 4 and earlier artifacts, for example data/step_5_*.json.
  • Outputs: Tables and figures, an executive summary, and actionable recommendations.
  • Checks: Link back to hypotheses, discuss threats to validity, outline future work.
  • ECLAIR role: Assist in drafting narratives, generating figures and tables, and ensuring conclusions map back to hypotheses.

Notes

• Keep edits minimal and localized; prefer clarity and efficiency.
• Track changes: file path, function names, imports or external calls, and rationale.
• Some folders in datax/ are large or cached; they are not required for quick exploration.

Repro tips

• Run notebooks in order; restart kernels between major phases.
• Use script entry points in scripts/ for batch operations.
• Pin random seeds when comparing methods; record environment details with outputs.

Appendix

Checklist overview

Below is a text version of the checklist in the image. It summarizes the items by phase.

• Observation

  • Problem has practical importance for software engineering decisions.
  • Comprehensive systematic literature review conducted.
  • Prior causal assumptions documented.
  • Knowledge gaps clearly identified.

• Hypothesis generation

  • Multiple competing hypotheses specified.
  • Hypotheses include naive, null, and current-practice alternatives.
  • Predicted directions and magnitudes documented.

• Experiment design

  • Experimental (not just observational) design used whenever possible.
  • Structural causal graph explicitly specified.
  • All confounders identified and addressed.
  • Complete replication protocol documented.
  • Methods are as simple as possible.

• Causal analysis

  • Data validity and reliability verified.
  • Simple, validated causal inference methods applied.
  • Multiple robustness checks conducted.
  • Effect sizes (not p-values) reported.
  • Practical significance assessed.

• Interpretation

  • Conclusions follow logically from the empirical evidence.
  • Limitations clearly stated.
  • Connections to prior literature made explicit.
  • Actionable recommendations provided.
  • Future research directions identified.

• Dissemination

  • Full disclosure enables replication.
  • References verified.
  • Peer reviews obtained before submission.
  • Compliance with Science Checklist completed.

License

To be determined. If using this in published work, please cite this repository and upstream datasets or projects referenced in the experiment specs.