This is a release of the Drug Mechanisms Counterfactuals Dataset, developed for evaluating Large Language Models on their ability to recall and reason about drug mechanisms.
The dataset is described in our accompanying paper published in FLLM 2025 (see below), and also available as a preprint.
The dataset and related code is available for use under the MIT license.
The counterfactual queries, and the related factual queries, are available as JSON files in the Data/Counterfactuals sub-directory. See this README file.
The counterfactual queries are derived from DrugMechDB; using the Add-Link queries, and any Closed World queries (where the prompt includes information about relevant drug mechanisms), requires downloading DrugMechDB. See this README file.
DrugMechDB is also required for the factual queries described in the accompanying paper. The code also allows you to create your own factual queries, for which both PrimeKG and MONDO are required. See this README file for the needed files.
Python code for the evaluation framework can be found under the src/ sub-directory. Requirements for using the code are listed in requirements.txt. The code has been tested with Python version 3.11.5, but should also work with more recent versions.
Shown below is an example setup for using the code. For convenience, we will use the environment variable PROJDIR to refer to the path where DrugMechCounterfactuals has been cloned, e.g. PROJDIR=/Users/Me/Projects/DrugMechCounterfactuals. Note that this environment variable is not needed by the code or the setup process.
-
Create a virtual environment for the project, e.g.
$ cd $PROJDIR $ uv venv --python 3.12 --seed --prompt dmcf $ source .venv/bin/activate (dmcf) $This creates a virtual environment in
$PROJDIR/.venv. -
Install the requirements, e.g.
(dmcf) $ uv pip install -r requirements.txt -
Invoke the appropriate Python scripts from the
srcsub-directory. For example, to pretty-print samples from the Add-Link positives surface-counterfactuals query set:(dmcf) $ cd $PROJDIR/src (dmcf) $ python -m drugmechcf.data.cfdata examples ../Data/Counterfactuals/AddLink_pos_dpi_r1k.json
Please consult the following documents for a more detailed description of the Dataset and the Evaluation Framework:
- The dataset
- The Evaluation Framework
- How to test new models on the dataset
- Evaluation of some LLMs on the dataset
@inproceedings{DrugMechCounterfactuals:FLLM,
title = {How Well Does {ChatGPT} Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Dataset},
author = {Sunil Mohan and Theofanis Karaletsos},
booktitle = {2025 3rd International Conference on Foundation and Large Language Models (FLLM)},
year = {2025},
pages = {},
doi = {},
publisher = {IEEE},
}
@misc{DrugMechCounterfactuals:arXiv,
title={How Well Do LLMs Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Dataset},
author={Sunil Mohan and Theofanis Karaletsos},
year={2025},
eprint={2511.06418},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2511.06418},
}
This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to opensource@chanzuckerberg.com.
If you believe you have found a security issue, please responsibly disclose by contacting us at security@chanzuckerberg.com.