An LLM decides which trials are eligible, and an LLM extracts every field. No human has reviewed any of those decisions, and there has been no accuracy audit against a reference set — so the error rate is not merely small, it is unmeasured. Assume false inclusions, false exclusions and misread fields are present throughout.
Use it to find trials fast. Do not cite it, and do not treat it as an evidence base without checking each record you rely on. Every record links back to PubMed, the DOI or ClinicalTrials.gov — that source is the authority; this is not.
See DATA_LICENSE.md for the full provenance statement.
An R package that scans for new randomized controlled trials (RCTs) relating to malaria, screens them for eligibility the way a systematic review would, extracts a structured record for each, and keeps a growing dataset plus a human-readable brief. Designed to run itself in GitHub Actions.
The pipeline is deliberately built like a systematic review — a sensitive search feeding a precise eligibility screen — because no search query alone can tell "malaria is the trial's purpose" from "malaria is mentioned in passing".
fetch (sensitive, malaria-scoped RCT search)
-> deduplicate across sources & identifiers
-> SCREEN (LLM eligibility: include/exclude + reason) <- precision
-> EXTRACT (LLM structured extraction on included only) <- quality
-> store
1. Search (favours recall; precision comes later):
- PubMed (NCBI E-utilities) — malaria scoped to MeSH/title/abstract, with a Cochrane-style sensitivity-maximising RCT filter. Carries recall.
- ClinicalTrials.gov (API v2) — interventional, randomized malaria registrations, incl. newly-posted results. Adds registered/ongoing trials and structured phase.
- Europe PMC — tighter query; its unique value is medRxiv preprints.
2. Screen — each candidate is judged by an LLM (systematic-review style): is this an RCT whose primary purpose is a malaria intervention, and not an early-phase study? Every candidate gets a recorded decision + reason. This removes non-trials (reviews, modelling papers, commentaries) and real trials that only touch malaria tangentially (iron, dengue, nutrition trials). Excluded candidates are logged for audit.
3. Extract — included records get structured extraction (LLM with a strict JSON schema,
or a regex fallback), including pulling place from the title.
Outputs:
data/trials.csv— one row per included trial (the database you'll use).data/trials.jsonl— full records, one JSON object per line (abstracts are not stored).data/screened_out.jsonl— excluded candidates + the reason (audit trail).data/state.json— dedupe memory, so nothing is screened or reported twice.outputs/Malaria_RCT_Brief.md— readable digest, newest week on top.docs/index.html— interactive explorer (see below); regenerated every run.
- RCTs only — randomized (incl. cluster-randomized). Reviews, observational and modelling studies are screened out.
- Malaria as the trial's primary purpose — drugs, vaccines, monoclonal antibodies, vector control (ITNs/IRS/spatial repellents/larviciding), chemoprevention (SMC/IPTp), diagnostics, elimination strategies.
- All phases except early-phase — Phase II/III/IV, "Not applicable" (typical for
vector-control and cluster trials), and unstated phase are kept; Phase I / first-in-human
/ dose-finding are excluded. Phase is recorded in the
phasecolumn so you can slice further. - Results-only (default) — the dataset holds trials that have reported results.
Study protocols, bare registrations, and ongoing/recruiting trials with no posted results
are excluded (published RCTs, results preprints, and ClinicalTrials.gov entries with
results posted stay in). Flip
screening.include_ongoing: trueinconfig.yamlto also keep the forward-looking registry pipeline. AdjustR/screen.Rif your scope differs.
Identity / screening: title, source & ID, journal, url, record_type
(trial/preprint/registration/protocol), screening_decision + screening_reason +
screening_confidence, phase.
Dates (kept unambiguous — no single overloaded date column):
publication_date— when the paper was published (PubMed/Europe PMC). Never overwritten by a registry edit.registry_updated— ClinicalTrials.gov last-update-posted date (a registry edit, not a publication).trial_start/trial_completion— when the trial actually ran (ClinicalTrials.gov structured dates, or pulled from the abstract by the extractor).first_seen— when the record first entered your dataset (the run date).
Tier 1: place, design, status, intervention(s) + normalised class, comparator, population, N, primary outcome, impact summary.
Tier 2: effect metric / estimate / 95% CI, p-value, follow-up, species, transmission
setting, age range, safety summary, funder. Plus extracted_by (which model/rules filled it).
Needs R (>= 4.1). Clone the repo, then install the package + dependencies:
install.packages(c("httr2", "jsonlite", "xml2", "yaml")) # runtime deps
devtools::load_all(".") # work with it live (or install.packages(".", repos = NULL, type = "source"))Copy .env.example to .env (git-ignored) and add your key(s):
ANTHROPIC_API_KEY— needed forscreening.mode: llmandextraction.mode: llm. A pay-as-you-go key from https://console.anthropic.com (separate from a Claude.ai subscription). A dedicated key in a spend-limited workspace is recommended.NCBI_API_KEY— optional, free, raises PubMed rate limits.
Always run from the project root.
library(malariarct) # or devtools::load_all(".")
run_weekly() # weekly run (last ~10 days)
run_backfill(days = 3650) # last 10 years
run_backfill(start = "2015-01-01", end = "2020-12-31")
run_backfill(days = 3650, mode = "rules") # extraction via regex only (screening still runs)Or from a shell:
Rscript scripts/run_weekly.R
Rscript scripts/backfill.R --start 2015-01-01 --end 2020-12-31Do a backfill once to seed the dataset locally, commit data/, then let the weekly
GitHub job take over. Re-runs are safe and cheap: already-decided records are never
re-screened or re-reported.
Long runs are checkpointed and resumable. The dataset is written to disk after every
checkpoint_every candidates (default 50, in config.yaml), so if a run is interrupted —
you stop it, the machine sleeps, the network drops, R crashes — all completed work is kept.
Just run the same command again and it resumes, skipping everything already decided and
processing only the remainder. So a 10-year backfill can be stopped and restarted freely, and
you never pay twice for the same record. A record is only ever re-processed if it was
deferred — either beyond the per-run screening cap, or because a transient API failure
(timeout, overloaded) meant it couldn't be screened this time; those retry on the next run
rather than being silently downgraded to a lower-quality rules decision.
After an interrupted run you can (but rarely need to) tidy up with reconcile_dataset(),
which removes any duplicate row a crash-at-the-wrong-instant might have left and rebuilds
state.json from what's actually stored.
Two model tiers (set in config.yaml):
- Screening —
claude-haiku-4-5(cheap; one short call per candidate). - Extraction —
claude-sonnet-5(stronger; runs only on the smaller included set).
A one-off 10-year backfill is roughly $30–50; the weekly job is pennies. For a free run,
set both mode: rules/off. Cap real spend in the Anthropic Console (prepaid credit
with auto-reload off, or a workspace spend limit) — that's the hard ceiling.
Since nothing here is validated, this is the part that matters most.
Abstracts are not stored. They are fetched, used in memory to screen and extract, then dropped — publisher copyright is not ours to redistribute in bulk. So reviewing an extraction means following the record back to its source:
library(jsonlite)
d <- stream_in(file("data/trials.jsonl"))
d[, c("id", "url", "phase", "place", "impact_summary", "effect_estimate", "extracted_by")]Every id is prefixed (doi:, pmid:, nct:) and every record carries url, so each
one resolves to the publisher or registry page. The explorer's detail drawer turns these
into one-click links.
data/screened_out.jsonl lets you audit what was excluded and why — spot-check for
false exclusions. This matters as much as checking inclusions: a systematically
over-eager screen is invisible if you only look at what made it in.
Encoding. The files are UTF-8 (titles contain ≥, –, accented place names, etc.).
In R they read correctly with readr::read_csv() or jsonlite::stream_in(). Excel ignores
UTF-8 on a plain double-click and shows mojibake like ≥ — open via Data → From Text/CSV
and pick 65001: Unicode (UTF-8), or read in R and export with writexl::write_xlsx().
Every run also writes a self-contained docs/index.html — an interactive explorer for
the whole dataset. It's a single file with the data embedded inline — no server, no build
step, no external assets — so you just open it:
render_explorer() # rebuild it any time from the stored dataset
browseURL("docs/index.html") # open it in your browserWhat's in it:
- Summary cards, a trials-over-time chart (toggle publication vs. trial-start date) and a by-intervention-class chart. Each chart states how much of the current selection it actually plots — the trial-start view covers well under half the dataset, and that should be visible rather than read as a real decline.
- Evidence gap matrix — intervention class against country / outcome family / population band / publication era. The point of it is the empty cells, which are drawn as explicit hatched blanks rather than white space. Click any cell to filter the table to it. Counts are trials, not evidence quality, and a trial with several classes or outcomes appears in each of its cells, so cells don't sum to the total.
- Where trials happen — countries ranked by trial count. An optional per-burden view exists but is off by default; see Burden denominators.
- Trial start to publication — the lag distribution, with a median. Covers the ~40% of records carrying both dates.
- Activity timeline — a GitHub-contribution-style density grid, intervention class down,
year across, shaded by how many trials were running that year (a trial occupies every
year between its start and completion, so a column is concurrent activity, not new trials).
Click a cell to filter to it. Two honest caveats are shown on the chart and matter:
trial_startis only ~41% filled and its coverage is uneven by class — 50% for ITN/LLIN, 29% for vaccine — so each row carries its ownn/N, and a sparse row may mean missing dates rather than no research. The recent years are also right-censored: a trial that started recently usually hasn't published, so it isn't in the dataset at all, and the fall-off on the right is an artefact. A Published toggle switches to publication year (98.6% filled) — near-complete, but it shows when evidence landed, not when the work was done. Shading uses five discrete levels with quantile thresholds, not a linear ramp: the counts are heavily skewed (median 3, max 35, 91% of cells ≤ 10) so a linear scale renders most of the grid as near-identical faint squares. Shared scale keeps rows comparable to each other; Per class rescales each row to its own range, so a small class shows its own shape at the cost of cross-row comparability. - Filters: free-text search (multi-word narrows, it doesn't match the raw phrase), plus
intervention class / phase / species / country / outcome family / population / record type,
over a sortable table.
The country facet is built from the normalised
countriescolumn, not from splitting the free-textplace— see Geography. - Click any row for the full record. The table shows 8 columns; the drawer shows all 39
fields grouped as the schema groups them, with links rebuilt from the record
id(DOI / PubMed / ClinicalTrials.gov). Fields the extractor didn't fill are shown as not reported rather than hidden — witheffect_ciat 52% andfunderat 17%, an absent value is itself information. - Shareable URLs. Every filter, the sort, and the open trial live in the address bar, so a view can be pasted into an email or a protocol and reopens exactly. Back closes the drawer.
- Extraction coverage panel (under the filters): per-field fill rates for the current selection, plus one-click flags for records the extractor struggled with — estimate without a CI, N that is really a cluster count, no publication date, no country, no class, rules-only extraction. Check this before you use any field as an analysis variable.
- Export the filtered set as BibTeX or RIS. (For CSV, just use
data/trials.csvdirectly — you have R.)
Because it's a plain static file, it works offline and could later be served with GitHub Pages
(Settings → Pages → Deploy from branch → main → /docs). Note that on a private repo,
GitHub Pages is public unless you're on a paid plan — so publishing it would expose the
embedded dataset. It's left unpublished by default; keep it local, or enable Pages deliberately.
The geography chart can show trials per unit of malaria burden instead of raw trial counts. Raw counts largely restate where research infrastructure is; the normalised view shows where the evidence base is thin relative to where malaria actually is, and it recomputes under whatever filter is active — so "per-burden coverage of vaccine trials" is answerable, not just the global picture.
It ships off. There is no data/burden.csv in the repo, so the toggle is hidden and the chart
shows plain counts. Nothing to maintain unless you want the view. To turn it on, create the file:
country,cases,year,source
Nigeria,66800000,2022,WHO World Malaria Report 2023
Uganda,12700000,2022,WHO World Malaria Report 2023
Then render_explorer(). No re-screening, no extraction, no cost.
Notes if you do populate it:
countrymust match the controlled names in thecountriescolumn (see Geography).- Countries absent from the file are omitted from the normalised view — never treated as zero burden — and the chart states how much of the current selection it could cover.
- Only relative burden matters to the ranking, so share-of-global-cases works as well as
absolute counts and ages far more slowly — WHO's per-country shares move by a fraction of a
percentage point a year, while absolute estimates move with global totals and methodology
revisions. Record the
yeareither way so staleness is visible. - Half-populating it is worse than leaving it off: the chart looks complete unless the note is read.
place is whatever the extractor read off the abstract — "Uganda", but also
"India (Gujarat: Kheda, Vadodara, Panchmahal districts)" and "Africa (five African countries)". It's kept verbatim for display, and two derived columns are stored
alongside it for slicing:
countries— controlled country names,"; "-joined ("Burkina Faso; Mali").region— a multi-country region ("Africa","Southeast Asia") when no country is named, so region-level records stay visible instead of vanishing from a country facet.
Two further controlled columns are derived the same way, and drive the gap matrix:
outcome_family— what the trial measured (therapeutic efficacy,clinical incidence,infection prevalence,entomological,mortality,pregnancy/birth,anaemia,immunogenicity,safety,coverage/cost). Taken fromprimary_outcome— the pre-specified endpoint — falling back toeffect_metric/impact_summaryonly when that names nothing recognisable. 84% coverage. It exists so intervention can be crossed against outcome; it is not a licence to compare effect sizes, since two trials in one cell can still differ in endpoint definition, comparator and follow-up.population_band—pregnant women,infants (<1y),children <5,school-age children,children (age unspecified),adults,all ages. 67% coverage.
Both come from an ordered gazetteer in R/normalize.R. Rules are applied in order and each
match is consumed before the next rule is tried, which is what keeps the containment cases
right: Papua New Guinea is claimed and removed before the bare Guinea rule sees the string,
likewise South Sudan before Sudan and South Africa before the Africa region. Spelling
variants collapse (both DRC forms, both apostrophes in Côte d'Ivoire, The Gambia/Gambia).
This is regex over text you already have — no LLM, no API calls, no cost. After editing the rules:
reclassify_store() # re-derive intervention_class + countries/region for every stored recordwhich rewrites trials.csv / trials.jsonl and regenerates the brief and explorer. Run it
after any schema change that adds a derived column, too — a plain append can't rewrite the
CSV header.
This is lossy, despite appearances.
intervention_classis derived from a blob that includes the abstract — and abstracts aren't stored. So re-classifying a stored record works from title and conditions alone, and can produce a worse value than the row already holds, because the original pass had the abstract in hand.reclassify_store()warns when it sees records with no abstract.To re-classify faithfully, re-fetch the abstracts first (you hold every PMID, DOI and NCT id — about 15 minutes with an
NCBI_API_KEY) and pass the rehydrated records in:reclassify_store(records = my_rehydrated_records).
If you review the dataset and want to remove a record — a false positive the screen
let through — the wrong way is to just delete its row from trials.csv. That works
only by accident (its id is still in state.json, which blocks re-adding), and it comes
straight back the day you reset state or run a fresh backfill.
The right way is the durable blocklist. From R:
exclude_records("doi:10.1016/j.xyz.2025.01.002", reason = "not actually a malaria trial")That does two things: appends the id to data/exclusions.txt and removes the row from
trials.csv / trials.jsonl. You can pass several ids at once (exclude_records(c("pmid:123", "nct:nct0456"))).
data/exclusions.txt is a plain, hand-editable list (one id per line, # comments allowed) —
you can also just open it and paste ids in. The pipeline reads it before screening on every
run and drops anything matching, by id or any identity key — so an excluded trial stays
out even if it later reappears under a different identifier, and even after a full re-backfill.
Use the id column (e.g. doi:…, pmid:…, nct:…) as the key. Commit data/exclusions.txt
along with the rest of data/ so the exclusion is honoured on other machines and by the
GitHub Actions runs.
.github/workflows/weekly.yml— Monday scan, commits results back..github/workflows/backfill.yml— manual "Run workflow" deep backfill (days / start / end / mode / max_items inputs)..github/workflows/check.yml— runs the test suite on code changes.
Setup: push to GitHub; add ANTHROPIC_API_KEY (and optionally NCBI_API_KEY) as an Actions
secret; Settings → Actions → General → allow Read and write workflow permissions. Works
on private repos (uses your free Actions minutes; this job uses very few).
data/ is committed, so anything that writes it is a writer: your local runs, the weekly
cron, and the manual backfill workflow. If two write at once (e.g. you run a local backfill
while the Monday job fires), you get a git merge conflict on the data/ files — and because
each run de-dupes against its own state.json, a trial in the overlapping window can be
appended twice.
Best practice — keep a single writer. The cleanest setup is to let GitHub own the
dataset: run backfills via the manual backfill.yml workflow, not locally. Both CI
workflows share a concurrency group, so they never overlap, and the weekly job does
git pull --rebase before pushing. Use local runs for testing (on throwaway data) or, if you
do write locally, git pull first and git push immediately after — or pause the weekly
workflow while a long local backfill runs.
If a conflict happens anyway, two things make it painless:
-
.gitattributessets a union merge ontrials.csv/trials.jsonl/screened_out.jsonl/exclusions.txt, so both sides' rows are kept automatically instead of raising a conflict. -
After resolving the merge (take either side of
state.json— it gets rebuilt), run:reconcile_dataset()
which removes any duplicate rows and rebuilds
state.jsonto match what's actually stored (plus the screened-out log).seen_idsis only ever unioned, so nothing is forgotten.
testthat::test_local()DESCRIPTION NAMESPACE config.yaml # package metadata + all settings
R/
http.R llm.R record.R # HTTP helper; Anthropic client; record schema
fetch_pubmed.R fetch_clinicaltrials.R fetch_europepmc.R fetch_ictrp.R
normalize.R dedupe.R screen.R extract.R store.R
render_brief.R render_explorer.R # markdown brief; interactive HTML explorer
config.R pipeline.R utils.R # config/.env; orchestration; helpers
inst/ explorer_template.html # the explorer page template (data injected in)
scripts/ run_weekly.R backfill.R
tests/testthat/ # unit tests
data/ trials.csv trials.jsonl screened_out.jsonl state.json
outputs/ Malaria_RCT_Brief.md
docs/ index.html # self-contained interactive explorer
.github/workflows/ # weekly.yml, backfill.yml, check.yml
legacy_python/ # the old Python version (safe to delete)
- Precision comes from the screen, not the query — if you see a wrong inclusion/exclusion,
tune the eligibility rules in
R/screen.R, not the search. - Phase is a drug/vaccine concept; vector-control trials are "Not applicable" phase and are deliberately kept (see Scope).
- One source failing (timeout etc.) is logged and skipped; the run completes with the others.
- Extraction never overwrites a value a source already provided; it only fills gaps.
- Keep this repo outside cloud-synced folders (OneDrive/Dropbox); git + file-sync fight over
.git.
- Code — MIT, see LICENSE.md.
- Data (
data/,outputs/,docs/index.html) — CC BY 4.0, see DATA_LICENSE.md, which also carries the full provenance statement.
Source abstracts are deliberately not stored or redistributed: they are fetched, used in memory to screen and extract, then dropped. Each record links back to the source instead.
If you reuse the data, please carry the limitation with it — that it is LLM-screened, LLM-extracted and unvalidated is the most important thing about it.