Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseResearch Paper · For R&DMIT News→ original

AI Read Nearly 4000 Medical Papers: Code Is Open in Only 1 of 8

A "code available" link in a medical paper doesn't mean you can actually run the code. A team from MIT and Dana-Farber used an LLM to read nearly 4000 papers on models that estimate disease risk, and found open code in only about one in eight. The researchers then checked hundreds of repositories: some come with detailed instructions, others with almost nothing. The practical takeaway: a README, a dependency list and a license matter as much as the code itself.

AI-processed from MIT News; edited by Hamidun News
AI Read Nearly 4000 Medical Papers: Code Is Open in Only 1 of 8
Source: MIT News. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceMIT News ↗
Editor-in-Chief
Key metric #1The share of papers with code ranged from 0 to more than 70% depending on the journal
Key metric #2Some authors published synthetic data instead of patient records
Humans checked the LLM on 500 papersit was more accurate on licenses than on documentation

Anyone who has tried to run code from a scientific paper knows the feeling: there is a GitHub link, but nothing works. There is no dependency list, it's unclear what data to feed in, and there is no license either. An international team led by MIT and the Dana-Farber Cancer Institute measured the scale of this problem in medicine.

Using large language models (LLMs), it read nearly 4000 papers on clinical prediction models, meaning algorithms that estimate, for example, a patient's risk of developing a disease. Only about 1 in 8 papers, roughly 12%, report publishing code. For a developer, the implication is simple: "code is published" and "code can be run and verified" are different things, and in medicine treatment decisions depend on that difference.

How LLMs read four thousand papers

The authors selected papers in which clinical prediction models were developed, updated or validated. They took articles that cite the TRIPOD or TRIPOD+AI checklists (lists of what should be described when publishing such a model) and are available in the open PubMed Central collection. The results are published in Nature Medicine, and the findings will form the basis of TRIPOD-Code, a guideline now in development.

The first difficulty is that authors mention code in very different ways. Some put it in the methods section, some in a footnote, some at the end of the paper. As first author Thomas Sounack puts it, they first had to work out whether the author had shared code at all. There is no ready-made table for that, only scattered text. So the LLMs screened the papers, found phrases about code availability and extracted the links. Then ordinary software downloaded the repositories, and they were scored against a set of criteria: completeness of documentation, description of software environment requirements, and reuse terms.

Humans checked the LLMs' output. Two researchers independently reviewed 500 papers and a separate sample of repositories, discussed their disagreements and only then compared their assessments with the automated ones. Agreement was strongest on concrete criteria, such as whether a license file exists. It was weaker where subjective judgment is needed: whether the documentation is detailed enough. The authors stress that classification errors are possible, so the overall picture matters more than any single figure.

What they found

  • Code was published in only about 1 in 8 studies.
  • Repository contents varied widely. Some, in Sounack's words, were "extremely well organized and documented," while others offered almost no explanation.
  • The share of papers with code ranged from zero to more than 70% across journals. Differences between countries were also noticeable.
  • Some journals with a low share of published code did have a written code-sharing policy. Rules alone, it seems, are not enough to motivate authors.

Among the successful practices, the researchers noted synthetic datasets. When real patient records can't be published, authors released artificial data with the same structure, so that others could run the code and understand how it works.

Why this matters to practitioners

The minimum set another person needs to reproduce a result: a description of what the code does, run instructions, a dependency list, a license and, where possible, test data. Good code sharing, according to Sounack, means "giving enough information to understand what the code does and to run it, reproducing the results from the paper." If the original data is closed, a synthetic dataset remains a workable compromise. It is also useful in corporate projects where client data can't be handed over to contractors.

There is a second benefit: the way the team worked with LLMs. The model did not make final judgments. It did the draft work on scattered text, humans then checked a sample, and accuracy was assessed separately for simple and complex criteria. This approach can be applied to repository audits or document review. LLMs identify simple criteria such as "is there a license" reliably, while humans are still responsible for judgments such as "is the documentation detailed enough."

The limitations are worth keeping in mind too. The sample included only papers with open full text in PubMed Central, and subjective assessments of documentation produce more disagreement.

The next step is TRIPOD-Code. Through expert discussion, the team is determining which code elements should be mandatory. According to Tom Pollard of MIT, the goal is to give researchers practical advice and journals clearer requirements for authors. The review provides the factual basis for that.

Story

Key Questions Answered

How LLMs read four thousand papers?

The authors selected papers in which clinical prediction models were developed, updated or validated. They took articles that cite the TRIPOD or TRIPOD+AI checklists (lists of what should be described when publishing such a model) and are available in the open PubMed Central collection. The results are published in Nature Medicine, and the findings will form the basis of TRIPOD-Code, a guideline now in development. The first difficulty is that authors mention code in very different ways.

What they found?

Code was published in only about 1 in 8 studies. Repository contents varied widely. Some, in Sounack's words, were "extremely well organized and documented," while others offered almost no explanation. The share of papers with code ranged from zero to more than 70% across journals. Differences between countries were also noticeable. Some journals with a low share of published code did have a written code-sharing policy. Rules alone, it seems, are not enough to motivate authors.

Why this matters to practitioners?

The minimum set another person needs to reproduce a result: a description of what the code does, run instructions, a dependency list, a license and, where possible, test data. Good code sharing, according to Sounack, means "giving enough information to understand what the code does and to run it, reproducing the results from the paper." If the original data is closed, a synthetic dataset remains a workable compromise. It is also useful in corporate projects where client data can't be handed over to contractors.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…