Why biomedical AI research needs controlled protocols to distinguish model capability, agentic architecture, and investigator contribution.
BBIU | Strategic Analysis | AI & Biomedical Research
The Question Behind the Results
The 2026 Nature Medicine study by Roberts and colleagues, An Open Agentic AI Co-Scientist for Clinical and Biomedical Discovery, introduces OpenScientist, an open-source artificial intelligence platform capable of conducting iterative scientific investigations, executing computational analyses, retrieving literature, and generating testable biomedical hypotheses.
The results demonstrate meaningful technological progress.
Across five featured biomedical case studies, OpenScientist reproduced established analytical findings, developed predictive models, generated biologically plausible hypotheses, and demonstrated agreement with independent human analyses.
Its open-source architecture also addresses important concerns surrounding proprietary scientific AI systems, particularly auditability, reproducibility, and institutional control over sensitive research data.
However, these achievements raise a methodological question with implications extending far beyond OpenScientist:
How much of the observed scientific performance originates from the artificial intelligence system itself, and how much is contributed by the investigator directing it?
A second question follows:
Does a specialized co-scientist architecture provide measurable scientific value beyond the capabilities of its underlying large language model and associated computational tools?
Answering these questions requires more than reproducing successful scientific outputs.
It requires an experimental framework capable of separating model performance, architectural contributions, human expertise, and the effects of iterative interaction.
1. Uncontrolled Procedural Variability: A Challenge to Scientific Attribution
Scientific reproducibility requires not only repeatable computational results, but sufficiently controlled conditions under which those results are generated.
The OpenScientist study provides an instructive example.
During its Alzheimer's disease biomarker investigation, researchers performed ten exploratory executions while refining their use of the platform.
Initial analyses revealed methodological discrepancies, including incorrect treatment of missing values, inappropriate data-handling assumptions, inconsistent reporting, and problems with cohort classification.
Investigators subsequently addressed these issues by improving data preparation and specifying analytical requirements more precisely.
The study presents the final optimized execution while also documenting the exploratory process and its limitations.
These corrective actions were scientifically appropriate. They demonstrate the importance of expert oversight, a conclusion explicitly acknowledged by the authors.
Nevertheless, the resulting performance reflects an optimized experimental configuration informed by investigator knowledge and previous execution outcomes.
It should therefore not be interpreted as an isolated measure of the system's initial autonomous capability.
Three sources of variation must be distinguished:
Uncontrolled procedural variability: differences in input formulation, data preparation, computational settings, and intervention procedures that confound experimental comparisons.
Systematic bias: distortions introduced through selective optimization, investigator expectations, or preferential interpretation of favorable outcomes.
Genuine performance variability: differences attributable to model capability, scientific expertise, system architecture, and the nature of human–AI interaction.
A robust experimental protocol should minimize the first, control the second, and quantify the third.
Importantly, removing uncontrolled procedural variability does not mean eliminating scientifically meaningful errors from test datasets.
The ability to detect such errors is itself an essential measure of scientific competence.
2. Standardization Without Suppressing Scientific Complexity
The five featured OpenScientist investigations addressed substantially different research questions, ranging from Alzheimer's disease biomarkers and survival prediction to neurodegeneration, multiple myeloma, and clinical-trial proteomics.
This diversity demonstrates the platform's potential applicability across biomedical disciplines.
However, the cases differed in analytical constraints, input materials, scientific objectives, and evaluation procedures.
Some investigations were guided by relatively specific analytical plans, while others allowed broader exploration and hypothesis generation.
These differences are appropriate for demonstrating versatility, but they limit direct comparison of scientific performance across cases.
The solution is not to impose identical biological problems.
Rather, it is to establish equivalent procedural conditions and prespecified evaluation standards.
A standardized framework should preserve domain-specific endpoints while applying consistent criteria for methodological validity, reproducibility, error detection, evidentiary support, and uncertainty reporting.
The objective is uniformity of experimental conditions, not uniformity of scientific conclusions.
3. Why a Co-Scientist Instead of a Conventional LLM?
OpenScientist combines a large language model (LLM) with scientific tools, executable code, literature retrieval, specialized analytical capabilities, and iterative agentic orchestration.
These components are intended to extend the capabilities of conventional conversational systems.
Yet the incremental scientific contribution of the specialized architecture should be established experimentally.
An apparent performance advantage may arise from several sources:
The underlying model's scientific reasoning capabilities.
Access to statistical software, datasets, and scientific literature.
The agentic architecture's ability to organize and execute multi-step investigations.
Human expertise applied through instructions, corrections, and methodological supervision.
Our proposed framework distinguishes three technological configurations.
Configuration 1 — Conventional LLM
The underlying language model operates without specialized scientific orchestration. This establishes a secondary reference for unaided model capabilities on tasks suitable for this interface.
Configuration 2 — Tool-Equipped LLM
The same model receives comparable access to computational tools, datasets, and scientific literature, but without the specialized OpenScientist orchestration layer.
Configuration 3 — OpenScientist
The same underlying model operates through the complete co-scientist architecture.
The primary comparison should be between Configurations 2 and 3, using equivalent data access, computational budgets, model versions, and resources wherever technically feasible.
This prevents attributing benefits from Python execution, literature retrieval, or additional computational resources exclusively to agentic orchestration.
It also distinguishes scientific performance from engineering complexity.
4. A Controlled Human–AI Interaction Protocol
We propose a complementary experimental study designed to identify the contribution of investigator expertise and unrestricted interaction to scientific performance.
Three participant groups
Participants would be stratified according to their scientific backgrounds:
Group 1 — Biomedical Domain Experts
Researchers with established expertise in the discipline being investigated.
Group 2 — Scientific Generalists
Participants with formal scientific training but without equivalent domain specialization.
Group 3 — Non-Specialists
Participants without specialized biomedical training.
Prior experience with artificial intelligence would be recorded separately.
Scientific expertise and AI interaction proficiency are distinct variables and should not be treated as interchangeable.
Two independent interaction conditions
Each participant would be evaluated under two experimental conditions using independent, clean AI sessions.
Condition A — Standardized Interaction
The tester receives a predefined pool of scientific questions and permitted interaction steps.
The participant must follow these instructions without introducing discretionary corrections, additional hypotheses, or independently formulated questions.
The AI session begins without prior conversational history, case-specific memory, or previously accumulated instructions.
The standardized condition establishes a common reference for scientific performance under controlled input.
Condition B — Unrestricted Interaction
The same participant begins a second independent AI session, equally free of prior conversational history.
The initial scientific objectives and predefined question pool remain unchanged.
However, the participant may now:
Reformulate questions and request additional analyses.
Challenge assumptions and statistical procedures.
Investigate contradictory evidence.
Identify potential errors.
Propose alternative hypotheses.
Redirect the analytical investigation.
All interactions are recorded.
The key experimental feature is the comparison of standardized and unrestricted interactions performed by the same individual.
This reduces variability associated with stable differences between participants.
Preserving first-exposure validity
A clean AI session does not automatically guarantee an experimentally inexperienced human participant.
Once a tester has completed one condition, the individual may have acquired knowledge that influences performance in the next.
Accordingly, interaction order should be randomized and counterbalanced, with scientifically equivalent but nonidentical cases assigned across conditions.
The first exposure should be analyzed separately from subsequent interactions.
Residual learning or carryover effects must be assessed explicitly. Randomizing execution order reduces systematic imbalance but does not eliminate knowledge transfer between sessions.
Time limits, data access, and computational resources should also be controlled.
Otherwise, differences between conditions could reflect unequal resources rather than the scientific value of interaction freedom.
5. Operationalizing Scientific Validity
The primary endpoint must measure more than whether a report appears scientifically convincing.
We propose a prespecified, task-level assessment of verified scientific decision validity.
Each biomedical task would contain explicitly defined critical decisions or analytical checkpoints.
Examples include appropriate cohort selection, correct handling of missing data, suitable statistical procedures, recognition of false-positive signals, and conclusions consistent with available evidence.
A decision would qualify as valid only when it satisfies the applicable methodological requirements and is adequately supported by evidence.
A structured adjudication framework
Each task would be evaluated against four domains:
Analytical correctness: whether relevant computations and statistical procedures are appropriate and correctly implemented.
Methodological integrity: whether data handling, experimental assumptions, controls, and uncertainty are properly addressed.
Evidentiary validity: whether conclusions are supported by the available data and do not exceed the strength of the evidence.
Scientific judgment: whether alternative explanations, contradictory findings, and limitations are appropriately considered.
Critical criteria would be prespecified for each task.
Objective reference answers, executable validation checks, and known negative controls should be used wherever available.
Expert judgment would remain necessary for complex mechanistic interpretations, but such judgments should be distinguished from objectively verifiable findings.
Unsupported mechanistic hypotheses should not be classified as experimentally validated discoveries.
Inter-rater reliability
At least two independent domain experts would evaluate each output.
Agreement would be measured using statistics appropriate to the endpoint, such as Cohen's kappa or Gwet's agreement coefficient for categorical judgments and intraclass correlation coefficients for continuous scores.
Acceptable reliability thresholds would be established during protocol development, with the final assessment procedure frozen before confirmatory testing.
Disagreements would be resolved through a prespecified adjudication process.
The confirmatory analysis should also report the frequency and nature of adjudicator disagreement.
Protecting evaluation blinding
A critical additional safeguard concerns the presentation of AI-generated reports.
Agentic systems often produce recognizable report structures, terminology, and formatting.
Removing the system's name may therefore be insufficient to blind evaluators.
All outputs should be converted into a standardized, provenance-neutral reporting format before adjudication.
This normalization must preserve scientific content, numerical results, relevant uncertainty, and methodological evidence.
A separate assessment should determine whether adjudicators can identify the originating technology more accurately than chance.
Blinding of the evaluator must be distinguished from blinding during execution: participants will generally recognize the systems they use, while scientific adjudicators should remain unaware of the assigned experimental condition.
6. Experimental Controls and Error Detection
The proposed protocol would include predefined positive and negative controls.
Negative controls would incorporate scientifically relevant challenges, including:
Missing observations and duplicate records.
Incorrect assumptions of statistical independence.
Multiple-testing problems.
Randomized disease labels.
Apparent associations lacking robust evidence.
Plausible but unsupported mechanistic explanations.
These challenges should be documented in the validation protocol but concealed from participants and AI systems.
Their purpose is not to generate artificial failures.
It is to determine whether a research system can recognize methodological problems that routinely arise in biomedical investigation.
Importantly, the experimental datasets should be prepared, quality-controlled, and frozen before confirmatory execution.
No investigator should modify the confirmatory data or evaluation rules in response to observed AI performance.
Every execution must remain in the analysis denominator, including unsuccessful, incomplete, and inconclusive attempts.
Exploratory refinement would be permitted only within a separately defined development phase.
This separation prevents optimized exploratory results from being mistaken for independent confirmatory evidence.
7. Statistical Attribution and Power
The primary comparison would examine whether unrestricted interaction improves the probability of a scientifically valid decision relative to standardized interaction.
A generalized linear mixed-effects model (GLMM) would be suitable for a binary primary endpoint.
An illustrative specification is:
logit[P(Y = 1)] = β0 + β1(Modality) + β2(Configuration) + β3(Modality × Expertise) + β4(Modality × Configuration) + β5(Period) + β6(Sequence) + uParticipant + vCase
Here:
Y indicates whether a prespecified scientific decision meets the validity criteria.
Modality distinguishes standardized and unrestricted interaction.
Configuration identifies the technological system.
Expertise represents the participant's scientific background.
Period and sequence account for execution order and experimental allocation.
Participant and case random effects account for repeated observations and differences in task difficulty.
The final model specification would depend on pilot results and the selected randomization structure.
Because the standardized condition prohibits discretionary human input, expertise should primarily be evaluated through its interaction with unrestricted participation rather than interpreted as an active independent influence during standardized execution.
The design should also investigate model–user interaction effects and allow for the possibility that unrestricted guidance performs differently across technological configurations.
Order effects can be estimated, but residual carryover cannot be assumed absent.
Pilot feasibility
An initial pilot of approximately 36 participants across three expertise groups could establish procedural feasibility and provide preliminary variance estimates.
This should not be interpreted as a sufficiently powered confirmatory sample.
The pilot would assess:
Recruitment and retention rates across expertise groups.
Completion of standardized and unrestricted interactions.
Adherence to predefined interaction rules.
Reliability of independent scientific adjudication.
Variability across cases and technologies.
Time and expense associated with execution and validation.
Recruitment of qualified biomedical experts may represent a substantial logistical constraint.
Participation may also be selective: researchers interested in AI could differ systematically from those who do not volunteer.
These limitations should be explicitly reported.
The confirmatory sample size should be determined using simulation-based power calculations informed by pilot estimates of event rates, within-participant correlation, between-case variability, and plausible interaction effects.
Primary contrasts and multiplicity adjustments should be prespecified to avoid generating an excessive number of exploratory comparisons.
8. Measuring Economic Value: Cost per Verified Valid Conclusion
Scientific accuracy alone does not establish operational value.
Research organizations must also consider the total resources required to produce and verify an acceptable scientific result.
We propose an economic metric:
Cost per Verified Valid Conclusion (C-VVC)
C-VVC = Total attributable research and verification cost / Number of independently verified valid conclusions
The numerator should include:
Model inference and computational infrastructure.
External tools and data processing.
Investigator interaction time.
Independent scientific validation.
Error correction and necessary reanalysis.
The denominator must be prespecified.
Otherwise, systems that produce large numbers of superficial findings could appear economically superior simply by increasing output volume.
We would therefore count only conclusions associated with predefined scientific objectives and meeting independent validity criteria.
When no conclusion satisfies the verification requirements, the corresponding cost and zero successful outcomes should be reported explicitly rather than assigning a misleading finite ratio.
C-VVC would be a secondary economic endpoint, interpreted alongside scientific validity and total resource consumption.
It would allow research executives to evaluate whether a more complex agentic system produces sufficiently reliable benefits to justify its computational, supervisory, and validation costs.
A conventional analysis performed by a qualified bioinformatician or biostatistician could provide a supplementary operational comparator when feasible, although the primary controlled experiment would remain focused on human–AI interaction.
9. Attribution Is Not the Opposite of Collaboration
An important distinction must be preserved.
The objective of the proposed framework is not to prove that autonomous AI is inherently preferable to human–AI collaboration.
In many biomedical settings, collaboration may be the desired operational model.
A system that performs moderately well autonomously but exceptionally well when guided by experienced scientists may still offer substantial scientific and commercial value.
Conversely, a system that performs strongly without supervision but cannot recognize or incorporate valid expert corrections may present different operational risks.
Several outcomes are possible.
Experienced researchers may significantly improve conventional LLM performance while providing limited additional benefit to an agentic platform.
Alternatively, specialized agentic architectures may reduce the dependence on extensive user expertise.
Unrestricted interaction may also increase analytical complexity, introduce confirmation bias, or produce diminishing returns.
Each outcome would be scientifically informative.
If expert-guided conventional LLMs achieve results comparable to more complex co-scientist architectures, organizations may derive greater value from investing in scientific expertise, training, and standardized interaction procedures.
If specialized orchestration produces measurable gains under equivalent conditions, its incremental value would be more directly justified.
If neither approach consistently improves independently verified scientific conclusions, apparent gains in automation may simply transfer work into downstream verification.
The purpose of the experiment is to distinguish among these possibilities, not to favor one architecture in advance.
10. Implications for Biomedical R&D Strategy
Pharmaceutical companies, biotechnology developers, contract research organizations, and academic institutions are increasingly evaluating AI-based scientific workflows.
Procurement and investment decisions often emphasize processing speed, automation, computational capacity, and the sophistication of generated reports.
These indicators are insufficient on their own.
Research executives need to understand:
What can the system accomplish independently?
What additional value does specialized agentic orchestration provide compared with an equivalently equipped LLM?
How much scientific expertise is required to achieve reproducible performance?
What is the total cost of producing an independently verified conclusion?
These questions influence recruitment, training, research governance, technology procurement, and capital allocation.
A more sophisticated AI system is not necessarily the more efficient scientific system.
The relevant measure is its contribution to valid, reproducible, and decision-relevant scientific knowledge.
Conclusion: From Impressive Outputs to Attributable Scientific Performance
OpenScientist represents an important contribution to transparent and extensible biomedical AI.
The study by Roberts and colleagues demonstrates meaningful capabilities, documents limitations, and provides a valuable foundation for evaluating AI-assisted scientific research.
Our proposed framework does not challenge those achievements.
It addresses a complementary question that becomes increasingly important as AI systems gain greater scientific autonomy.
Can we independently attribute scientific performance to the underlying model, its computational tools, its agentic architecture, and the expertise of the investigator?
Answering this question requires controlled experimental conditions, independent AI sessions, standardized and unrestricted interaction arms, objective scientific checkpoints, blinded evaluation, and transparent accounting of successful and unsuccessful executions.
It also requires recognizing that scientific value may arise from interaction rather than autonomy alone.
The objective is not to eliminate human participation.
It is to identify when human participation improves scientific outcomes, when it introduces bias, and when specialized AI architectures provide demonstrable benefits beyond conventional alternatives.
The next standard for biomedical AI validation should move beyond asking whether artificial intelligence can produce scientific results. It should establish how those results were generated, who—or what—contributed to them, and whether the resulting knowledge is scientifically valid, reproducible, and economically justified.
Research Status and References
This article presents an independent methodological proposal. The proposed experiment has not been conducted or preregistered. Its hypotheses, evaluation framework, and expected comparisons do not constitute empirical findings.
Primary publication
Roberts, K. F., Abrams, Z. B., et al. (2026). An open agentic AI co-scientist for clinical and biomedical discovery. Nature Medicine.
https://doi.org/10.1038/s41591-026-04808-6
OpenScientist source code
https://github.com/openscientist-io/openscientist
Supplementary research materials
https://doi.org/10.5281/zenodo.18852838
BBIU — Strategic Research & Independent Analysis