Scientific value, scarcity, and sensitivity
A central paradox of AI in science is that its highest impact applications depend on data that are simultaneously rare and ethically sensitive.
Effective healthcare models require extensive, longitudinal population-level data to capture the diversity of clinical trajectories. The scarcity of such data drives research programmes toward repositories like the United Kingdom’s National Health Service (NHS) patient records, which comprise millions of patient histories. Technical proponents argue that only national-scale datasets can provide sufficient diversity to achieve the precision that is required for reliable and meaningful forecasts, including for rare diseases and minority populations. The NHS England ‘Foresight’ project – involving the training of generative AI on approximately 57 million de-identified patient records – was promoted as a way to “improve patient outcomes” by anticipating clinical events before they occur, thus shifting healthcare from reactive to preventative.1
Notwithstanding the purported clinical value, population health data are scarce in a practical sense. Such data contain deeply personal information, gathered under expectations of clinical confidentiality and governance. The richer and more detailed the dataset, the harder it is to fully anonymise in a way that effectively eliminates the risk of re-identification. In the UK, regulatory authorities explicitly treat anonymisation not as a guarantee of acceptability, but as a risk-management judgement that depends on the complexity and context of the data, rather than as a binary status that per se resolves concerns.2
This combination of high utility and high ethical sensitivity means that the most scientifically valuable data also carry the greatest moral weight. This moral weight, however, does not arise from the intrinsic nature of the data alone: as Solove has argued, the regulatory focus on the nature of data itself becomes unworkable in an era of large-scale analytics, where seemingly innocuous data can generate sensitive inferences and real-world harms depending on context and use.3 From this perspective, ethical and legal evaluation cannot rely solely on whether data are formally classified as sensitive or anonymised. Instead, it needs to assess the risk and magnitude of harm arising from specific circumstances. In high-impact scientific applications, this means that even lawfully accessed and technically anonymised data may remain ethically weighty when the potential ‘downstream’ consequences are substantial.
Public consent, trust, legitimacy, and institutional debate
Once data scarcity motivates access to sensitive human data, the question of legitimacy emerges. Legitimacy, in this sense, involves (a) legal compliance, (b) the meaningful consent of data subjects or justified ethical waivers, and (c) public understanding and acceptance of how data are being repurposed.
The current UK health governance frameworks do not require explicit consent for all secondary uses of health data. Instead, they rely on procedural safeguards and trust in institutional governance.
The aforementioned ‘Foresight’ initiative has faced public and professional pushback precisely because of perceived gaps in legitimacy.4
In June 2025, concerns about the Foresight AI project were raised by professional bodies representing general practitioners through the Joint GP IT Committee. The committee asked NHS England to refer the matter to the UK Information Commissioner’s Office in order to clarify whether data protection and governance requirements had been met. NHS England subsequently agreed to pause the Foresight project while a review was conducted and while discussions with the Information Commissioner’s Office took place.5,6
This episode exposes an important, if not crucial, issue that is often overlooked in technical AI debates: clinical professionals and patient advocates are not always convinced that research governance matches patient expectations. In this particular case, professional bodies representing clinicians – who, in turn, play a fiduciary role with respect to patient trust – openly questioned whether governance processes and public expectations had been met in using GP records for model training.
Furthermore, there is little evidence to suggest that there is either a broad public consensus or active public engagement regarding the use of national health records to train large generative models.
The “AI” label and ethical confusion
A further ethical problem is posed by the use of “AI” as a catch-all label. The tendency to treat “AI” as a single, monolithic technology rather than a heterogeneous set of methods obscures its distinct uses and ethical profiles. When everything from simple rule-based decision support to generative models is labelled “AI”, ethical debates rapidly become muddled, and controversies from one domain (e.g., ‘hallucinations’ – fabricated outputs in large language models) are imported into others (e.g., clinical prediction) without proper qualification.
This semantic inflation has two harmful effects. First, it allows everyday controversies – for instance, debates about economy, markets, geopolitics, and the so-called “AI race” – to overshadow the precise ethical analysis of scientific data governance. Second, it conflates distinct ethical issues – such as algorithmic bias and informed consent – into a single “AI ethics” discussion that can neither be effectively operationalised nor mapped to practical governance levers. The NHS guidance on AI, for example, emphasises that AI systems may use personal data for training purposes, provided that the data are minimised and the confidentiality is ensured, but it does not differentiate between ethical expectations by technical subtype or societal impact of the specific model.7
Thus, the project of AI for science must disentangle the science of interest from the generic “AI” label. A predictive model trained on well-curated clinical data for epidemiological insight differs significantly, both ethically and technically, from a generative model trained on uncurated internet text. Conflating these under a single banner risks obscuring the very ethical trade-offs that ought to be subject to governance.
Conclusion
Addressing the ethical problems in AI for science requires simultaneous progress on three fronts:
1. Explicit governance of data use that acknowledges scarcity and sensitivity.
This includes transparent governance processes, a clear articulation of the lawful basis for processing personal data under the UK General Data Protection Regulation (GDPR), compliance with common law duties of confidentiality, and – if relevant – explicit statutory authorisation to set those duties aside.
2. Institutional legitimacy exceeding simple compliance – and grounded in participation.
Professional bodies, patients, and – not less importantly – the public must be involved in shaping the use of data. Being informed afterwards is a way to diminish not only public trust in bureaucracy, but also in technology itself, and, perhaps most importantly, in science and scientific institutions.
3. Conceptual disambiguation of “AI”.
Ethical analysis should clearly differentiate between types of systems (predictive, generative, rule-based clinical decision support, etc.) and reflect on specific ethical challenges into these categories.
Only with these steps taken can AI for science justify its reliance on data that are both scientifically valuable and ethically sensitive. Without this clarity, the field risks replicating broader controversies and undermining public trust.
Disclaimer
The views expressed in this article are those of the authors. Publication in Open Research Europe does not imply endorsement of the European Commission.
Data availability
No datasets were generated or analysed during the current study. This article is based on publicly available sources, which are cited within the reference list. Therefore, there are no data to deposit in a repository and no associated DOI or accession number.
Ethics and consent
Ethical approval and consent were not required for this study. The article is based exclusively on analysis of publicly available documents and does not involve human participants, personal data, or identifiable information.
Acknowledgments
The authors acknowledge the use of AI tools for assistance with formatting and language refinement.
References
- 1.
NHS England:
Foresight AI Case Study.2025.
Reference Source
- 2.
NHS England:
Proof-of-Concept: Using Anonymous or Artificial Health Data.2025.
Reference Source
- 3.
Solove DJ:
Data Is What Data Does: Regulating Based on Harm and Risk Instead of Sensitive Data.
Northwest. Univ. Law Rev.
2024; 118: 1081–1138.
Reference Source
- 4.
Wilkins A:Concerns Raised Over AI Trained on 57 Million NHS Medical Records.
New Sci.
2025.
Reference Source
- 5.
Armstrong S:
NHS England Faces Investigation Over Granting Foresight AI Access to GP Data.
BMJ.
2025; 389: r1192. Publisher Full Text
- 6.
Royal College of General Practitioners:
RCGP Raises Concerns Over Foresight AI Through Joint GP IT Committee.2025.
Reference Source
- 7.
NHS England:
Artificial Intelligence.2025.
Reference Source
