World-first platform to evaluate AI disease detection systems demonstrates safety for NHS use, study involving Kingston University finds

Researchers have developed the world’s first independent platform for rigorously testing whether commercial artificial intelligence (AI) algorithms are fit for NHS use to detect disease in a fair, equitable, transparent and trustworthy way – using diabetic eye disease as the first example.

The revolutionary platform was trialled in a study published in The Lancet Digital Health, led by Professor Alicja Rudnicka of City St George’s, University of London. The Kingston University team, including Professor Sarah Barman and Dr Jiri Fajtl, collaborated with experts from Moorfields Eye Hospital, University College London and Homerton Healthcare NHS Foundation Trust.

During the study, funded by the NHS Transformation Directorate and The Health Foundation, the platform was used to compare commercial AI algorithms designed to detect diabetic eye disease. The algorithms work by identifying signs of blood vessel damage at the back of the eye that may be invisible to human observers.

Currently, NHS AI selection primarily focuses on cost-effectiveness and matching human performance. However, there's a critical need for robust testing of algorithmic fairness across diverse populations and ethnicities, an oversight that can lead to health disparities and the care people receive. This new platform puts all AI companies on a level playing field, removing potential biases from vendor-led testing.

The researchers used the new platform to test eight AI algorithms using 1.2 million eye images from the North East London Diabetic Eye Screening Programme, evaluating more than 200,000 screening visits from White, Black and South Asian ethnic groups.

It found the AI systems were able to analyse each visit in between 240 milliseconds to 45 seconds, compared to up to 20 minutes for a trained human. The accuracy for identifying diabetic eye disease needing intervention ranged from 83.7 per cent to 98.7 per cent, while for the most advanced, sight-threatening disease, accuracy ranged from 95.8 per cent to 99.5 per cent – performing similar or better than human graders in a fraction of the time.

The algorithms also performed consistently well across different ethnicity groups – the first time this has been assessed.

The researchers believe this work paves the way for a centralised NHS AI infrastructure to host approved algorithms for the National Diabetic Eye Screening Programme. This would ensure consistent, equitable service delivery nationwide, eliminate redundant infrastructure setup costs, and speed up diagnosis for patients.

Professor Barman said the study will help to understand if results from AI algorithms are subject to bias. “This large-scale evaluation of the effectiveness of AI algorithms has allowed us to demonstrate how different algorithms perform across subgroups of the population. It also provides a clear approach that can be applied to other medical domains to help ensure that AI is fair and works well for everyone.”

Professor Rudnicka said the level of scrutiny placed on the AI systems was far higher than anything placed on human performance. "Our revolutionary platform delivers the world’s first fair, equitable and transparent evaluation of AI systems. We’ve shown that these AI systems are safe for use in the NHS by using enormous data sets, and most importantly, showing that they work well across different ethnicities and age groups."

The researchers hope the transparent evaluation approach established here could become the blueprint for assessing AI tools in other chronic diseases like cancer and heart disease, boosting public trust in healthcare AI adoption.

• Read the full study published in The Lancet Digital Health.