PROLIFERATE: A Tool to Measure Impact and Usability of AI-Powered Technologies

Problem: Health researchers and professionals of several scientific disciplines appreciate, recognise, and are expected to demonstrate the importance and impact of their performance. Founding bodies and sponsors of health research are also requesting that research programs provide evidence and measurements of their effectiveness and impact.

Measuring research impact and effectiveness can support the creation of sustainable responses to systemic or complex healthcare and societal problems. Developing standardised ways to understand and track research impact and effect is a fundamental part of any investigation; particularly within research projects or technological studies that claim to be human-centred, involve health consumers, different stakeholders, end-users or citizens as partners in research (participatory research). However, a lack of standardised and generalisable metrics for evaluating research has been recognised as an important gap for academics and practitioners.

Approach: We created and tested a mixed methods participatory evaluation tool denominated “PROLIFERATE”. In this presentation, we will explore the process of automatizing the tool, as a computerised technique that can measure, track, and predict, via cutting-edge Bayesian methods, the effectiveness and impact of participatory research projects and their products or outcomes so that these align with end-users’ needs. Besides the easiness of probabilistic interpretation allowed by the Bayesian framework, PROLIFERATE also features a Bayesian approach to extract knowledge from users and translate it in a probabilistic format suitable for prediction modelling.

Status: The PROLIFERATE tool handles the complexity of evaluation and simplifies it via its new digitised testing, evaluation, and optimisation implementation. A previous version of PROLIFERATE that has been pilot-tested is used as the foundation on which the new digitalised version was conceived. The digitalised modification of PROLIFERATE has received ethics approval (2020/HRE00964) as an ‘End-user evaluation and optimisation plan’ to test/evaluate RAPIDx AI, an artificial intelligence-based decision support health tool, for accurate diagnosis of heart attacks in hospitals.

Conclusion: Participatory research efforts are recognised as excellent accountability and optimization strategies concerning end-user involvement in research and technology development. Based on that principle, we described the new digitalised PROLIFERATE tool and explain how it can measure, track, and predict the impact and effect of AI-powered research technologies/products/programs from the perspective of end-users.

We have automatised and standardised a way to assess the comprehensibility, emotional resonance, motivation to change, barriers and future accessibility of research programmes and products according to the perspectives of end-users. We use the AI-powered technology RAPIDx AI to exemplify how PROLIFERATE works.

Learning objectives:
1. Explore the process of automatizing a tool, as a computerised technique that can measure, track, and predict, via cutting-edge Bayesian methods, the effectiveness and impact of participatory research.
2. Feature a Bayesian approach to extract knowledge from users and translate it in a probabilistic format suitable for prediction modelling.
3. Explain how to assess, according to the perspectives of end-users, the comprehensibility, emotional resonance, motivation to change, barriers and future accessibility of an artificial intelligence-based decision-support health-tool, for accurate diagnosis of heart attacks in hospitals.