Applying the Digital Health Literacy Instrument to Diversify Usability Evaluation Feedback: A Strategy to Support the Design of Truly Inclusive Health Interfaces

In small usability testing samples, developers face difficulties balancing convenience with recruiting a representative sample, including digitally excluded individuals who face online barriers. Biased recruitment and inadvertent exclusion can leave feedback lacking in types crucial to shaping interfaces for all users across the digital divide. An alternative strategy applying the Digital Health Literacy Instrument (DHLI) to stratify recruitment was explored. The impact of participants’ digital health literacy (DHL) level on performance and suitability for various usability evaluation methods was assessed. Six participants were purposively selected from a survey cohort (N = 28, MedianDHLI = 3.118). Based on their assessed DHL levels, three were placed into high (H-DHLI, MH-DHLI = 3.762) or low (L-DHLI, ML-DHLI = 2.762) groups and evaluated a palliative care interface. Sessions were remotely moderated with observations, performance, and narratives measured across five methodologies. Results were analysed with DHLI group outcomes compared. Findings suggest that DHL-stratified pre-selection of participants for evaluations could generate diverse usability feedback reflecting their digital competencies. Ease of activity completion was perceived similarly across the sample. H-DHLI individuals outperformed L-DHLI members in target success and task completion rates. Participants with lower DHLI experienced setbacks when initiating tasks, struggling with complex interactions, increasing the likelihood of task failure.

When applied, DHL stratification was a successful mechanism to potentially improve practice efficiencies whilst diversifying usability feedback generated by participants. Feedback could optimise interfaces, particularly where digital experience, literacies, or capabilities impact success. However, caution is warranted. DHL influences feedback acuity; discretion is required to align participant DHL levels with methodology selection and feedback type.