An empirical investigation of conventional ranking versus best worst scaling generated preferences for attributes of quality of life: one and the same or differentiable?

Aims: To investigate the degree of consistency in the rank ordering of a series of quality of life attributes generated via successive best worst (a form of ranking) and conventional ranking methods of data collection. Methods: A web-based survey was developed for administration to two general population based samples comprising younger people (aged 18 to 64 years) and older people (aged 65 years and above). Conventional ranking and best worst choice tasks were administered to the same respondent. For the ranking task, respondents were instructed to drag and drop 12 quality of life attributes in order of their relative importance in determining their overall quality of life. For the best worst task, respondents were presented with a series of successive best worst choice questions involving an identical setof quality of life attributes. The Swait-Louviere test was applied to determine the poolability of data by sample and/or method of data collection. Data were analysed in STATA using heteroskedastic conditional logit, mixed logit and generalised multinominal logit (GNML) models. Results: Target sample sizes of N=500 younger people (39% of those initially approached) and N=500 older people (60% of those initially approached) were achieved. In all instances the X2 statistics from the Swait-Louviere test were higher than the critical value of 18.310, rejecting the poolability of data by sample and/or method of data collection. For the total combined sample, ranking exhibited more consistent responses than best worst as exhibited by higher scale and lower error variance (tau = 0.529, p <= 0.01). In general, older respondents exhibited more consistent responses (tau = 0.515, p <= 0.01 ) than younger respondents for ranking but not for best worst. Conclusions: Whilst our findings indicate broad agreement overall, some inconsistencies are evident highlighting that these two methods of data collection may not be interchangeable.