Background So far limited evidence exist comparing the difference between urban and rural elder residents in relation to how living arrangements correlates to health-related quality of life (HRQOL) of the elderly. Objective This study aims to compare the HRQOL of the elderly with four living arrangements: living with spouse only (LS), living alone (LA), living with a spouse and adult children(LSC) and the single elderly living with adult children (SLC) in urban and rural areas of China. Methods The data were drawn from the 2013 wave of Chinese National Health Service Survey in Shaanxi Province, which included 11,729 elderly people. The Chinese version of the EQ-5D-3L questionnaire was used to measure the HRQOL. Tobit regression model and logistic regression models were employed to estimate the associations between living arrangements and the HRQOL of the elderly. Results The EQ-5D utility scores of the urban elderly with four different living arrangements (LS, LA, LSC and SLC) were 0.9141, 0.8392, 0.8176 and 0.9080, which were almost all higher than their rural counterparts. After controlling other confounding variables, tobit regression estimates showed that the EQ-5D utility scores of the single elderly either living alone or living with adult children were lower than the elderly living with a spouse in urban areas. In rural areas only the single elderly living with adult children were more disadvantaged. Additionally the logistic regression results showed living-alone elderly had worse psychological health and the single elderly living with adult children had worse physical health. Conclusion The findings suggest that the urban elderly have better HRQOL than the rural elderly and the elderly with different living arrangements in urban and rural area have different HRQOL. More attention should be given to the poor mental health of the elderly living alone and the worse physical health of the single elderly living with adult children.