Purpose: Obstructive sleep apnea (OSA) is now highly prevalent but largely undiagnosed. Quality of life is an indicator of both the impact of undiagnosed OSA and the need for strategies to increase OSA diagnosis. We determined age-related impacts of undiagnosed OSA on health-related quality of life (HRQL) and whether this was independent of sleepiness and comorbidities. Methods: In 2010–2012, 837 participants from the Men Androgen Inflammation Lifestyle Environment and Stress Study (population cohort n = 1869, ≥40 years, Adelaide, Australia), without a prior OSA diagnosis underwent full in-home polysomnography (Embletta X100) and completed the Epworth Sleepiness Scale and SF-36 questionnaire. The effects of the apnea-hypopnea index (AHI) on SF-36 physical (PCS) and mental (MCS) component summary scores and standardized SF-36 scale z-scores were estimated using multiple linear regression adjusted for major comorbidities and sleepiness, stratified by age. Results: Men ≤69 years demonstrated significant (p < 0.05) decrements/event increase in AHI in PCS score [unstandardized B coefficient (SE) = −0.068 (0.023)], physical functioning, role physical, general health, and vitality z-scores in fully adjusted models. Severe OSA (AHI ≥30) was associated with significant reductions in PCS [B = −4.1 (1.1)] and MCS score [B = −3.6 (1.2)] independent of sleepiness and comorbidities which were attenuated but persisted in men <69 years without depression. In men aged ≥70 years, statistically significant AHI-associated impairments were generally not seen. Conclusions: Undiagnosed OSA was a major independent contributor to HRQL impairments in men <69 years. Improved strategies to identify undiagnosed OSA are indicated that may require a reduced focus on daytime sleepiness.