We examined how positive and negative social exchanges with friends, family, and spouses were related to cognitive aging in episodic and working memory, and perceptual speed. To do so, we used a large sample of cognitively intact young-old participants from the PATH Through Life Study (PATH; aged 60 to 64 years at baseline, n = 1,618) who were assessed on 3 occasions over 8 years. Additional replication analyses were conducted using the Health and Retirement Study (HRS), which provided data on episodic memory. The main analysis of PATH Through Life showed that positive exchanges with friends and family were associated with less decline in perceptual speed, with these associations attenuated by adjustment for physical functioning and depressive symptoms. Negative exchanges with spouses were associated with poorer working memory performance. Positive exchanges with friends were associated with better initial episodic memory in both PATH and HRS. More frequent negative exchanges with friends and family were associated with better episodic memory in the PATH sample. However, these findings were not replicated in HRS. Our findings provide indirect support for the role of social exchange quality in contributing to cognitive enrichment. However, the inconsistent pattern of results across cognitive and social exchange domains points to possibilities of reverse causality, and may also indicate that social exchange quality plays a less important role for cognitive enrichment than other psychosocial characteristics.