Background Although several longitudinal studies indicate that weight loss precedes dementia in men and women, the relationship between weight changes and cognitive performance is unclear. This study investigated the relationship between changes in adiposity and cognitive function in community-dwelling women. Methods Data were derived from the Longitudinal Assessment of Women Study, a population-based study of 511 urban women initially aged 40-79 years. We analyzed data from 334 women who had complete information on demographics, cardiovascular risk factors, medical status, weight, height, and waist-to-hip ratio and cognitive scores at baseline and after a mean of 7.45 years of follow-up. Cognition was assessed at baseline and follow-up using the Mini Mental State Examination; the Auditory Delayed Index, Visual Delayed Index, and Working Memory Index from the Wechsler Memory Scale, Third Edition; and the Processing Speed Index from the Wechsler Adult Intelligence Scale, Third Edition. Associations were adjusted for age, education, baseline cognitive performance, cardiovascular risk factors, menopausal status, and apolipoprotein E-4 status. Results In multivariate analysis, both weight gain and loss were associated with poor Visual Delayed Index performance at follow-up compared with stable weight (β = -4.02 ± 1.57, p = 0.011; β = -6.50 ± 2.39, p = 0.007, respectively). No significant associations were found between body mass index, waist circumference, or waist-to-hip ratio and any cognitive domains at follow-up. Changes in cognitive performance were not associated with changes in adiposity measures. Conclusion Weight loss and weight gain were associated with poor cognitive performance in middle-aged and older women compared with women with stable weight.