We demonstrate that voltage losses due to both radiative and non-radiative recombination of charge carriers are strongly dependent on D/A phase separation. By processing the active layer with various solvent additives, we create distinct morphologies that lead to significantly different device open-circuit voltages (VOC), even though the charge transfer state energy (ECT) of the D/A blend remains rather constant. We find that radiative recombination losses are significantly increased for a finely intermixed morphology, due to the large D/A interface area. This leads to a total recombination loss of ECT - qVOC ≈ 0.7 eV. However, considerably smaller losses (0.5 eV), due to suppressed non-radiative recombination, are possible in solar cells where the D/A materials are organized to only allow for selective charge carrier extraction. Using a drift diffusion model, we show that the origin of the reduced non-radiative recombination losses is related to an effect which has not been considered for 'optimized' solar cells-the suppression of minority carrier diffusion to the 'wrong' contact. Our results suggest that the built-in field is not sufficiently strong even in 'optimized' organic solar cells and that selective carrier extraction is critical for further improvements in VOC.