The origin of sterile worker castes, resulting in eusociality, represents one of the major evolutionary transitions in the history of life. Understanding how eusociality has evolved is therefore an important issue for understanding life on earth. Here we show that in the large bee subfamily Xylocopinae, a simple form of sociality was present in the ancestral lineage and there have been at least four reversions to purely solitary nesting. The ancestral form of sociality did not involve morphological worker castes and maximum colony sizes were very small. True worker castes, entailing a life-time commitment to non-reproductive roles, have evolved only twice, and only one of these resulted in discrete queen-worker morphologies. Our results indicate extremely high barriers to the evolution of eusociality. Its origins are likely to have required very unusual life-history and ecological circumstances, rather than the amount of time that selection can operate on more simple forms of sociality.