Clearly, every abelian group has this property; but there are non-abelian examples as well. The most familiar is the quaternion group of order 8.
It can be shown that every Hamiltonian group is a direct sum of the form G = Q8 + B + D, where Q8 is the quaternion group of order 8, B is the direct sum of some number of copies of the cyclic group C2, and D is a periodic abelian group with all elements of odd order.