He was born in Orel and studied at the conservatory in Moscow. His public debut came at the age of 14 in a performance of Ludwig van Beethoven's Piano Concerto No. 5 (the Emperor) in a performance conducted by Anton Rubinstein.
In 1898 he married Rosina Bessie, also a pianist, and the two gave a number of concerts together. They later moved to Berlin. Josef gained a reputation as one of the leading virtuosi of his day.
In 1919 the couple moved to New York City in the United States of America, where Josef continued his concert career and taught piano at the Juilliard School (Rosina also taught there, with Van Cliburn among her pupils).
Josef Lhévinne died in New York City.