Kempe was born in Dresden where he studied music. He played the oboe in a number of orchestras before becoming conductor of the Leipzig Opera in 1935. He directed the Dresden Staatskapelle from 1949 to 1952, and the Bavarian Staatsoper in Munich from 1952 to 1954 (succeeding Georg Solti). His debut at the Bayreuth Festspielhaus was in 1960. From 1961 to 1975 he was conductor of the Royal Philharmonic Orchestra, from 1965 to 1972 worked with the Zürich Tonhalle Orchestra, and from 1967 to his death conducted the Munich Philharmonic. In the final months of his life, he was closely associated with the BBC Symphony Orchestra. He died in Zürich.
He is particularly remembered for his performances and recordings of Richard Strauss (whose complete orchestral works he recorded) and Richard Wagner.