Born in Babenhausen, Jochum studied the piano and organ in Augsburg, and took further studies in Munich. His early conducting posts including music director at the Berlin Radio, and condcutor of the Hamburg Opera, where he performed music by composers such as Paul Hindemith and Béla Bartók elsewhere banned by the Nazis. He later formed the Bavarian Radio Symphony Orchestra and conducted the Concertgebouw Orchestra in Amsterdam. In 1975 he was appointed conductor laureate of the London Symphony Orchestra.
Jochum is particularly remembered for his interpretations of Anton Bruckner.