Mitropooulos was born in Athens and studied music there and in Brussels and Berlin, with Ferruccio Busoni among his teachers. From 1921 to 1925 he assisted Erich Kleiber at the Berlin State Opera, then took a number of posts in Greece. At a 1930 concert with the Berlin Philharmonic, he played the solo part of a piano concerto and conducted the orchestra from the keyboard, becoming one of the first modern musicians to do so.
Mitropoulos made his United States debut in 1936 with the Boston Symphony Orchestra, and he subsequently settled in the country, becoming a US citizen in 1946. From 1937 to 1949 he was principal conductor of the Minneapolis Symphony Orchestra, after which he worked with the New York Philharmonic Orchestra, from 1951 to 1957 as principal conductor. In 1954 he succeeded Bruno Walter at the Metropolitan Opera. He introduced many new works there, and gave the premiere of Samuel Barber's opera Vanessa (work he had helped the composer to orchestrate). He died in Milan while rehearsing Gustav Mahler's Symphony No. 3.
Mitropoulos was noted as a champion of modern music, such as that by the members of the Second Viennese School. He wrote a number of pieces himself for orchestra and solo works for piano, and also arranged some of Johann Sebastian Bach's organ works for the orchestra.