Mehta was born in Bombay, the son of the violinist Mehli Mehta who founded and conducted the Bombay Symphony Orchestra. Zubin initially studied medicine before studying music in Vienna and made his conducting debut there in 1958. The same year he won the International Conducting Competition in Liverpool which led to an appointment as assistant conductor of the Royal Liverpool Philharmonic Orchestra.
He was musical director of the Montreal Symphony Orchestra from 1960 to 1967, of the Los Angeles Philharmonic Orchestra from 1962 to 1978, and of the New York Philharmonic Orchestra from 1978 to 1991, making him the longest ever holder of the post. He has conducted the Israel Philharmonic Orchestra since around 1970, being made their music director for life in 1981. Since 1998 he is Music Director of the Bavarian State Opera, Munich.