He was born in Paris to Chinese parents (with ancestry in Zhejiang), and had a musical upbringing. His mother, Marina Lu (盧雅文), was a singer, while his father, Hiao-Tsiun Ma (馬孝駿), was a conductor and composer. Ma began to study the violin, then the viola, before taking up the cello. His family moved to New York City, United States when he was seven, and he continues to live there.
Ma was a child prodigy, appearing on American television at the age of eight in a concert conducted by Leonard Bernstein. He entered the Juilliard School, and then went to Harvard, but was questioning whether he should continue his studies until, in the 1970s, Pablo Casals's performing inspired him.
Since that time, he has steadily gained in fame, and has performed with most of the world's major orchestras. His recordings and performances of Johann Sebastian Bach's suites for solo cello are particularly acclaimed, and he has also played a good deal of chamber music, often with the pianist Emanuel Ax. One of his cellos, the "Davidov Strad", was previously played regularly by Jacqueline Du Pre.
Ma married his long-time girlfriend, Jill Horner, a violinist, in 1977. They have two children, Nicholas and Emily. Ma's elder sister, Yeou-Cheng Ma (馬友乘), also born in Paris, is a violinist, married to Michael Dadap, a New Yorker guitarist.