Born in Kremenetz in the Ukraine, he and his family moved to San Francisco when he was a year old. He had his first music lessons from his mother, and entered the San Francisco Conservatory at a very early age in 1928. There he studied the violin. He was proud to have been the student of Nahum Blinder. His public debut came on February 18 1936, when he played the Violin Concerto No. 3 by Camille Saint-Saëns with the San Francisco Symphony conducted by Pierre Monteux.
Stern was famous for his great recordings and his championing of younger players (among his discoveries were Yo-Yo Ma, Itzhak Perlman, and Pinchas Zukerman).
He recorded concertos by Brahms, Beethoven and Mendelssohn among others, as well as more modern works by Samuel Barber, Bela Bartok, Igor Stravinsky, and Leonard Bernstein. He also dubbed several actors pretending to play the violin in films, Fiddler on the Roof being one example of his work.