Shepp was born in Fort Lauderdale, Florida, on May 24, 1937, but raised in Philadelphia, Pennsylvania, where he studied piano, clarinet and alto saxophone before focusing on tenor saxophone.
He studied drama at Goddard College from 1955 to 1959 before turning to music profesionally. He has recorded or performed with John Coltrane, Bill Dixon, Cecil Taylor and Phish, among others.