When Carter was a youth, he lived in Harlem around the corner from Bubber Miley, Duke Ellington's star trumpeter. Carter was inspired by Miley and bought a trumpet. When he found he couldn't play like Miley, he traded the trumpet in for a saxophone.
Carter began performing professionally at the age fifteen. He first recorded in 1927 and organized his first big band the following year. He performed with Fletcher Henderson in 1930 and 1931, then briefly led McKinney's Cotton Pickers before returning to leading his own band in 1932. He also arranged for Henderson and Duke Ellington during these years and wrote two hits, "Blues in My Heart" and "When Lights are Low." By the early 1930s he and Johnny Hodges were considered the leading alto players of the day.
In 1935 he moved to Europe, where he became staff arranger for the British Broadcasting Corporation dance orchestra and made several records. He returned to the United States in 1938 and led a big band and sextet before moving to Los Angeles in 1943 to write for movie studios. Carter continued writing and performing into his 90s. He arranged for Louis Armstrong, Ray Charles, Ella Fitzgerald, Peggy Lee, and Sarah Vaughan, among many others.
His biggest hit was "Cow Cow Boogie," a song he co-wrote with Don Raye and Gene DePaul, which was a hit for Ella Mae Morse in 1942.
In the 1940s and 50s, Carter was one of the first black men to compose music for movies. He was an inspiration and a mentor for Quincy Jones when Jones began writing for television and movies in the 1960s.
He also appears uncredited in the 1952 film, the Snows of Kilimanjaro, as a tenor sax player.
Carter was admired for his ability to write saxophone soli, where the entire section plays as one unit.
Carter was a member of the music advisory panel of the National Endowment for the Arts. He was also a member of the Black Film Makers' Hall of Fame and in 1980 received the Golden Score award of the American Society of Music Arrangers.
He died, aged 95, at Cedars Sinai Hospital, Los Angeles of what is thought to have been bronchitis.