Webb was born in Baltimore, Maryland. He suffered from childhood tuberculosis, leaving him with short height and a badly deformed spine. He supported himself as a newspaper boy and saved up money to buy drums, and first played professionally at age 11. In 1925 he moved to New York City and by the following year was leading his own band in Harlem.
He alternated New York city residencies with band tours through 1931, when he became bandleader at Manhattan's Savoy Ballroom. He became one of the best regarded bandleaders and drummers of the new "Swing" style. In 1935 he added a young girl singer to his band who would become famous, Ella Fitzgerald.
In November of 1938 his health began to decline, and from then until his death he alternated time on the bandstand with time in hospitals.