Sweatman started out playing violin, then switched to clarinet. He toured with circus bands in the late 1890s, and developed a famous act of playing three clarinets at once. He spent time playing with the bands of W.C. Handy and Mahara's Minstrels. He led a dance band in Minneapolis, Minnesota in 1902, where he made his first recordings on (now lost) phonograph cylinders that year or the following one. He wrote a number of rags, Down Home Rag being the most commercially successful. Sweatman moved to New York in 1913, where he became close friends with Scott Joplin, and Joplin named Sweatman as executor of his estate in his will. Sweatman enjoyed popularity with both White and Black audiences in New York, and started issuing recordings in 1916 for Emerson Records, then for Pathé.
After the commercial success of the Original Dixieland Jass Band the following year, Sweatman changed the sound and instrumentation of his band along the line of the early New Orleans jazz bands such as the Original Creole Orchestra and the Original Dixieland Jass Band. Sweatman was the first African American to make recordings labeled as "Jass" and "Jazz". (Since Sweatman can be heard making melodic variations even in his 1916 recordings, it might be argued that Sweatman recorded an archaic type of jazz earlier than the Original Dixieland band.) Sweatman's was the leading jazz band for Columbia Records until his popularity was surpassed by that of Ted Lewis.
Sweatman opened the well known Harlem club Connie's Inn in 1923. He continued playing in New York through the early 1940s, then concentrated his efforts on the music publishing business.