Sherman was born in Utica, New York and graduated from Hamilton College in 1878. He was admitted to the bar in 1880 and commenced practice in Utica. He was president of the Utica Trust & Deposit Co. and of the New Hartford Canning Co., then mayor of Utica in 1884. He was elected as a Republican to the Fiftieth and Fifty-first Congresses (March 4, 1887-March 3, 1891). He was an unsuccessful candidate for reelection in 1890, but was elected to the Fifty-third and to the seven succeeding Congresses (March 4, 1893-March 3, 1909) He was not a candidate for reelection, having been nominated as the Republican candidate for Vice President on the ticket with William Howard Taft. He was elected Vice President in 1908 and served from March 4, 1909, until his death. He had been renominated for Vice President in June 1912, but died in Utica, Oneida County, New York. He was intered in Forest Hill Cemetery.