Wheeler was born in Malone, Franklin County, New York, and attended the University of Vermont. He was admitted to the bar in 1845 and practiced in Malone. He was district attorney for Franklin County from 1846 to 1849. He was a member of the state Assembly in 1850 and 1851 and a member of the state Senate from 1858 to 1860. He was elected as a Republican to the Thirty-seventh Congress (March 4, 1861 - March 3, 1863). He was a delegate to the state constitutional conventions in 1867 and 1868, and was elected to the Forty-first and to the three succeeding Congresses (March 4, 1869 - March 3, 1877).
Wheeler was not a candidate for reelection, having been nominated in 1876 as the Republican candidate for Vice President. He was elected on the Republican ticket with Rutherford B. Hayes in 1876. He was inaugurated in March 1877 and served until March 1881. He retired from public life and active business pursuits because of ill health, and died in Malone. He was intered in Morningside Cemetery.