He was born in Lancaster, Ohio, and was the younger brother of the American Civil War general William Tecumseh Sherman. His father became a judge of the Ohio Supreme Court about the time he was born, but died when John was only six years old.
Before entering politics, he practiced law, studying under an older brother and joining him as a partner in 1844. In 1848 he married a judge's daughter, Margaret Sarah Stewart. In 1848 he served as a delegate to the Whig convention that nominated Zachary Taylor for President. In 1852 he again served as a convention delegate, at the Baltimore convention that nominated Winfield Scott. He served in the House of Representatives from 1855 to 1861.
He was first elected to a term in the Senate beginning 1861, and served until 1877, when he resigned to join the Cabinet.
He served as Secretary of the Treasury in the cabinet of President Rutherford B. Hayes from 1877 to 1881.
He sought the Republican nomination for the Presidency in 1880, hoping to become a compromise candidate between Ulysses S. Grant, who was being promoted for a third term, and James G. Blaine. However, his campaign manager, Representative and Senator-elect James A. Garfield, received the nomination instead. Sherman returned to the Senate, taking the seat to which the Ohio legislature had originally elected Garfield, replacing Allen G. Thurman, in 1881, serving until 1897, when he resigned once more to join the Cabinet.
President William McKinley appointed Sherman as his Secretary of State on assuming office, and Sherman served in that position from 1897 to 1898.
He died in Washington, D.C.
Sherman is best known for the Sherman Antitrust Act, of which he was the chief author.