He was born in Mount Morris, New York and gained his education in Illinois at Wheaton College and Oberlin College. Powell had a deep interest in the natural sciences. In 1856 he rowed the Mississippi River from St. Anthony to the sea, in 1857 he rowed the Ohio River and in 1858 the Illinois River. He was elected to the Illinois Natural History Society in 1859.
During the Civil War he first served with the 20th Illinois Volunteers. Despite losing much of his right arm at the Battle of Shiloh he returned to the army and was present at Champion Hill and Black River Bridge. Further medical attention to his arm did little to slow him, he was made a major and served as chief of artillery with the 17th Army Corps. In 1862 he married Emma Dean.
After leaving the army he took the post of professor of geology at the Illinois Wesleyan University. He also lectured at Illinois Normal University and became curator of the museum there.
From 1867 he led a series of expeditions into the Rocky Mountains and around the Green and Colorado rivers. In 1869 he set out to explore the Grand Canyon. He gathered nine men, four boats and food for ten months and set out from Green River, Wyoming on May 24. Passing through dangerous rapids the group passed down the Green River into the Grand River and Utah. One man quit after the first month and another three in the third, only days before the group reached the mouth of the Virgin River on August 30, after traversing almost 1,500 km. Powell retraced the route in 1871 with another expedition, producing an accurate map and various papers.
In 1881 he became the second director of the US Geological Survey, a post he held until 1894. He was also the head of the Bureau of Ethnology at the Smithsonian Institution until his death. Powell was buried in Arlington National Cemetery.