Born in Barnesville, Ohio, Gray was brought up on a farm. He spent two years at Oberlin College where he worked with electricity. In 1867 he received a patent for an improved telegraph relay and he went on to receive patents in over 70 other inventions.
In 1872, Gray founded the Western Electric Manufacturing Company. In 1874, he retired and did independent research and taught at Oberlin.
Gray was a charter member of the Highland Park Presbyterian Church (which still exists) and gave the first public demonstration of his invention in its sanctuary in 1874. On February 14, 1876 he submitted an announcement to the patent office, but it turned out to be just two hours after Alexander Graham Bell did. Although Bell did not have a working prototype, and the device described in his patent did not work, after two years of litigation he was awarded rights to the invention. As a result, Bell is usually credited as the inventor.