Remsen was born in New York City and earned an MD from the College of Physicians and Surgeons in 1867 to please his parents. He then travelled to Germany studied chemistry - his true passion. He earned a PhD from University Göttingen. In 1875, after researching pure chemistry at University of Tübingen, Remsen returned to the United States and became a professor at Williams College, where he wrote the popular "Theoretical Chemistry". His book and reputation brought him to the attention of Daniel Coit Gilman who invited him to become one of the original faculty of Johns Hopkins University. He accepted and founded the department of chemistry there, where he ran his own laboratory. In 1879 he founded the American Chemical Journal which he edited for 35 years.
In 1879 he made the greatest discovery of his career by accident. When he ate rolls at dinner after a long day in the lab researching coal tar derivatives, he noticed that the rolls tasted initially sweet but then bitter. Since his wife tasted nothing strange about the rolls, Remsen tasted his fingers and noticed that the bitter taste was probably from one the chemicals in his lab. The next day at his lab tasted the chemicals that he had been working with the previous day and discovered that it was the oxidation of o-toluenesulfonamide he had tasted the previous evening. He named the substance saccharin and he and his research partner Constantine Fahlberg published their finding in 1880. Later Remsen became angry after Fahlberg patented saccharin, claiming that he had discovered saccharin.
In 1901 Remsen was appointed the president of Johns Hopkins and he proceded to found a School of Engineering and helped establish the school as a research university. He introduced many of the German laboratory techniques he had learned and wrote several important chemistry textbooks. In 1912 he stepped down as president and retired to Carmel, California. After his death the new chemistry building was named after him at Johns Hopkins. His ashes are located behind a plaque in Remsen Hall; he is the only person buried on campus. According to legend, undergraduates who rub the plaque the night before their chemistry exam will do well.