Harmar started his career during the American Revolutionary War, receiving a commission as a captain in 1775. He served under George Washington and Henry Lee during the war, and at its conclusion, by then a colonel, was chosen by Congress in 1784 to relay the ratified Treaty of Paris (1783) to commissioner Benjamin Franklin in Paris.
Harmar was promoted to brigadier general in 1787. In 1790, he was sent on expeditions against Native Americans and remaining British in the Northwest Territory. After a few initial military successes, his force of federal troops was defeated by a tribal coalition led by Michikinikwa, in an engagement known as the "Battle of Kekionga" or "Harmar's Defeat." Later, Harmar returned with a somewhat larger force and engaged the coalition, but fought to a draw. Consequently he was relieved of this command and replaced by General Arthur St. Clair. Harmar was subsequently court-martialed, at his own request, on various charges of negligence, and exonerated.
After his retirement from the Army in 1792, Harmar served as adjutant-general of Pennsylvania (1793-1799). He died near Philadelphia at his estate, "The Retreat."