After playing football in his youth, Hary switched to sprinting at age 16. Only a few years later, in 1958, he won his first international titles when he came first in the 100 m and the 4 x 100 m at the European Championships.
That same year, Hary appeared to have run a new World Record with a time of 10.0 seconds, but the track was found to be too short. In 1960, another try at the World Record seemed to have failed again, after judges believed Hary's start of another 10.0 race had been false - Hary was well-known for his fast starts. In a re-run at the same evening, Hary finally set the World Record.
That same year, at the Olympics, he achieved his greatest moment of fame. After a nerve-racking number of near-starts, Hary sprinted to the gold in 10.2 seconds to become the first non-American winner of the event since Canadian Percy Williams.
In the final of the 4 x 100 m, Hary and his teammates appeared to have finished second behind the team of the United States, but 15 minutes after the final it was announced that the USA had been disqualified for an incorrect exchange. Germany's time, 39.5 equalled their own World Record.
After the Olympics, Armin Hary was suspended by the German federation, and Hary retired from the sport. In 1980, he was found guilty of a 3.2 million swindle, and served a prison sentence until 1983.