When the Númenóreans returned to the coasts of Middle-earth in the Second Age, they found Mannish peoples which spoke languages which were distantly related to the Númenórean tongue Adûnaic, and the scholars declared that this was because these Men were descendants of the fathers of the Edain, the Atanatari, which had not crossed the Ered Luin in the First Age and gone to Beleriand. The Númenóreans set up friendly relations with them, and declared them to be Middle Men, as opposed to the High Men (the Edain) or the Men of the Shadow, meaning those Men that were hostile to Númenor.
After the founding of the Realms in Exile, Arnor and Gondor, many Middle Men became subjects of the Dúnedain, and eventually intermarried with them until they became one people.
The Northmen of Rhovanion were counted as Middle Men, as were most people living in Eriador. During the Third Age the term Middle Men was still applied to the Men of Bree (Middle-earth), and the remaining Northmen, such as the Men of Dale (Middle-earth) and Esgaroth the Lake-town, and of course the Rohirrim. King Valacar of Gondor became so friendly with the Middle Men of Rhovanion that he married Vidumavi, a princess of the Northmen, and his son Eldacar was of mixed blood. This lead to the disastrous Kinstrife of Gondor.
Most Middle Men were related to the Bëorian or Marachian Houses of the Edain, but there were also Men in Middle-earth related to the Second House, the Haladin: these were the Men of Enedwaith and Minhiriath, later to be known as the Dunlendings. Because they spoke an alien tongue to the Númenóreans they were not at first recognised as Middle Men, and became hostile to the Númenóreans: a hostility which would endure on to the end of the Third Age.