The village name is Anglo Saxon in origin, and means 'Ylla's boundary', referring to the ancient boundary with Oxfordshire. In the Domesday Book of 1086 the village was recorded as Imere.
In the reign of King James I of England, Ilmer was home to the master falconer of the king.
The family name Elmer, common in this part of the country, comes from this village.