The village name is Anglo Saxon in origin, and means 'Haefer's homestead'. In the Domesday Book of 1086, when it belonged to the Peverell family, it was listed as Hæfærsham.
The ancient manor in the village, that was fortified in 1304 has largely been pulled down now, but parts of it still remain in a farm house just outside the main village.