The village name is Anglo Saxon in origin, and means 'hill where asses are pastured'. In the Domesday Book of 1086 it was recorded as Esenberge.
The parish church, dedicated to St Peter and St Paul, stands separate from the rest of the village, high on the hill overlooking the rest of Ellesborough. When the Prime Minister is in residence at Chequers, this would normally be the church where he or she would go for Sunday morning worship, however Tony Blair caused outrage locally by electing to go instead to a church in Great Missenden.