The village name is originally Brythonic and means simply "hill": the village stands on a particular promentory of the Chiltern Hills and from the tower of the Holy Trinity church in the village it is supposedly possible to see into not less than eight other counties. There is also a beacon hill with a signal post on it in the village boundary.
Segraves Manor, the principle manor in Penn anciently belonged to the Penn family. Sybil Penn, wife of Daniel, was nursemaid to King Edward VI of England and to Queen Elizabeth I of England. William Penn (after whom Pennsylvania is named) is also a descendant of this family. However in 1735 the manor passed from the unmarried Roger Penn to his only heir and sister who was married to Lord Curzon.
The hamlet of Penn Street lies within the village boundary about a mile from the main village, and refers to the road that leads to Penn from nearby Amersham.
Today the area is a very beautiful part of the Chiltern Hills, and is very popular with London executives due to its close proximity to road links into the city.