Until the administrative boundary changes in 1974 Lathbury was in Buckinghamshire.
The village name is Anglo Saxon in origin, and means 'fortification built with laths or beams'. In the Domesday Book of 1086 the village was listed as Latesberie.
There is a grand manor in Lathbury, which in the medieval period belonged to the abbey in Lavendon (since 1272). Following the dissolution of the monasteries in 1547 the manor stayed in the hands of the Crown until the Restoration of the Monarchy, when it was given to Christ Church College in Oxford.
There was also once a grammar school in Lathbury, founded in the reign of Queen Elizabeth I, supported by Christ Church College. The school was pulled down, however, in 1698 and the materials thereof used to repair the ancient rectory.
The parish church in Lathbury is dedicated to All Saints.