The hamlet lies largely within the boundary of the house's grounds, and houses the servants and other staff of the house.
Prior to the Norman conquest there was an abbey at Ascott, that had been given by Empress Maud to a Benedictine convent in Angiers .In 1415 however, the same year as the Battle of Agincourt, the convent was seized by the English church because it belonged to the French and awarded to the Convent of St Mary du Pre, near St Albans.
In the early Sixteenth century the abbey (along with the manor in Wing) was seized by the Crown and given to Cardinal Wolsey, however not long after it was seized once again in the dissolution of the monasteries and given to Lord Dormer. It was in 1554 that Dormer entertained Princess Elizabeth at the house, when she was on the road to London under arrest as a Protestant because her sister Mary had just taken the throne.
The house itself once featured many additions that were built by Inigo Jones. However the house fell into decay following the death without heir of its owner the Earl of Caernarvon in 1709. In 1727 the house was broken up, the deer sold and all the timber cut down and sold off. The modern house is a replacement of the ancient abbey, and in more recent times was the home of the Rothschild family, who beautified the house such that it is currently owned by the National Trust.