It was the family seat of the Bankes family (until the 17th century, they had resided at Corfe Castle but this was destroyed in the English Civil War). The original building - designed by Sir Roger Pratt (c.1663) with interiors designed by Inigo Jones but executed by his heir John Webb - was extensively remodelled by Sir Charles Barry during the 19th century.