It was owned by Sir Walter Raleigh, then by Richard Boyle, Earl of Cork, and another Richard Boyle, the 3rd Earl of Burlington. When his daughter, Lady Charlotte Boyle (1731-1754), married William Cavendish of Chatsworth, Derbyshire the Castle came into the possession of the Dukes of Devonshire, along with other properties (Chiswick House, Bolton Abbey and others).
Edmund Spenser wrote The Faerie Queen at Lismore.