According to legend, a wooden castle was constructed on the site in the 8th century by a Saxon lord called Dud or Dado. A castle was recorded at the time of the Domesday Book of 1086.
Much of the castle which can be seen today dates from the 12th and 13th centuries. It became obsolete as a fortress after the English Civil War, and was destroyed by fire in 1750.
The castle was considered as a possible place of imprisonment for Mary Queen of Scots.