In 1209, Ranulph, Earl of Chester granted the manor of Bretby to Stephen de Segrave who built a church and a mansion there. There was also Bretby Castle which was destroyed during the reign of King James I of England to make way for the construction of Bretby Hall.
In 1585, Thomas Stanhope bought the manor house which was known as Bretby Hall and from then on was the home of the Earls of Chesterfield.