On March 4, 1767 the Duke of Cumberland allegedly married Olive Wilmot (later Mrs Payne), a commoner, in a secret ceremony. There reportedly was one child, Olivia Wilmot (1772-1834) from this relationship, though the duke's parenthood was never proven. A landscape painter and novelist, Olivia Wilmot married John Thomas Serres, 1759-1825, and later, controversially, assumed the style of Princess Olivia of Cumberland.
The Duke's marriage to the commoner Lady Anne Horton (or Houghton) (1743-1808) on October 2, 1771 was the catalyst for the Royal Marriages Act of 1772, which forbids any descendant of George II to marry without the monarch's permission. There were no children from this marriage. Lady Anne, though from a good family -- she was a daughter of Simon Luttrell, Earl of Carhampton, and the widow of Christopher Horton of Catton Hall -- seems to have been rather loose with her favors, given one wag's comment that she was "the Duke of Grafton's Mrs Houghton, the Duke of Dorset's Mrs Houghton, everyone's Mrs Houghton."