He married Elizabeth Albana (d. 1844), daughter of Clotworthy, 1st Baron Templetown, by whom he had six sons and three daughters. In 1826 he was created marquess of Bristol and Earl Jermyn. He was succeeded by his son Frederick William (1800-1864), M.P. for Bury St Edmunds 1830-1859, as 2nd marquess; and by the latter's son Frederick William John (1834-1907), M.P. for West Suffolk 1859-1864, as 3rd marquess. The latter's nephew, Frederick William Fane Hervey (b. 1863), who succeeded as 4th marquess, served with distinction in the royal navy, and was M.P. for Bury St Edmunds from 1906 to 1907.
{| border="2" align="center"
|-
|width="30%" align="center"|Preceded by:
New Creation
|width="40%" align="center"|Marquess of Bristol
|width="30%" align="center" rowspan="2"|Followed by:
Frederick William Hervey
|-
|width="30%" align="center"|Preceded by:
Frederick Augustus Hervey
|width="40%" align="center"|Earl of Bristol
|}
Reference