He was the third son of the George Venables-Vernon, 1st Lord Vernon (1710-1780), and took the name of Harcourt alone instead of Vernon on succeeding to the property of his cousin, the last Earl Harcourt, in 1831.
One of his younger sons was William Vernon Harcourt, the founder of the British Association for the Advancement of Science.Another,Admiral Frederick Harcourt,is the ancestor of today's Lord Vernon.