In his role as a merchant he was said to own a third share in slave estates in Grenada, using profits from the slave trade to build up his art collection (and also benefitting from Lloyds' underwriting of the slave trade).
His collection of paintings, consisting of about forty of the most exquisite specimens of the art, purchased by the British government, after his death, formed the nucleus of the National Gallery.
He lived for some years in Greenwich in south-east London, building a house, Woodlands, in 1774 in an area now known as Westcombe Park, part of a wide area on the north-eastern fringes of Blackheath that he sought to enclose in 1801.
In 1972, his house, now owned by the local council, the London Borough of Greenwich, opened - appropriately enough - as an art gallery, the Woodlands Art Gallery. (In late 2003, there were fears that the Gallery might have to close; the council was seeking new funding for the adjacent Mycenae House and proposals included closure of the Gallery and redevelopment of the building.)
An active churchgoer, Angerstein worshipped in Greenwich town centre at St Alfege's Church - where he was also churchwarden.
His connections with the borough are still remembered. Angerstein Lane, near the heath at Blackheath, bears his name. A public house, The Angerstein, is on Woolwich Road, close to the Woolwich flyover (Blackwall Tunnel A102(M) southern approach) - on the opposite side of which lies the Angerstein Business Park (owned by Greenwich Enterprise Board). Just behind this, is the 'Angerstein Railway Line' (in 2003 believed to be only used for commercial freight, mainly aggregate transport) linking the peninsula at north Greenwich with the main railway network; as a result, an area of largely industrial land in between the lines to the east of the A102(M) is still sometimes referred to as the 'Angerstein Triangle'.