The son of a Polish cabinetmaker and his Irish wife, he was born in London, England, and died in Nether Wallop, Hampshire, England.
He married three times. His first wife was Lucie Hickenlooper (a.k.a. Olga Samaroff, former wife of Boris Loutzky), a Texas-born concert pianist and musicologist, to whom he was married from 1911 until 1923 (one daughter: Sonia Stokowski, an actress). His second wife was Johnson & Johnson heiress Evangeline Love Brewster Johnson, an artist and aviator, to whom he was married from 1926 until 1937 (two children: Gloria Luba Stokowski and Andrea Sadja Stokowski). His third wife, from 1945 until 1954, was railroad heiress Gloria Vanderbilt (born 1925), an artist and fashion designer (two sons, Christopher Stokowski and Leopold Stanislaus Stokowski). He also had an affair with Greta Garbo in 1937-8.