He attended Harvard and graduated in 1771. James then studied law at Oxford and traveled widely in Europe until 1775. When he got the news of the Battle of Lexington he returned home. He served in the state Assembly and on the council before attending the Massachusetts' constitutional convention in 1779 and 1780.
James devoted several years to scholarly pursuits, until he was appointed the United States minister to Spain in 1804. He arrived in Madrid in May, 1805. In March of 1806 he and John Armstrong of New York were named commissioners to negotiate boundaries and other issues with Spain. He returned home in 1808 when the talks in Paris ended without success.
When Bowdoin College was founded in Maine, he gave the new school 6,000 acres and $5,500. When he died, he also bequeathed his considerable library, papers, mineral collection, scientific apparatus, and art collection to the school. He died on October 11, 1811 on Naushon Island in Buzzard's Bay, Massachusetts.