Agee was born in Knoxville, Tennessee. He lost his father at the age of six. Much of his early education was at a boarding school for boys. He attended Phillips-Exeter Academy, where he edited the Monthly and Harvard University, where he was president of the Harvard Advocate.
After graduation, he wrote for Fortune and Time magazines. In 1934, he published his first volume of poetry, Permit Me Voyage, with a foreword by Archibald MacLeish. In the summer of 1936, he spent eight weeks with the photographer Walker Evans living among sharecroppers in Alabama. Although Fortune never published his article, the material became a book in 1941, Let Us Now Praise Famous Men.
In 1951, Agee suffered the first in a series of heart attacks, which ultimately claimed his life four years later, at the age of 45.