Born in White Plains, New York, he graduated from Yale University before serving in the United States Marine Corps during World War II. Following the war, he appeared on Broadway, where he won a special Tony Award in 1948. His first major movie was Battleground, for which he was nominated for the Academy Award for Best Supporting Actor. Other major films included The Asphalt Jungle, The Next Voice You Hear, Mrs. O'Malley and Mr. Malone, Kiss Me, Kate, Them, Battle Cry, Black Like Me and Give 'em Hell, Harry, for which he was nominated for the Academy Award for Best Actor for his portrayal of former President of the United States Harry S Truman.
He is the father of actor James Whitmore Jr and the grandfather of actor James Whitmore III.
Whitmore has a star on the Hollywood Walk of Fame at 6611 Hollywood Blvd.