James Matthew Dillon was born in Dublin. He was the son of John Dillon, the last leader of the Irish Parliamentary Party (1918), which had been swept away by Sinn Féin in the 1918 general election. He was educated in Gorey, University College Galway and King's Inns. He qualified as a barrister and was called to the bar in 1931. Dillon studied business methods at Selfridge's in London. After some time at Marshall Fields in Chicago he returned to Ireland where he became manager of the family business in Roscommon.
Between 1932 and 1937 Dillon served as TD for West Donegal for the Centre Party and after its merger with Cumann na nGaedhael, the new party Fine Gael. He remained as TD for Monaghan from 1937 to 1969. Dillon became deputy leader of Fine Gael under W.T. Cosgrave. He was expelled from Fine Gael in 1942 over his controversial views on Irish neutrality during World War II, when he urged Éire to abandon neutrality and side with the Allies. In the first inter-party government (1948-1951) Dillon was appointed Minister for Agriculture as an Independent TD. Dillon rejoined Fine Gael in 1953. He became Minister for Agriculture again in the second inter-party government (1954-1957).
In 1959 James Dillon became the leader of Fine Gael, the party he was expelled from in 1942. He became president of the party in 1960. In 1965 Fine Gael lost the election to Sean Lemass and Fianna Fáil. Dillon retired as party leader due to old age. He remained on as a TD until 1969. He then retired from politics completely.