Prior to entering elected politics, Gray pursued studies in Commerce at McGill University in Montreal, Quebec and in Law at Osgoode Hall in Toronto, Ontario. He was first elected to Parliament for the riding of Essex West on June 18, 1962, as a member of the Liberal Party of Canada. He was re-elected in twelve subsequent federal elections, making him the longest continuously-serving Member of Parliament in Canadian history.
Gray served in a variety of roles during his parliamentary career, including cabinet ministries and committee chairmanships during the Liberal governments of Lester Pearson and Pierre Trudeau, and as Opposition House Leader from 1984 to 1990.
From February 6, 1990 to December 21, 1990, he was Leader of the Opposition, between the resignation of John Turner as Liberal leader and the election to Parliament of his successor, Jean Chrétien.
When the Liberals returned to power in 1993, Gray was appointed Leader of the Government in the House of Commons and Solicitor General of Canada. On June 11, 1997, he was appointed Deputy Prime Minister.
Gray retired from Parliament on January 14, 2002, and was appointed Canadian Chair of the International Joint Commission, a bilateral organization which deals with Canada-United States transboundary issues on water and air rights. On January 15, 2002 the Governor General of Canada granted Gray the title "Right Honourable", in honour of his distinguished and record-setting contribution to Canadian political life.
On July 23, 1967, Gray married lawyer Sharon Sholzberg, with whom he has two children.
Gray is also a Companion of the Order of Canada, a designation which can be bestowed on only 165 outstanding Canadians at any given time.