In 1949 he founded the nationalist Sierra Leone People's Party (SLPP), which won the [1951]] election to the Legislative Council. After heading the departments of Health, Agriculture, and Forestry, he was elected chief minister in 1954. In 1958, his leadership of the party was challenged by his younger brother, Albert Margai, but even though Albert narrowly won the internal party election, he declined the leadership of the party, and left to form the opposition People's National Party, rejoining his brother in a coalition government in 1960.
Over the next two years, as Sierra Leone headed for independence, Margai oversaw the creation of a new constitution for the colony, Upon its adoption, in 1958, he became premier. Knighted in 1959, he was prime minister at the time of independence on April 27, 1961, and won the ensuing election in 1962. Margai died in office in Freetown and was succeeded by his brother Albert Margai.