He was born in Matabeleland and educated in South Africa, where he met Nelson Mandela and other regional nationalist leaders.
After returning to Bulawayo in 1948, he became a trade unionist for black railway workers.
He founded a number of small organisations, all banned by the British colonial authorities, before founding ZAPU in 1962. It was immediately banned. ZAPU split along ethnic grounds a year after its formation, with Robert Mugabe breaking away with the Shona majority, forming Zanu-PF, leaving ZAPU as a mostly Ndebele organisation. Nkomo was detained numerous times until 1974, when he fled to Zambia to fight for Zimbabwean independence. Elections were held in 1980, and to most observers surprise Nkomo's ZAPU lost to Mugabe's ZANU. Nkomo was offered the ceremonial post of President, but declined. He was appointed to the cabinet, but in 1982 was accused of plotting a coup, and Mugabe unleashed the notorious Fifth Brigade upon Nkomo's Matabeleland homeland.
In 1987 Nkomo was reconciled with Mugabe and two parties merged, leaving Zimbabwe as effectively a one-party state, and leading some Ndebeles to accuse Nkomo of selling out. In a powerless post, and with his health failing, his influence declined until his death in 1999.