John was appointed to lead the first commission on Justinian's new legal code, the Corpus Juris Civilis, and became Justinian's chief legal advisor. He was also appointed praetorian prefect, giving him the power to introduce new taxes on the population. The new taxes were very unpopular, and the mob involved in the Nika riots of 532 demanded that both John and the quaestor Tribonian be dismissed. Justinian did so, until the riots had been suppressed, after which he reinstated John as prefect and Tribonian as quaestor.
After the riots, which had been supported by the upper-class Senators, John, who had the same lower-class background as Justinian, became even more important in political affairs. John influenced Justinian's military decisions, helping to draft the Perpetual Peace with Khosrau I of Persia and convincing Justinian not to empty the treasury with a large expedition to North Africa. John worked with the emperor to reduce the size of the bureaucracy, both in Constantinople and in the provinces, developing a rudimentary meritocracy.
Justinian's wife Theodora resented the amount of power and influence held by John. In 541 Theodora accused him of treason and had him dismissed. The bureaucracy of the empire fell back into corruption under John's successors.
A different John the Cappadocian was Patriarch from 518-520. See John of Cappadocia.