It can be a derogatory term that may be used to describe the dispute resolution mechanism used by prison inmates to resolve disputes within the prison based upon the pecking order of the prisoners.
It can also be a positive term. For example, a baseball team might have a kangaroo court to punish players for errors and mistakes on the field. Fines are allotted, and at the end of the year, the money collected is given to charity.
Although it might be an obvious guess that the phrase was coined in Australia, it seems to have originated during the California Gold Rush, with the first written reference in 1853. It is possible that the phrase arose out of a synergistic combination of informal courts convened to deal with 'claim jumpers', the many Australian participants in the Gold Rush and a bit of word play.