He joined the Bolshevik Party on May 5, 1917 in Vitebsk before the Revolution in October (November) 1917. He fought in the Red Army during the Russian Civil War, and made a career in the communist party after the war.
On September 26, 1936, he was appointed Head of the NKVD and Member of the Presidium Central Executive Committee. From 1936 to 1938, he led the purging of "politically unwanted" members from the communist party and the army, in what was later known as Stalin's Great Purges. On August 21, 1938, he was made "People's Commissar of Water Transport". This "promotion" was tantamount to his downfall as Lavrenty Beria now was the de facto head of the NKVD.
Sometime in 1939 Yezhov was arrested, and he was probably killed in 1939 or 1940. The rumours and speculations around his death are many, but none has been officially confirmed.