Following graduation from the Naval Academy in June 1942, he joined the destroyer USS Phelps, and in August 1943 was detached for instruction in the Operational Training Command, Pacific, at San Francisco, California. In January 1944 he reported on board the USS Robinson, and for "heroic service as Evaluator in the Combat Information Center ... (of that destroyer), in action against enemy Japanese battleships during the Battle for Leyte Gulf, 25 October 1944 ..." he was awarded the Bronze Star with Combat "V.".
After the cessation of hostilities in August 1945, until December 8th of that year, he commanded (as prize crew officer) HIMJS Ataka, a 1200-ton Japanese river gunboat with two hundred officers and men. In that capacity he took the first ship since the outbreak of World War II, flying the United States flag, up the Whangpoo River to Shanghai. There they helped to restore order and assisted in disarming the Japanese.
He next served as Executive Officer of the destroyer USS Saufley, and in March 1946 was transferred to the destroyer USS Zellars, as Executive Officer and Navigator. In January 1948 he was assigned to the Naval Reserve Officers Training Corps Unit of the University of North Carolina, where he remained until June 1950. That month he assumed command of USS Tills, in commission in reserve status. That destroyer escort was placed in full active commission at Charleston Naval Shipyard on 21 November 1950, and he continued to command her until March 1951, when he joined the battleship USS Wisconsin as Navigator.
Detached from USS Wisconsin in June 1952, he attended the Naval War College, Newport, Rhode Island, and in June 1953 reported as Head of the Shore and Overseas Bases Section, Bureau of Naval Personnel, Navy Department, Washington, D. C. He also served as Officer and Enlisted Requirements Officer and as Action Officer on Medicare Legislation. Completing that tour of duty in July 1955, he assumed command of the destroyer USS Arnold J. Isbell, participating in two deployments to the Seventh Fleet. In this assignment he was commended by the Commander, Cruiser-Destroyer Forces, U. S. Pacific Fleet for winning the Battle Efficiency Competition for his ship and for winning Excellence Awards in Engineering, Gunnery, Antisubmarine Warfare, and Operations. In July 1957 he returned to the Bureau of Naval Personnel for further duty. In December 1957 he was transferred to the Office of the Assistant Secretary of the Navy (Personnel and Reserve Forces), and served as Special Assistant for Naval Personnel until November 1958, then as Special Assistant and Naval Aide until August 1959.
Ordered to the first ship built from the keel up as a guided missile ship, USS Dewey (DLG-14), building at the Bath (Maine) Iron Works, he assumed command of that guided missile frigate at her commissioning in December 1959, and commanded her until June 1961. During this period of his command, Dewey earned the Excellence Award in Engineering, Supply, Weapons, and was runner-up in the Battle Efficiency Competition. He was a student at the National War College, Washington, D. C., during the 1961-1962 class year. In June he was assigned to the Office of the Assistant Secretary of Defense (International Security Affairs), Washington, D. C., where he served first as Desk Officer for France, Spain and Portugal, then as Director of Arms Control and Contingency Planning for Cuba. From December 1963 until 21 June 1965 he served as Executive Assistant and Senior Aide to the Honorable Paul H. Nitze, Secretary of the Navy. For duty in his tour in the offices of the Secretary of Defense and the Secretary of the Navy, he was awarded the Legion of Merit.
After his selection for the rank of Rear Admiral, he assumed command in July 1965 of Cruiser-Destroyer Flotilla Seven.
In September 1968 he became Commander Naval Forces, Vietnam and Chief of the Naval Advisory Group, U. S. Military Assistance Command, Vietnam. President Richard M. Nixon nominated him as Chief of Naval Operations on 14 April 1970. Upon being relieved as Commander Naval Forces, Vietnam, on 15 May 1970, he was awarded a Gold Star in lieu of a second Distinguished Service Medal for exceptionally meritorious service. He assumed command as Chief of Naval Operations on 1 July 1970 and retired from that position on 1 July 1974. In 1976, he unsuccessfully ran as a Democratic candidate for the Senate from Virginia. Later he held the presidency of the American Medical Building Corporation in Milwaukee, Wisconsin.
Admiral Zumwalt died on 2 January 2000 at the Duke University Medical Center in Durham, North Carolina. His home was in Arlington, Virginia. He was married to the former Mouza Coutelais-du-Roche of Harbin, Manchuria, and they had two sons, Elmo R. Zumwalt III, who died of cancer in 1988, and James Gregory Zumwalt, and two daughters, Ann F. Zumwalt Coppola and Mouza C. Zumwalt-Weathers. He was also survived by six grandchildren.