Brown was born in Lunenburg, Massachusetts. His early musical background was in jazz. Intent on an aeronautical career, he studied engineering and mathematics at Northeastern University (1944-5) before joining the Army Air Corps. Subsequently, he attended the Schillinger School of Music, Boston (1946-50), and studied the trumpet and composition privately. His encounter with the work of Jackson Pollock and Alexander Calder was particularly influential to his developing musical aesthetic. After moving to Denver (1950-52), he painted, taught the Schillinger method and explored various compositional techniques. At Cage's invitation, he travelled to New York to work on the Project for Music for Magnetic Tape, the results of which included Octet I (1952-3). This studio experience proved valuable in his later roles as an editor and recording engineer for Capitol Records (1955-60) and as the director of artists and repertoire, and producer for Mainstream-Time Records' Contemporary Sound series (1960-73).
Through Cage and Tudor, Brown was brought to the attention of the European avant garde. Boulez was helpful in establishing contacts with publishers, performers and orchestras; later a strong relationship developed with Maderna. Beginning in 1956, Brown visited Europe on numerous occasions: he lectured at Darmstadt (notably in 1964-5) and received several commissions, including those for Penthatis (Domaine Musical, 1957-8), Available Forms I (City of Darmstadt, 1961) and ''Available Forms II'\' (Rome Radio Orchestra, 1962). His notational and structural innovations were widely copied; as Feldman noted: "I think he's been ripped off more than any of us, in an overt way" (1989). Later European distinctions included appointments as composer-in-residence with the Künstler Programm, West Berlin (1970-71), and the Rotterdam Philharmonic Orchestra (1974), visiting professor at the Basel Conservatory (1974-5) and guest conductor with the Cologne Radio Symphony Orchestra (1963-5) and Saarbrücken RSO (1981).
Brown also received recognition in America. He held the W. Alton Jones chair of composition and was composer-in-residence at the Peabody Conservatory, Baltimore (1968-73), where he was awarded an honorary DMus in 1970. His visiting or guest appointments included positions at SUNY, Buffalo (1975), the California Institute of the Arts (1973-83), Yale University (1980–81) and the Aspen and Tanglewood music festivals (each on several occasions). In addition, he served as director of the Fromm Music Foundation at Harvard University and programming director of the Fromm Weeks of New Music at Aspen, Colorado (1985-90). Among his numerous honours are a Guggenheim Fellowship (1965-6), an award from the National Institute of Arts and Letters (1972), a commission from the Koussevitzky Foundation (Cross Sections and Color Fields, 1972-5), the Brandeis University Creative Arts Award (1977), a Letter of Distinction from the AMC (1996) and the Foundation for Contemporary Performance Art's John Cage Award for Music (1998).
Brown died in Rye, New York.