Feldman was born in New York City. He studied piano with Madame Maurina-Press, a pupil of Ferruccio Busoni, and later composition with Wallingford Riegger and Stefan Wolpe. He did not agree with many of the views of these composition teachers, and he spent much of his time simply arguing with them. Feldman was composing at this time, but in a style very different to that he would later be associated with.
In 1950, Feldman went to hear the New York Philharmonic give a performance of Anton Webern's Symphony. At the concert, he met John Cage, and the two became good friends. Under Cage's influence, Feldman began to write pieces which had no relation to compositional systems of the past, such as the constraints of traditional harmony or the serial technique. He experimented with non-standard systems of musical notation, often using grids in his scores, and specifying how many notes should be played at a certain time, but not which ones. Feldman's experiments with the use of chance in his composition in turn inspired John Cage to write pieces like the Music of Changes, where the notes to be played are determined by consulting the I Ching.
Through Cage, Feldman met many other prominent figures in the New York arts scene, among them Jackson Pollock, Philip Guston and Frank O'Hara. He found inspiration in the paintings of the abstract expressionists, and though the 1970s wrote a number of pieces around twenty-minutes in length, including Rothko Chapel (1971, written for the building of the same name which houses paintings by Mark Rothko) and For Frank O'Hara (1973).
Later, he began to produce his very long works, often in one continuous movement, rarely shorter than half an hour in length and often much longer. These works include Piano and String Quartet (1985, around eighty minutes), For Philip Guston (1984, around four hours) and, most extreme, the String Quartet II (1983), which is over five hours long without a break. It has still never been performed in its entirety, although a recording of it was made in 2001. Typically, these pieces do not change in mood throughout and tend to be made up of mostly very quiet sounds. Feldman said himself that quiet sounds had begun to be the only ones that interested him.
Feldman married the composer Barbara Monk shortly before his death in 1987 at his home in Buffalo, New York.