Alphonse Picou was born in a prosperous middle class Creole of Color family in downtown New Orleans. He was working as a professional musician by age 16 on both guitar and clarinet, but then concentrated on the later instrument. As his family frowned on music being a person's sole trade, Picou trained and worked as a tin smith, including putting the copper sheeting on church steeples; however soon Picou was so much in demand as a clarinetist that he made most of his living from music. He played classical music with the Creole section's Lyre Club Symphony Orchestra. He also played with various dance bands and brass bands, including those of Bouboul Fortunea Augustat, Bouboul Valentin, Oscar DuConge, Manuel Perez, Freddie Keppard, Bunk Johnson, the Excelsior Brass Band, the Olympia Brass Band and others. The light skinned Picou sometimes worked with white bands as well in his youth, including at least on occasion Papa Jack Laine. (This opportunity was not available to musicians with darker skin due to racial discrimination in the U.S. Southern States at the time.)
Picou was one of the early musicians playing in the new style that was developing in the city, not yet known as "jazz". He sometimes played in the band of perhaps the most important force in the musical change, Buddy Bolden.
Many younger clarinetists sited Picou as an important influence, including Johnny Dodds and Jimmy Noone. Picou's style (those who knew him for many years said that his style when he recorded was little changed from how he played early in the 20th century) is lilting with a gentle raggy feel with subtle variations that are usually more melodic embelishments than what would later be called improvisation. His style struck many who heard Picou late in his career as either "not quite jazz" or "just barely jazz".
Picou is perhaps best known for originating the clarinet part on the standard "High Society". Some have mistakenly stated that he wrote the number, which was actually a 1901 marching band composition by Porter Steele. Picou rearranged it giving it a gentle swing and paraphrased the Piccolo part to create his famous clarinet solo. This became a local standard part, and no younger New Orleans clarinetist was considered proficient until he could duplicate Picou's part. Unusually in a music that values improvisation it became a set piece; commonly later clarinetists would solo once through reproducing or sticking close to Picou's solo, and then do their own improvisations on a second solo.
Alphonse Picou at least once followed fellow musicians up north to Chicago about 1917-1918 (and possibly briefly to New York City in the early 1920s) but said he didn't like it up north and spent the bulk of his career in his home city.
"King" Joe Oliver commissioned Picou to write new tunes for his band. Picou's compositions include "Alligator Hop", "Olympia Rag".
During the Great Depression Picou returned to metal smithing. In the 1940s he was able to return to playing professionally regularly, made his first recordings, and opened a bar in a building he owned on Claiborne Avenue.
For years into the 1950s he was a regular on Bourbon Street in the French Quarter with Papa Celestin's Band (with whom he also did radio broadcasts) and leading his own group.
Picou's funeral procession in 1961 was one of the largest the city had seen, with several brass bands and many additional musicians playing to give Alphone Picou a send off. Many commentators said it marked the end of an era as the death of the last prominent still working musician from the very birth of jazz music.