He was born in Knoxville, Tennessee and suffered from polio as a child, which incapacitated his leg. McGhee spent much of his youth immersed in music, singing with local harmony group (the Golden Voices Gospel Quartet) and teaching himself the guitar. At the age of 22 he was became travelling musician, meeting and befriending Blind Boy Fuller, whose guitar playing influenced him greatly, (to the extent that after Fuller's death in 1941, McGhee adopted his mentor's name, styling himself Blind Boy Fuller II). By that McGhee was recording for Okeh records in Chicago, but his real success did not come until his 1942 relocation to New York City, where he was teamed up with Terry. The pairing was an overnight success, recording and touring extensively until the early 1970s. During the "folk revival" of the 1960s Terry and McGhee were highly popular on the concert and festival circuits, occasionally adding new material but usually remaining faithful to their roots.