McPhatter recruited several members of his former group, the Mount Lebanon Singers, and finally settled on Gerhart and Andrew Thrasher on tenor and baritone, respectively, Willie Ferbee, bass, and Walter Adams on guitar. Adams' presence on guitar made the group unusual among R&B groups of the time. He was soon replaced by Jimmy Oliver. This line-up released the group's first major hit, "Money Honey", occasionally considered the first rock and roll record. By this time, Bill Pinkney had taken over on bass, and the group released several more hits before McPhatter left in 1954 to pursue a solo career. McPhatter had demanded a large share of the group's profits, which he had been denied in the Dominoes, but, upon his departure, did not ensure that this would continue for his successor. He sold his share of the group to George Treadwell, manager, former jazz musician and husband of legendary singer Sarah Vaughan. As a result, the Drifters' cycled through copious members, none of whom made much money. McPhatter later expressed regret at this action, recognizing that it doomed his fellow musicians to unprofitability.
McPhatter was first replaced by David Baughn, whose erratic behavior and similar sound to McPhatter made him unsuitable in the eyes of Atlantic Records executives. Treadwell soon added Bobby Hendricks (of The Swallows and The Five Crowns), but to no success. Baughn soon quit the group, and was replaced by Johnny Moore (of The Hornets). This line-up had a major R&B hit in 1955 with "Adorable", followed by several others. The group, however, was not able to break into mainstream markets.
In the early 1950s, The Drifters began working with Jerry Leiber and Mike Stoller, legendary songwriters, who eventually became the group's producers as well. This is widely considered the group's golden age, inaugurated by the 1956 hit "I Gotta Get Myself a Woman". Low salaries contributed to burnout among the members, particularly Bill Pinkney, who was fired after asking Treadwell for more money. Thrasher left as well in protest, and Pinkney and Bobby Hendricks soon formed The Flyers.
Jimmy Ricks filled in as bass, and he was soon replaced by Tommy Evans (of The Dominoes). Charlie Hughes also joined as a baritone. The group's momentum was stalled by Johnny Moore's being drafted. Hendricks returned and Jimmy Millander took over as baritone. By 1958, Hendricks and Olivier left, followed by the rest of the group after Treadwell refused to give them any more money. Treadwell still had performing obligations for the now-nonexistent Drifters, and approached Lover Patterson, manager for The Crowns (formerly The Five Crowns), who agreed. All but one member of The Crowns agreed to become the Drifters. The new line-up was led by Charlie Thomas, and also included Dock Green (baritone), Elsbeary Hobbs (bass) and Ben Nelson (baritone).
This new line-up released several singles that have become known as landmark recordings, but was not able to release any hits except "There Goes My Baby". Ben Nelson was particularly unhappy with his share of the profits, and left to launch a solo career under the name Ben E. King. He was replaced by Johnny Williams, and then Rudy Lewis (of The Clara Ward Singers). This new line-up was still innovative, in spite of a lack of major mainstream success. While recording "Please Stay", songwriter Burt Bacharach met Dionne Warwick, a back-up singer, thus beginning a legendary partnership.
In 1964, the group was scheduled to record "Under the Boardwalk" on May 21. Rudy Lewis died under mysterious circumstances the night before, and his position was filled by Johnny Moore (who had just rejoined), the group's last big hit. Though the band toured and recorded for several years, the group's career was essentially over. In the 1970s, several line-ups used the name on the oldies circuit. There were a few minor British hits, and a brief revival of the name for an essentially unrelated disco group in the latter part of the decade, and the confusion continued into the 1980s when even Ben E. King returned to lead a touring line-up.