Phipps was at Eton College with Joseph Banks, but left early to go to sea with his uncle Captain the Hon. A. J. Hervey. In 1766 he sailed to Newfoundland as Lieutenant on HMS Niger. Banks accompanied him as ship's naturalist.
On June 4, 1773 Phipps set off from Deptford on a voyage towards the North Pole. He had two boats, the Racehorse and the Carcass. Phipps took with him Dr Irving as naturalist and doctor, and Israel Lyons (1739-1775) as astronomer. One of the crew was a young Horatio Nelson. They sailed beyond Svalbard to the Seven Islands, but were forced back by the ice and returned to Orfordness on September 17. During the voyage Phipps was the first European to describe the Polar Bear and the Ivory Gull.