It was mentioned in the Domesday Book as an important fishing port, and it received a charter from Henry VII in 1489.
But in the following centuries, a shingle bar built up across the harbour mouth and ruined any chance of the town becoming a major port.
In 1659, a fire devastated most of the town and damaged the Church of St. Edmund, whose original structure dated from 12th century.
Southwold Pier was built in 1900, was practically destroyed by a gale in 1934, and had a major refurbishment in 2001. Whilst many English seaside piers are in decline, Southwold Pier is enjoying renewed popularity.
There is an unusual Amber Museum and a famous brewery.
Southwold Lighthouse was constructed in 1887 by Trinity House. It replaced three local lighthouses which were under serious threat from coastal erosion. It started to operate in 1890. It was electrified and de-manned in 1938. Trinity House organise visits to the lighthouse during the summer.
In a square just beside the beach, expressively named Gun Hill, the defensive cannons are still at their place, remembering the battle of Sole Bay, fought in 1672 between English and French fleets on one side, and the Dutch (under de Ruyter) on the other. The battle was bloody but indecisive, and many bodies were washed ashore. A museum collects mementoes of the event.
Six 18 lb. guns standing on the cliff were captured from the Scots at Culloden and given to the town by the Duke of Cumberland.
A further memento of maritime heritage is the 1912 Looe Lugger named "Girl Sybil".