The village name is Anglo Saxon in origin, and means 'island held by the de Turs family'. This refers to a dry patch of land in the marshes of the Aylesbury Vale, on the edge of which the village stands.
The village was anciently part of Buckinghamshire, but the border was moved in 1974.
There is an annual music festival that takes place in Towersey on the August bank holiday weekend, called Towersey Village Festival. It has been running every year for nearly forty years and is centred around folk music.