The village name is from both modern and old English. The affix 'Cold' refers to the village's bleak location on the banks of the River Ouse. The last part 'Brayfield' means 'open land by higher ground' and was referred to in the Anglo-Saxon Chronicle of 967 as Bragenfelda.
Anciently the manor of Brayfield belonged to the Blossomville family of nearby Newton Blossomville.