Invertebrate is a term coined by Chevalier de Lamarck to describe any animal without a backbone or vertebra, like insects, squids and worms.
He divided them into two groups, the Insecta and the Vermes. However, the invertebrates are not a coherent group of animals, as many are much more closely related to vertebrates than to one another.
This is one of the places where formal taxonomy and common usage diverge. People tend to think of "things like us" and "everything else." In a very broad sense, "vertebrates" are things like us: they at least have bones, unlike all those weird things. Dividing the world into X and not-X is conceptually appealing and sometimes useful, even when the only thing the not-X have in common is that definition by exclusion. In this case, the excluded group, invertebrates, makes up 97% of animals in the world.