The problem of universals is the problem of accounting for the fact that some things are of the same type--for example, Fluffy and Kitzler are both cats--or, to put it another way, the fact that certain properties are repeatable--such as: the grass is green, my shirt is green, Kermit is green, etc... One wants to know in virtue of what make Fluffy and Kitzler both cats; in virtue of what makes the grass, my shirt and Kermit all green.
The realist's answer is that all the green things are green in virtue of the existence of a universal--a single abstract thing. In this case, that is a part of all the green things. That is to say, with respect to the colour of the grass, my shirt and Kermit, one of their parts is identical. In this respect, the three parts are literally one. Greenness is repeatable because there is one thing, the universal, that manifests itself wherever there are green things.
Nominalism denies that there are such things as universals. The motivation to deny universals flows from several concerns. First, where are they? Plato famously held that there is a realm of abstract forms or universals apart from the natural world of particulars we see with our own eyes. Particular real-world objects merely exemplify or instantiate the universal up in universal heaven. But where is universal heaven? Is it outside of space and time? But nothing is outside of space and time. And in any case, what is the nature of the instantiation or exemplification relation? It seems mysterious.
A variety of realists, the moderate realists, hold that there is no heaven in which universals live, but rather universals are located in space and time wherever they are manifest. Now, recall that a universal, like greenness, is supposed to be a single thing. Nominalists find it weird that there could be a single thing that exists in a bunch of places all at once. The realist maintains that all the instances of greenness are held together by the exemplification relation, but, again, this relation seems mysterious.
Finally, many philosophers prefer stripped down ontologies populated with only the bare minimum of kinds of entities (they have a taste for "desert landscapes," as W. V. Quine put it). Since everyone has to include particulars like cats in their inventory of being anyway, wouldn't it be nice if we could explain everything we want to explain without adding univerals like "catness" into the mix?
There are various forms of nominalism ranging from extreme to almost-realist. On the extreme end is "predicate" nominalism. Fluffy and Kitzler are both cats simply in virtue of the fact that the predicate 'cat' applies to both of them. That's all there is to it. However, the realist will object in frustration that we haven't been told in virtue of what the predicate applies.
The resemblance nominalist will reply that 'cat' applies to both cats in virtue of either the fact that Fluffy and Kitzler resemble an exemplar cat closely enough to be classed together with it as members of its kind, or that they differ from each other (and other cats)quite less than they differ from other things, and this warrants classing them together. Some resemblance nominalists will concede that the resemblance relation is itself a universal, but is the only universal you need. This betrays the spirit of nominalism. Others argue that each resemblance relation is a particular, and is a resemblance relation simply in virtue of its resemblance to other resemblance relations. This generates an infinite regress, but many agree that it is not vicious.
One way to be a nominalist without being an "ostrich nominalist" like the predicate nominalists ("ostrich" because they seem to simply stick their heads in the sand and pretend there isn't a problem--the phrase is D. M. Armstrong's) is to build a theory of resemblance nominalism on a theory of tropes. A trope is a particular instance of a property, like the specific greenness of this here shirt, or the singular coyness of Gwyneth's smile. One might argue that there is a primitive, objective resemblance relation that holds among like tropes. But that seems arbitrary. Another route is to argue that all apparent tropes are constructed out of more primitive tropes and that the most primitive tropes are the entities of complete physics. Primitive trope resemblance may thus be accounted for in terms of causal indiscernibility. Two tropes are exactly resembling if switching them would make no difference to the events in which they are taking part. Varying degrees of resemblance at the macro level can be explained by varying degrees of resemblance at the micro level, and micro-level resemblance is explained in terms of something no less robustly physical than causal power. Armstrong, perhaps the most prominent contemporary realist, argues that such a trope-based variant of nominalism has promise, but holds that it is unable to account for the laws of nature in the way his theory of universals can.
Note: Problem of universals has an overlapping discussion of nominalism.