Let's say we have a family of models over a certain space which admits rescalings which are automorphisms but not isometries. Let me explain what I mean by that. For example, in Euclidean space, the isometries preserve the distance between any two points. Even though a rescaling of a Euclidean space is an automorphism in the sense that a rescaled n-dimensional Euclidean space is simply another n-dimensional Euclidean space, which are isomorphic, it's not an isometry because it changes distances by a constant factor. The same thing goes for Minkowski space. However, this isn't true for conformal geometries because rescalings are isometries there. The set of all models of the family is called the parameter space, which is sometimes a manifold. At any rate, it usually admits a differentiable structure. Because of the rescaling automorphisms of the underlying space, given any particular model in the family, by rescaling the space, we get another model which may or may not be the same as the original model. Here, we make the further assumption that by rescaling the underlying space, any rescaled model of the family also belongs to the family. The group of rescalings is isomorphic to R+, the group of positive real numbers under multiplication. What I've said previously amounts to saying that there's a group action of the rescaling group on the parameter space. In addition, we will assume this group action is differentiable (or maybe continuous/smooth, depending on the needs the renormalization group is put to). The rescaling group is called the renormalization group and the group action is called the renormalization group flow.
Relevant, Marginal and Irrelevant
Under the action of enlarging rescalings, a parameter could have a positive, zero or negative Lyapunov exponent. That parameter is then called relevant, marginal or irrelavant respectively. In the limit as the rescaling parameter approaches infinity, the RG flows converge to infrared attractors. The points on this attractor are called universality classes because many different models in parameter space start to look like this model at large enough scales, which basically means small scale effects only affect large scale effects insignificantly (a scale independence of sorts). Oftentimes, the parameter space is infinite-dimensional (very huge), but the infrared attractors are only finite dimensional, so that the space of universality classes are much much smaller than the original parameter space. This means, provided we work at large enough scales and don't mind using approximations, we can reduce the entire parameter space to the space of universality classes. The group action of the RG restricted to this attractor is still a group action. So, for models within a sufficiently small neighborhood of the attractor in parameter space, we can project this neighborhood to the attractor, so that running the renormalization group action forward leads to even better approximations but running it backwards eventually leads to divergence out of the neighborhood for almost every point in the neighborhood. This means the RG should really be treated as a monoid in this restriction. Similarly, RG flows can have ultraviolet attractors.
See also Critical exponents, Lyapunov exponent.
In statistical mechanics, a second order phase transition corresponds to an infrared repellor (i.e. an "unstable" infrared fixed point).