Let's say we have a family of models over a certain space which admits rescalings which are automorphisms but not isometries. Let me explain what I mean by that. For example, in Euclidean space, the isometries preserve the distance between any two points. Even though a rescaling of a Euclidean space is an automorphism in the sense that a rescaled n-dimensional Euclidean space is simply another n-dimensional Euclidean space, which are isomorphic, it's not an isometry because it changes distances by a constant factor. The same thing goes for Minkowski space. However, this isn't true for conformal geometries because rescalings are isometries there. The set of all models of the family is called the parameter space, which is sometimes a manifold. At any rate, it usually admits a differentiable structure. Because of the rescaling automorphisms of the underlying space, given any particular model in the family, by rescaling the space, we get another model which may or may not be the same as the original model. Here, we make the further assumption that by rescaling the underlying space, any rescaled model of the family also belongs to the family. The group of rescalings is isomorphic to R+, the group of positive real numbers under multiplication. What I've said previously amounts to saying that there's a group action of the rescaling group on the parameter space. In addition, we will assume this group action is differentiable (or maybe continuous/smooth, depending on the needs the renormalization group is put to). The rescaling group is called the renormalization group and the group action is called the renormalization group flow.
See also Critical exponents, Lyapunov exponent.
In statistical mechanics, a second order phase transition corresponds to an infrared repellor (i.e. an "unstable" infrared fixed point).