A rudder is a device used to steer a ship or other watercraft. In its simplest form, a rudder is a flat sheet of material attached with hinges to the ship's stern. A tiller - basically, a stick or pole - is attached to the top of the rudder to allow it to be turned in different directions.
On an aircraft, the rudder is a control surface, usually attached to the fin (or vertical stabiliser) which allows the pilot to control the aircraft in the yaw axis. It is not used to effect turns - the ailerons are used for that - but the rudder is necessary to correctly balance the various acting forces in a turn. In some designs, an all-moveable fin is used instead of a rudder.