Historically these flows simply were not carefully measured, and the flow proceeded in many commodities and currencies without restriction, clearing being a matter of judgement by individual banks and the governments that licensed them to operate. Mercantilism was a theory that took special notice of the balance in payments and sought simply to monopolize gold, in part to keep it out of the hands of potential military opponents (a large "war chest" being a prerequisite to start a war, whereupon much trade would be embargoed).
As mercantilism gave way to classical economics, these crude systems were later regulated in the 19th century by the gold standard which linked central banks by a convention to redeem "hard currency" in gold. After World War II this system was replaced by the Bretton Woods institutions (the International Monetary Fund and Bank for International Settlements) which pegged currency of participating nations to the US dollar, which was redeemable nominally in gold. In the 1970s this redemption ceased, leaving the system without a formal base. Some consider the system today to be based on oil, a universally desirable commodity due to the dependence of so much infrastructural capital on oil supply. Since OPEC prices oil in US dollars, the US dollar remains a reserve currency, but is increasingly challenged by the Euro, and to some degree the yuan (which is not traded outside China legally, but due to this is almost immune to any degree of chaos on the world's financial markets).
The balance of trade is a more fundamental measure of underlying goods and services, rather than the money that pays for them. Obviously the two are related to some degree, or else trade and payments would probably not continue.
See also: money supply