The younger son of Captain James Haldane of Airthrey House, Stirlingshire, he was born at Dundee. Educated first at Dundee and afterwards at the high school and University of Edinburgh, at the age of seventeen he joined the "Duke of Montrose" East Indiaman as a midshipman. After four voyages to India he was nominated to the command of the "Melville Castle" in the summer of 1793; but having begun a careful study of the Bible during his voyages, and also come under the evangelical influence of David Bogue of Gosport, one of the founders of the London Missionary Society, he abruptly decided to leave the navy for a religious life, and returned to Scotland.
In about 1796 he became acquainted with the celebrated evangelical, Charles Simeon of Cambridge, in whose company he toured Scotland, distributing tracts and trying to awaken others to an interest in religious subjects. In May 1797 he preached his first sermon, at Gilmerton near Edinburgh, with encouraging success. In the same year he established a non-sectarian organization for tract distribution and lay preaching called the "Society for the Propagation of the Gospel at Home". During the next few years he made repeated missionary journeys, preaching wherever he could obtain hearers, and generally in the open air.
Originally loyal to the Church of Scotland, he was gradually driven by the hostility of the Assembly and the exigencies of his position to leave it. In 1799 he was ordained as pastor of a large Independent congregation in Edinburgh. This was the first congregational church known by that name in Scotland. In 1801 a permanent building replaced the circus in which the congregation had at first met. To this church he continued to minister gratuitously for more than fifty years. In 1808 he made public avowal of his conversion to Baptist views. As advancing years compelled him to withdraw from the more exhausting labours of travel and open-air preaching, he sought to influence the discussion of current religious and theological questions by means of the press.
His son was Daniel Rutherford Haldane the prominent physician.
Among JA Haldane's numerous contributions to current theological discussions were: