A finitary argument is one which can be translated into a finite set of symbolic propositions starting from a finite set of axioms. In other words, it is a proof that can be written on a large enough sheet of paper (including all assumptions).
The emphasis on finitary methods has historical roots.
In the early 20th century, logicians aimed to solve the problem of foundations, that is, answer to the question: what is the true base of mathematics? The program was to be able to rewrite all mathematics starting using an entirely syntactical language without semantics. In the words of David Hilbert (referring to Geometry), "it does not matter if we call the things chairs, tables and cans of beer or points, lines and planes."
The stress on finiteness came from the idea that human mathematical thought is based on a finite number of principles and all the reasonings follow essentially one rule: the modus ponens. The project was to fix a finite number of symbols (essentially the numerals 1,2,3,... the letters of alphabet and some especial symbols like "+", "->", "(", ")", etc.), give a finite number of propositions expressed in those symbols, which werer to be taken as "foundations" (the axioms), and some rules of inference which would model the way humans make conclussions. From these, regardless of the semantic interpretation of the symbols the remaining theorems should follow automatically using the stated rules (which make mathematics look like a game with symbols more than a science). The hope was to prove that from these axioms and rules 'all the theorems of mathematics could be deduced.
The aim itself was proved impossible by Kurt Gödel in 1931, with his Incompleteness Theorem, but the general mathematical trend is to use a finitary approach, arguing that this avoids considering mathematical objects that cannot be fully defined.