When a client attempts to start a TCP connection to a server, the client and server exchange a series of messages which normally runs like this:
The client requests a connection by sending a SYN (synchronise) message to the server. The server acknowledges this request by sending SYN-ACK back to the client, which finally responds with an ACK: the connection is established.
A malicious client can skip sending this last ACK message. The server will wait for this bit for some time, as simple network congestion could also be the case of the missing ACK.
If this so called half-open connection binds resources on the server, as is the case in many operating systems, it may be possible to take up all these resources by flooding the server with SYN messages. Once all resources set aside for half-open connections are reserved no new connections (legitimate or not) can be made, resulting in denial of service. Some systems may malfunction badly or even crash if other OS functions get starved of resources this way.
Countermeasures include SYN cookies or limiting the number of new connections from a source per timeframe.