Nicholas explains the protocol we use for messaging over TCP (net.tcp) and Windows Named Pipes (net.pipe) in a nice series of blog posts:

1. Introduction
2. Framing Protocol Overview
3. Preamble Records
4. Encoding Records
5. Upgrade Negotiations (security, compression, etc)
6. Message Data

#3-5 are the details behind the "handshake" I alluded to here.