Nicholas explains the protocol we use for messaging over TCP (net.tcp) and Windows Named Pipes (net.pipe) in a nice series of blog posts:
- Framing Protocol Overview
- Preamble Records
- Encoding Records
- Upgrade Negotiations (security, compression, etc)
- Message Data
#3-5 are the details behind the "handshake" I alluded to here.