You probably want to take first at pipelines for proper string framing, coding/decoding and backressure:

There is a simple example in the TCP section using pipelines:

(don't be confused that it uses SSL, you can leave that part out)

And yes, the FSM module is probably what you want to use to implement the protocol states.

Related Query

More Query from same tag