title Markov Process author Keith A. Lewis \newcommand\bs[1]{\boldsymbol{{#1}}} \newcommand\NN{\bs{N}} Let $S$ be a set of states and define $S^* = \cup_{n\in\NN} S^n$ to be the set of finite sequences of states.