Handbook of Linear Algebra, 2nd Edition

Chapter 72

Markov Chains

Beatrice Meini

Universitá di Pisa, Italy

Markov chains are encountered in several applications arising in different contexts, and model many real problems which evolve in time. Throughout, we denote by P[X = j] the probability that the random variable X takes the value j, and by P[X = j|Y = i] the conditional probability that X takes the value j, given that the random variable Y takes the value i. Moreover, we denote by E[X] the expected value of the random variable X and by E[X|A] the conditional expectation of X, given the event A.

72.1 Basic Concepts

Definitions:

Given a denumerable set E, a discrete stochastic process on E is a family {Xt : t ∈ T} of random variables Xt indexed by some denumerable set T and with values ...