Learning about Markov chain

I’ve been hearing about “Markov chain” for long enough – it was time I learned something. Wikipedia seemed like a good starting point. I have to warn you though, be careful with scrolling on that page, because you can easily end up looking at something like this:

partial Markov chain

If you aren’t a rocket scientist or someone who solves integrals for fun, by all means, use the contents menu or jump directly to the Applications section.That’s where all the fun is. Here are some quotes for you to get interested and for me to remember.


Markovian systems appear extensively in physics, particularly statistical mechanics, whenever probabilities are used to represent unknown or unmodelled details of the system, if it can be assumed that the dynamics are time-invariant, and that no relevant history need be considered which is not already included in the state description.


Several theorists have proposed the idea of the Markov chain statistical test, a method of conjoining Markov chains to form a ‘Markov blanket’, arranging these chains in several recursive layers (‘wafering’) and producing more efficient test sets — samples — as a replacement for exhaustive testing.

Queuing theory:

Claude Shannon’s famous 1948 paper A mathematical theory of communication, which at a single step created the field of information theory, opens by introducing the concept of entropy through Markov modeling of the English language. Such idealised models can capture many of the statistical regularities of systems. Even without describing the full structure of the system perfectly, such signal models can make possible very effective data compression through entropy coding techniques such as arithmetic coding. They also allow effective state estimation and pattern recognition

Internet applications:

The PageRank of a webpage as used by Google is defined by a Markov chain.


Markov models have also been used to analyze web navigation behavior of users. A user’s web link transition on a particular website can be modeled using first or second order Markov models and can be used to make predictions regarding future navigation and to personalize the web page for an individual user.


Markov chain methods have also become very important for generating sequences of random numbers to accurately reflect very complicated desired probability distributions – a process called Markov chain Monte Carlo or MCMC for short. In recent years this has revolutionised the practicability of Bayesian inference methods.


Markov chains can be used to model many games of chance. The children’s games Snakes and Ladders, Candy Land, and “Hi Ho! Cherry-O”, for example, are represented exactly by Markov chains. At each turn, the player starts in a given state (on a given square) and from there has fixed odds of moving to certain other states (squares).


Markov chains are employed in algorithmic music composition, particularly in software programs such as CSound or Max. In a first-order chain, the states of the system become note or pitch values, and a probability vector for each note is constructed, completing a transition probability matrix

Markov parody generators:

Markov processes can also be used to generate superficially “real-looking” text given a sample document: they are used in a variety of recreational “parody generator” software

Markov chains for spammers and black hat SEO:

Since a Markov chain can be used to generate real looking text, spam websites without content use Markov-generated text to give illusion of having content.

This is one of those topics that makes me feel sorry for sucking at math so badly. Is there a “Markov chain for Dummies” book somewhere? I haven’t found one yet, but Google provides quite a few results for “markov chain” query.

9 thoughts on “Learning about Markov chain”

  1. Hello again Leonid!

    I have to admit the same “topic makes me too feel sorry for sucking at math so badly.”

    However I managed to get pass my fears this summer since I decided to wear my “black hat” :-) for a few days and created a “not for release plugin markov chain article generator plugin” – it was very funny trying out different parameters to mutate original non-copyrighted articles as much to beat copyscape detection while keeping an “illogical” :-) sense of text!

    Of course I was just playing – I do not plan to become a Splogger/Spammer after so many years on the “Good” side!

    However Akismet must have supernatural powers or even worst spying on us :-), since I bet you had to dig my last comment from your Akismet spam folder :-)

    Actually my blog become suddenly popular and a few not so good people saw the rise with an bad eye and rushed to add me in every blacklist out there !

    Cheers Nick!

    10++++/10 for the new !! AWESOME !! blog design/theme!

  2. Nick,

    Of course I was just playing – I do not plan to become a Splogger/Spammer after so many years on the “Good” side!

    I see you. :) We all have to know what and how the “bad guys” are doing, so that we can protect The Good from The Evil.

    However Akismet must have supernatural powers or even worst spying on us :-), since I bet you had to dig my last comment from your Akismet spam folder

    Oh, well, this happens once in a while. Gladly, I noticed your comment and brought it back. Now your comments should just work (like this one).

    10++++/10 for the new !! AWESOME !! blog design/theme!

    Thanks, I’ll pass these to all the person who did it. :)

  3. An interesting fact, that Markov Chains have been used for long time, but now you can hear almost everyone trying to get into it.

    Another additional instance: Markov Chains are also used in Sensor’s Fuzzy Logic :)

  4. The good folks at Brown University have taken pity on the math-challenged among us and written a Markov tutorial that is devoid of any formulas.


    While this is intended “for dummies” it’s still not something the average joe will find easy to grasp. Nonetheless, many kudos to Brown. It certainly helped extend my limited understanding.

Leave a Comment