Blog Posts

Trying to Understand Entropy

This post is going to try to answer the question: “What exactly is entropy”? Mathematically, the question is answered easily enough. Suppose we have some probability distribution $p(x)$ over a state space with $m$ outcomes $\{x_1, x_2, … x_m\}$. In this post we will stick with a discrete state space since they make the intuition more clear. The entropy $H$ is defined for such a distribution as:

Read More