- #1
- 7,861
- 1,599
- TL;DR Summary
- Is there a measure of entropy for probability models that obeys the second law of thermodynamics?
Thinking of the common language notion of "entropy" as "uncertainty", how can running a simulation based on a probability model implement entropy increasing? After all, the simulation picks definite outcomes to happen, so (intuitively) there is less uncertainty about the future as definite outcomes occur.
For a slightly more technical way of putting it, a probability model for phenomena provides a way to simulate outcomes of processes using various probability distributions. Probability distributions can be assigned an entropy by the Shannon definition of entropy. In the case where there are dependent random variables X,Y in the model, a definite outcome for random variable X might decrease or increase the entropy of the resulting conditional probability distribution for the future event Y compared to its prior distribution before X was realized. Does it require care in designing probability models to force Shannon entropy to increase as a simulation progresses in time? Is there a measure of entropy for a simulation other than Shannon entropy, which will "naturally" increase as the simulation is run?
For a slightly more technical way of putting it, a probability model for phenomena provides a way to simulate outcomes of processes using various probability distributions. Probability distributions can be assigned an entropy by the Shannon definition of entropy. In the case where there are dependent random variables X,Y in the model, a definite outcome for random variable X might decrease or increase the entropy of the resulting conditional probability distribution for the future event Y compared to its prior distribution before X was realized. Does it require care in designing probability models to force Shannon entropy to increase as a simulation progresses in time? Is there a measure of entropy for a simulation other than Shannon entropy, which will "naturally" increase as the simulation is run?