Understanding the Uniform Probability Distribution in Statistical Ensembles

[A. Neumaier]

sending someone to Mars [...] What do we do?

We developed uncertainty software for ESA - see Robust and automated space system design for a related publication.

The way of arriving at a design that is acceptable to all parties that have a say is a far more complex process than assigning of faulty probabilities to epistemic uncertainties. Idealized textbook decision procedures are as irrelevant there as are the textbook state reduction recipes when doing a real measurement of a complex process.

 

[A. Neumaier]

There is never a point when you can say with certainty

Physics is never about certainty but about the art of valid approximation.

We treat photons as massless not because we know it for certain but because the mass is known to be extremely small.

Every argument in physics that demands the exact knowledge of the numbers involved is extremely suspicious.

 

[stevendaryl]

We developed uncertainty software for ESA - see Robust and automated space system design for a related publication.

The way of arriving at a design that is acceptable to all parties that have a say is a far more complex process than assigning of faulty probabilities to epistemic uncertainties.

That doesn't mean it is more accurate or more rational. The actual process is as much about psychology (reassuring all interested parties) as it is dealing with uncertainty.

 

[stevendaryl]

Physics is never about certainty but about the art of valid approximation.

But there is no point where you know that your approximation is valid.

 

[A. Neumaier]

You are smart, but I am smart too.
I want the minimal possible accuracy that will trigger you to assign some definite numbers as probabilities.

I'll never assign definite probabilities. probabilities are always estimates, with an intrinsic uncertainty.

 

[A. Neumaier]

But there is no point where you know that your approximation is valid.

This is why physics is slightly in flux, and why these are sometimes controversies.

 

[stevendaryl]

I'll never assign definite probabilities. probabilities are always estimates, with an intrinsic uncertainty.

It doesn't make any difference if you include uncertainty in the probability, because the assignment of uncertainty is itself uncertain.

 

[A. Neumaier]

That doesn't mean it is more accurate or more rational. The actual process is as much about psychology (reassuring all interested parties) as it is dealing with uncertainty.

Yes, as everywhere in life. Including in science.

You need to convince your peers that you did your study according to scientific standards. Claiming a probability based on two coin tosses will convince no one.

 

[Demystifier]

I'll never assign definite probabilities. probabilities are always estimates, with an intrinsic uncertainty.

OK, then just estimate probabilities, with the lowest possible accuracy, for the lowest possible number of coin flips. Can you perform such a lowest possible task?

 

[A. Neumaier]

the assignment of uncertainty is itself uncertain.

Indeed. There is no certainty about real numbers obtained from experiment. It is never like $\pi$ or $e$ or $\sqrt{2}$.

 

[A. Neumaier]

OK, then just estimate probabilities, with the lowest possible accuracy, for the lowest possible number of coin flips. Can you perform such a lowest possible task?

Play this game by yourself, using a book on elementary statistics!

 

[stevendaryl]

OK, then just estimate probabilities, with the lowest possible accuracy, for the lowest possible number of coin flips. Can you perform such a lowest possible task?

You know how it's really done. People pick a cutoff $C$-- 5% or 1% or 0.1% or whatever. Then they flip the coin often enough so that they can say:

If the probability of heads were outside of the range $p \pm \delta p$, then the probability of getting these results would be less than $C$.​

So, relative to the cutoff choice $C$, they conclude that the probability is in the range $p \pm \delta p$. That's not actually what you want to know, but it's the best you can get, using frequentist methods.

 

[Demystifier]

Play this game by yourself, using a book on elementary statistics!

Are you a politician? (Never giving a direct answer to a tricky question.)

 

[A. Neumaier]

to a tricky question

To a physically meaningless question.

 

[Demystifier]

That's not actually what you want to know, but it's the best you can get, using frequentist methods.

Sure, but I want to force Neumaier to admit that sometimes Bayesian methods make more sense. But he's smart, he understands what I am trying to do, so that's why he doesn't answer my questions.

 

[stevendaryl]

Sure, but I want to force Neumaier to admit that sometimes Bayesian methods make more sense. But he's smart, he understands what I am trying to do, so that's why he doesn't answer my questions.

A way to win any argument is to declare any question that you don't have a good answer for "meaningless". I'm not saying that's what Neumaier is doing, but sometimes it seems that way.

 

[Demystifier]

To a physically meaningless question.

It's meaningful for practical purposes.

 

[A. Neumaier]

It's meaningful for practical purposes.

Only for this practical purpose:

to force Neumaier to admit that sometimes Bayesian methods make more sense

I prefer to spend my time on more interesting topics.

 

[Demystifier]

Only for this practical purpose:

I prefer to spend my time on more interesting topics.

Trying to trick you is fun and challenging, but not practical.
I head some really practical purposes in mind. For instance, assigning p(head)=1/2 with N=0 coin flips may be practical in many actual situations. For instance, I used that once for playing a strip game with a girl, but that's another story. (In short she used intuition while I used Bayesian probability, and the final result was ... well, more fun than practical. )

 

[stevendaryl]

The Bayesian criticism of frequentist methods is that what you want to know is: "What is the probability that my hypothesis is true, given these experimental results?" but all that frequentism can tell you is "What is the probability of getting these experimental results, under the assumption that my hypothesis is true?" The two are of course related through Bayes' formula, but only if you allow for there to be prior probabilities for non-statistical facts.

 

[vanhees71]

You know how it's really done. People pick a cutoff $C$-- 5% or 1% or 0.1% or whatever. Then they flip the coin often enough so that they can say:

If the probability of heads were outside of the range $p \pm \delta p$, then the probability of getting these results would be less than $C$.​

So, relative to the cutoff choice $C$, they conclude that the probability is in the range $p \pm \delta p$. That's not actually what you want to know, but it's the best you can get, using frequentist methods.

Well, what else can you do to test a hypothesis than to just use statistics to do this test? Which other way to test a probabilistic statement do you think you can do, if not repeating the measurement on a sufficiently large ensemble? I've never understood these "Bayesian" statements that the probabilities of the Kolomogorov axioms have another meaning to real-world phenomena than a statistical analysis in the spirit of the frequentist interpretation.

There are ways to "guess" probabilities given some limited information about the phenomena at hand like the maximum-entropy method using the Shannen-Jaynes (or for QT von Neumann) entropy as a measure of relative missing information given a probability distribution against a prior distribution. Nevertheless, to be sure to estimate the probabilities successfully you have to check it, and this is possible only in the frequentist spirit. Then you gain more information about the system.

On top of all these purely statistical analyses you have to keep in mind that the real art of experimental science is not the statistical analysis but the analysis of systematic uncertainties too! This is not solvable with pure mathematics but needs further tests and reference measurements etc. etc. Science is much more complicated and plagued by much more uncertainties than pure math!

 

[vanhees71]

A way to win any argument is to declare any question that you don't have a good answer for "meaningless". I'm not saying that's what Neumaier is doing, but sometimes it seems that way.

That's then called "philosophy", right? SCNR

 

[stevendaryl]

Well, what else can you do to test a hypothesis than to just use statistics to do this test?

I'm not saying that you can do anything else, but in terms of conditional probabilities, what you're doing is: (letting $H$ be your hypothesis, and letting $R$ be your results)

If $P(R | H) < C$, then the hypothesis is unlikely to be true.

That is just not logically valid reasoning. What you want is something like

If $P(H | R) < C$ then the hypothesis is unlikely to be true.

The difference between these is illustrated with tests for cancer.

Suppose that you have a test for some rare type of cancer (affecting 1 in a million people). The test is 99% accurate. If a patient tests positive, is it likely that he has cancer? No, it's actually very unlikely. The probability that the test is wrong is much more likely than the probability that he has cancer.

Now, in this case, we actually already have a known prior probability of the hypothesis (that the patient has cancer) being true: 0.0001% But using $P(R|H)$ as a stand-in for $P(H|R)$ is just as fallacious if we don't have a prior probability of the hypothesis.

 

[atyy]

Regarding the Jaynes maximum entropy or whatever - in what sense is the entropy unique? For example, why should we use the Shannon entropy? Why not some other Renyi entropy? If there is no uniqueness, can the objective Bayes position be sustained?

 

[atyy]

Sure, but I want to force Neumaier to admit that sometimes Bayesian methods make more sense. But he's smart, he understands what I am trying to do, so that's why he doesn't answer my questions.

I thought I already did that. Neumaier wants to be coherent. Only subjective Bayesians can be coherent. Which is why I'm a frequentist, so I can be incoherent.

 

[stevendaryl]

That's then called "philosophy", right? SCNR

Well, there is a distinction between "A. Neumaier does not have a good answer to the question" and "It is impossible to answer the question"

 

[N88]

Well, there is a distinction between "A. Neumaier does not have a good answer to the question" and "It is impossible to answer the question"

Let A = A. Neumaier does not have a good - as judged by stevendaryl - answer to the question.
Let B = It is impossible to answer the question.
In Bayesian terms: P(AB) = P(A).P(A|B) = 1?

 

[Demystifier]

I think I have a simple way to resolve the frequentist vs Bayesian probability dilemma in physics. The rule is this:
1) To measure probability, you need frequentist probability.
2) To predict probability by theoretical methods, you need Bayesian probability.

Of course, true physics is always an interplay between experiment and theory, so true probability in physics is an interplay between frequentist and Bayesian. With such a view, I don't see any reason for further fight between the two philosophies. This is like fighting on whether physics is experimental or theoretical science, while it should be obvious that it is both.

 

[atyy]

I think I have a simple way to resolve the frequentist vs Bayesian probability dilemma in physics. The rule is this:
1) To measure probability, you need frequentist probability.
2) To predict probability by theoretical methods, you need Bayesian probability.

Of course, true physics is always an interplay between experiment and theory, so true probability in physics is an interplay between frequentist and Bayesian. With such a view, I don't see any reason for further fight between the two philosophies. This is like fighting on whether physics is experimental or theoretical science, while it should be obvious that it is both.

Did you take into account that at the operational level, the Frequentist results can be derived from Bayesian probability via the assumption of exchangeability?

http://www.stats.ox.ac.uk/~steffen/teaching/grad/definetti.pdf

 

[Demystifier]

Did you take into account that at the operational level, the Frequentist results can be derived from Bayesian probability via the assumption of exchangeability?

Isn't that just a special case of the idea that results of measurements can be derived from the theory?

 

[atyy]

Isn't that just a special case of the idea that results of measurements can be derived from the theory?

No, I don't think so. Exchangeability allows one to derive operations which in Frequentist practice assume iid. Here is another explanation http://www.cs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture1.pdf

 

[Demystifier]

I think I have a simple way to resolve the frequentist vs Bayesian probability dilemma in physics. The rule is this:
1) To measure probability, you need frequentist probability.
2) To predict probability by theoretical methods, you need Bayesian probability.

Of course, true physics is always an interplay between experiment and theory, so true probability in physics is an interplay between frequentist and Bayesian. With such a view, I don't see any reason for further fight between the two philosophies. This is like fighting on whether physics is experimental or theoretical science, while it should be obvious that it is both.

With that insight, I think I can give a better answer to the initial question of this thread. The question was the meaning of the ensemble in statistical physics. Well, the concept of an ensemble is an experimental concept. On the other hand, statistical physics is a branch of theoretical physics. This suggests that one should not use ensemble in statistical physics. But somebody used to think in experimental terms may feel uneasy without ensembles, so he may want to restore ensembles in theoretical statistical physics. But then he must proclaim that the ensemble is a virtual imagined ensemble, not a real ensemble, and this is what creates confusion. What a virtual ensemble really means? How is it related to a real ensemble? Should I replace virtual ensemble average with a time average, so that the ensemble becomes more real? Do I need ergodic theorems for that? Or perhaps only quasi-ergodic ones? What is the relevant time scale that makes (quasi)ergodic theorems applicable?

Yes, it is possible to answer such questions, but it's not that simple. For that reason I think that statistical physics, as a branch of theoretical physics, is easier to formulate without ensembles, using only probability in the Bayesian form as primitive. Frequentist probability is needed too, but only as a derived concept, relevant for making measurable predictions.

 

[Demystifier]

No, I don't think so. Exchangeability allows one to derive operations which in Frequentist practice assume iid. Here is another explanation http://www.cs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture1.pdf

So do you think that this influences my conclusions in #133?

 

[vanhees71]

Regarding the Jaynes maximum entropy or whatever - in what sense is the entropy unique? For example, why should we use the Shannon entropy? Why not some other Renyi entropy? If there is no uniqueness, can the objective Bayes position be sustained?

Exactly that's my point. Whatever you do to make a better educated guess about probabilities than just making up something, to be sure you have to do experiments with sufficiently large ensembles. Of course, you use Bayes's formula for conditional probabilities, but that doesn't mean that you are a Bayesianist denying the frequentist interpretation of probabilities.

The Shannon entropy is so much dominant, because it's the right entropy for the usual kinetics with short-range interactions, where the entropy is additive, i.e., you don't have higher correlations. It's valid for gases of neutral particles, because there you have van der Waals interactions with potentials falling much faster then the inverse distance of the particles. It's even valid for plasmas, where the long-ranged Coulomb potential between the particles is screened (Debye screening), and you can use the Vlasov equation to push the long-range correlations in the mean field and keep the collision term with short-ranged interactions of an effective screened potential. This is not valid anymore, e.g., when simulating the structure formation in the universe, because the gravitational interaction is not screened, and there Shannon entropy is the wrong measure. So it depends on the underlying microscopic dynamics which kind of statistics leads to good descriptions. The more important are experimental validations of such models, and again you can only achieve this by statistics. In the universe you assume the cosmological principle and then average over large enough distances, because here of course you cannot "prepare" an ensemble of universes.

 

[vanhees71]

With that insight, I think I can give a better answer to the initial question of this thread. The question was the meaning of the ensemble in statistical physics. Well, the concept of an ensemble is an experimental concept. On the other hand, statistical physics is a branch of theoretical physics. This suggests that one should not use ensemble in statistical physics. But somebody used to think in experimental terms may feel uneasy without ensembles, so he may want to restore ensembles in theoretical statistical physics. But then he must proclaim that the ensemble is a virtual imagined ensemble, not a real ensemble, and this is what creates confusion. What a virtual ensemble really means? How is it related to a real ensemble? Should I replace virtual ensemble average with a time average, so that the ensemble becomes more real? Do I need ergodic theorems for that? Or perhaps only quasi-ergodic ones? What is the relevant time scale that makes (quasi)ergodic theorems applicable?

Yes, it is possible to answer such questions, but it's not that simple. For that reason I think that statistical physics, as a branch of theoretical physics, is easier to formulate without ensembles, using only probability in the Bayesian form as primitive. Frequentist probability is needed too, but only as a derived concept, relevant for making measurable predictions.

What's the "probablity in the Bayesian form"? In statistical mechanics you always "coarse grain" over many microscopic details to describe the relevant classical observables. E.g., you can take fluid mechanics, where you start with (quantum) many-body theory and derive transport equations, given a separation of spatial and temporal scales, i.e., microscopic high-frequent and short-ranged fluctuations you average over by your observations (measurement apparati) to get sufficiently detailed description for the macroscopic low-frequent and long-ranged changes of macroscopic variables. Here the phase-space cells must be large enough to lead to a sufficient averaging and "smoothing" of the highly fluctuating microscopic distributions to macroscopic phase-space distributions, and you have to truncate the practically infinite tower of $N$-body phase-space distribution functions (BBGKY Hierachy). On the level of transport equations it's usually the single-particle phase-space distribution function you are after, and this quantity makes sense only in this coarse-grained definition to begin with. Formally it can be achieved by gradient expanding the Wigner function, which is not a phase-space distribution but becomes one if appropriately coarse grained. Another step of simplification is to switch to hydrodynamics rather than transport descriptions, where you go in the limit that the mean-free path is much smaller than the typical variations of the usual macroscopic quantities like density, pressure, flow-velocity field, etc. Then you can assume local thermal equilibrium (leading to ideal fluid dynamics) and expansions (in terms of moments or Chapman-Enskog, etc.) (leading to various levels of viscous fluid dynamics, in the next linear order of deviations from local thermal equilibrium the Navier-Stokes equation).

All this uses more or less explicitly the frequentist interpretation too, and the "ensembles" are the states of a partial system, i.e., "microscopically large" but "macroscopically small" large region of space (and sometimes also time), "smearing" over the microscopic fluctuations and details to get macroscopic quantities in terms of such averages. These averages in practice are done by the measurement apparati themselves, and that's what statistical physics also does in the theoretical description/modeling. So there is no dichotomy between experiment and theory in statistical physics!