Exploring the Cosmological Constant Problem & Zero-Point Energy

[AndreasC]

Reading the Wikipedia page on it, one reads:

In cosmology, the cosmological constant problem or vacuum catastrophe is the disagreement between the observed values of vacuum energy density (the small value of the cosmological constant) and theoretical large value of zero-point energy suggested by quantum field theory.

But on the other hand, as far as I know and if I'm not mistaken, zero point energy is not a physical thing, and it is merely a mathematical artifact in QFT. Someone correct me if I'm wrong on that. So if that is the case, then why is it a "problem" that it does not "agree" with the cosmological constant? Isn't trying to plug it into general relativity to calculate the cosmological constant just a non-sensical idea to begin with? It certainly seems so to given that the calculations are off by I don't know how many orders of magnitude (I found a paper claiming that the typical figures of 120 orders of magnitude deviation are not correct, and it is instead "only" about 60. It doesn't seem that much better to me).

 

[PeroK]

The observed vacuum energy must be due to something. It ought to be calculable, not just some unjustifiable observed quantity.

 

[AndreasC]

The observed vacuum energy must be due to something. It ought to be calculable, not just some unjustifiable observed quantity.

But then the problem is that we don't know what it is caused by, not the fact that it does not agree with the calculation of vacuum energy, right? I don't see why that calculation is relevant to the problem if we do not expect vacuum energy to be physical and thus have anything to do with the cosmological constant to begin with... That is why I'm confused about why it comes up.

 

[PeroK]

But then the problem is that we don't know what it is caused by, not the fact that it does not agree with the calculation of vacuum energy, right? I don't see why that calculation is relevant to the problem if we do not expect vacuum energy to be physical and thus have anything to do with the cosmological constant to begin with... That is why I'm confused about why it comes up.

Okay, but don't you think it was worth trying? Just to see?

 

[AndreasC]

Okay, but don't you think it was worth trying? Just to see?

Well, I guess maybe but it seems like there is a lot of emphasis put on that still, and many people still seem to believe they are somehow connected... The Wikipedia article I quoted has some references which seem to imply there is still research claiming that, case in point the paper I mentioned: https://arxiv.org/abs/1205.3365

This paper (which I may be misunderstanding) also seems to talk about quantum vacuum gravitating: https://arxiv.org/abs/1703.00543

So this is why I'm confused.

 

[phyzguy]

Isn't trying to plug it into general relativity to calculate the cosmological constant just a non-sensical idea to begin with?

Yes!

 

[AndreasC]

Yes!

Well then the question becomes, why does this keep getting brought up? Why do so many physicists seemingly treat these as being related?

 

[phyzguy]

Well then the question becomes, why does this keep getting brought up? Why do so many physicists seemingly treat these as being related?

Well, since the source of the cosmological constant isn't known, people keep trying to understand. As @PeroK said, it was worth a try. Maybe it's on the right track and we just dropped a factor of 10^120 somewhere. You might be interested in this paper, "

Why all these prejudices against a constant?"​

https://arxiv.org/abs/1002.3966v3

 

[PeterDonis]

zero point energy is not a physical thing, and it is merely a mathematical artifact in QFT

That depends on which physicist you talk to. There's no way to prove or disprove such statements theoretically; ultimately any such theoretical claim has to be tested against observations. Our best current observations indicate that our universe has a small positive cosmological constant. GR itself allows for a nonzero cosmological constant but makes no prediction at all about what its value should be; so the only theoretical framework we have at present for even investigating the question is quantum mechanics.

 

[PeterDonis]

trying to plug it into general relativity to calculate the cosmological constant

I'm not sure what you mean by this. You don't need to "plug" anything into GR to try to estimate from QM what the energy density of the vacuum should be. And the fact that a nonzero cosmological constant is identical in the Einstein Field Equation to a nonzero energy density of the vacuum is obvious.

Isn't trying to plug it into general relativity to calculate the cosmological constant just a non-sensical idea to begin with?

Yes!

Why? It's not nonsense that the cosmological constant can be nonzero in GR. It's not nonsense that a nonzero cosmological constant is the same, from the standpoint of the Einstein Field Equation, as a nonzero energy density of the vacuum. And it's not nonsense that QM can be used to at least try to estimate the energy density of the vacuum. So what, exactly, are you saying is nonsense here?

 

[phyzguy]

So are you saying your chain of:
Nonzero cosmological constant -> Nonzero vacuum energy density -> QM vacuum fluctuations
makes "sense"? If so, why is it off by 120 orders of magnitude? It seems to me that any reasonable person would discard it as "nonsense" when the agreement is so bad.

 

[PeterDonis]

So are you saying your chain of:
Nonzero cosmological constant -> Nonzero vacuum energy density -> QM vacuum fluctuations
makes "sense"?

I said nothing about "QM vacuum fluctuations".

why is it off by 120 orders of magnitude?

We don't know. But we also don't know how to do the calculation exactly anyway; all of the numbers quoted are approximations, invoking things like ad hoc cutoffs. Obviously we don't have a full understanding of the underlying theory. But that doesn't mean the idea is nonsense.

 

[PeterDonis]

It seems to me that any reasonable person would discard it as "nonsense" when the agreement is so bad.

It seems to me that any reasonable person would want to look at what other models we have before discarding the QM model as "nonsense". So what other models do we have to explain a nonzero cosmological constant?

 

[Vanadium 50]

Is the zero-point energy in the simple harmonic oscillator "real" or "not a physiacl thing"? Before discussing grand and grandiose things like The Entire Universe, shouldn't we get this straight? At a minimum, it tells us that the issue is present in much simpler systems.

Next, there is absolutely no calculation of the cosmological constant from QM. Nada. Zilch. Zip. What we have is an order of magnitude estimate and some plausible numbers as inputs to this estimate, but this is not a calculation. "We don't have a calculation, but if we did, I'm sure it would be too big" is a very different kind of "problem" than "these numbers should be equal and aren't" (or "these numbers have no reason to be equal but are")

 

[TeethWhitener]

Weinberg’s 1988 Rev Mod Phys paper is relevant here and gives a really good (IMO) exposition of the issue:
https://repositories.lib.utexas.edu/bitstream/handle/2152/61094/Weinberg_1989.pdf;jsessionid=F306B9DD843B1834614FC774EC08CBE9?sequence=1
I’m not an expert, so I don’t know what kinds of advances have been made in the past 30 odd years since this paper was written.

 

[strangerep]

It seems to me that any reasonable person would want to look at what other models we have before discarding the QM model as "nonsense".

The so-called dark energy associated with the cosmological constant also has a rather unusual equation of state (EoS) -- very different from all known matter and radiation fields. Hence, it seems unlikely that the VeV of these known fields would have such a different EoS. The "problem" is not just the size of estimated vacuum energy, but also the weird EoS required.

So what other models do we have to explain a nonzero cosmological constant?

Umm, a universal constant?

 

[PeterDonis]

The so-called dark energy associated with the cosmological constant also has a rather unusual equation of state (EoS) -- very different from all known matter and radiation fields.

This equation of state is also the only EoS that is consistent with a nonzero energy density of the vacuum, since any such energy density must be Lorentz invariant. That's why a nonzero energy density of the vacuum is a viable possible explanation for dark energy.

a universal constant?

That isn't an explanation, it's just a bare assertion. It is, of course, possible that our universe has a nonzero cosmological constant simply because it does, with no deeper explanation possible. But we are in no position to just assert this when there is so much we still don't know about other possibilities.

 

[PeterDonis]

it seems unlikely that the VeV of these known fields would have such a different EoS.

Actually, the VeV of a scalar field, like the Higgs field, has a similar EoS, correct?

 

[strangerep]

Actually, the VeV of a scalar field, like the Higgs field, has a similar EoS, correct?

Reference please?
(Maybe I've been away from particle physics for too long.)

 

[PeterDonis]

Reference please?

The general EoS of a scalar field is $\rho = K + V$, $p = K - V$, where $K = \left( \partial \phi \right)^2$ is the kinetic energy and $V$ is the potential energy. For a constant scalar field, which will be the case if the field is in its vacuum state with a nonzero VEV, $K = 0$ and we have $p = - \rho$, which is the vacuum energy EoS. (This is the same general argument that is invoked for the case of slow roll inflation.) See, for example, here:

https://supernova.lbl.gov/~evlinder/umass/sca.html

 

[AndreasC]

And the fact that a nonzero cosmological constant is identical in the Einstein Field Equation to a nonzero energy density of the vacuum is obvious.

That is what I mean. If the energy of the quantum vacuum or zero point energy or fluctuations or whatever is considered to not be physical, why do people try to make this calculation? Unless it is physical which doesn't seem to be at all universally accepted.

 

[AndreasC]

Is the zero-point energy in the simple harmonic oscillator "real" or "not a physiacl thing"? Before discussing grand and grandiose things like The Entire Universe, shouldn't we get this straight? At a minimum, it tells us that the issue is present in much simpler systems.

I'm not at home so I can't check it right now but I remember doing the quantization of the free electromagnetic field via assigning operators to momenta and positions in the classical theory in a slightly different way such that the weird infinite zero point energy of the field disappears. I don't remember what I did exactly, I think I wrote p^2+x^2 as a difference of the squares of p and ix (which is a valid and entirely equivalent way to express the problem classically obviously) and then replaced these with the operators, and used the fact they do not commute (unlike the classical case) to get a slightly different Hamiltonian that does not have that problem. I think the same happens if you write it as (p+x)^2-2px and do the same thing. If it can be removed like this, it seems to me like it's probably not really a physical thing. Unless there is somewhere else where it crops up that I don't know about. But isn't the zero point energy in general and the quantum vacuum energy different things anyways?

 

[haushofer]

Maybe it has been mentioned, but apart from trying the naive hunch this vacuum energy calculation has the peculiar property that it' a UV-divergence with a long distance (IR) effect. So it defeats Wilson's "divide and conquer energy scales"-philosophy in effective field theories. That's an interesting lesson on its own.

 

[AndreasC]

To expand on what I was trying to say before about the harmonic oscillator, let's try to quantize the simple harmonic oscillator. The classical hamiltonian is (setting all constants to 1) $H=p^2/2+x^2/2$. If you try to quantize it now by replacing p and x with the associated quantum operators, and you then rewrite them using the ladder operators, you get:
$$H= [i(a^\dagger-a)/\sqrt{2}]^2/2+[(a^\dagger+a)\sqrt{2}]^2/2 = a^\dagger a - [a^\dagger, a]/2 = a^\dagger a + 1/2 $$
This can be written using the number operator:
$$ H = N+1/2 $$
So there appears to be a "zero point energy", this 1/2 term. But if instead we wrote the classical hamiltonian as $H=(p-ix)(p+ix)/2$, which seems every bit as valid to me, and then you quantize, it is obvious that this is just: $$H=a^\dagger a = N$$ Thus the zero point energy goes away. This procedure can be used to remove the infinity in the quantization of the free EM field, and if I recall correctly these annoying zero point energy terms also appear in statistical physics calculations where they cause issues, and they can be removed as such. So regarding the question of whether the simple harmonic oscillator zero point energy is physical, I think the answer is probably not, unless I'm missing something.

 

[AndreasC]

Maybe it has been mentioned, but apart from trying the naive hunch this vacuum energy calculation has the peculiar property that it' a UV-divergence with a long distance (IR) effect. So it defeats Wilson's "divide and conquer energy scales"-philosophy in effective field theories. That's an interesting lesson on its own.

Hmm, do you have a reference that talks more about that? Otherwise, can you expand on it a bit?

 

[gggnano]

Then what causes the casimir pressure, if you do the math on abysmal distance of a distance a little over the Planck scale the pressure is literally higher than the pressure inside a neutron star! I'm not saying the Casimir formula is ideal though. Yet: what causes spontaneous emission or Lamb shift? It's like saying that gravity is artifact because it's many times weaker than the other 3 forces.
Ruling out quantum fluctuations, ex nihil, will also rule out another interesting idea: the Bolzmann brain, which is unfalsifiable if you think about it but still interesting thought experiment.

 

[PeterDonis]

If the energy of the quantum vacuum or zero point energy or fluctuations or whatever is considered to not be physical, why do people try to make this calculation?

Who said the people trying to make the calculation did not consider the energy density of the vacuum to be physical?

Unless it is physical which doesn't seem to be at all universally accepted.

That's not at all the same as it not being accepted at all. It should be obvious that there are some physicists who think it is physical and some who think it is not, and since we don't have a good underlying theory that makes a prediction about its value that matches what we observe, this area of research is still open.

 

[PeterDonis]

if instead we wrote the classical hamiltonian as $H=(p-ix)(p+ix)/2$, which seems every bit as valid to me, and then you quantize, it is obvious that this is just: $$H=a^\dagger a = N$$

No, it isn't, because $p$ and $x$ don't commute. So you have

$$
H = \left( p^2 - i x p + i p x + x^2 \right) / 2 = \left( p^2 + x^2 + i [p, x] \right) / 2 = N + \frac{1}{2}
$$

 

[AndreasC]

Then what causes the casimir pressure

There are a lot of papers claiming zero point energy is not required to explain that. I actually found tons of discussion in this forum on the subject, just search for Casimir effect physicsforums and you will see. About Lamb shift etc I don't know much.

 

[AndreasC]

No, it isn't, because $p$ and $x$ don't commute. So you have

$$
H = \left( p^2 - i x p + i p x + x^2 \right) / 2 = \left( p^2 + x^2 + i [p, x] \right) / 2 = N + \frac{1}{2}
$$

Check your calculation again, the ladder operators are p-ix and p+ix, both divided by the square root of 2. The number operator N is their product. It is exactly because they don't commute that this result is different from the same result with just p^2 and x^2.

To be more clear:
$$ H = (p-ix)(p+ix)/2 = \frac{(p-ix)}{\sqrt{2}} \frac{(p+ix)}{\sqrt{2}} = a^\dagger a = N $$

 

[AndreasC]

I am correcting my previous post, since I made a small error regarding the definitions of the operators. The ladder operators are (x+ip) and (x-ip). My point still stands as long as we look at the Hamiltonian:
$$ H = (x-ip)(x+ip)/2 $$
...And carry out the same procedure. The end result is indeed N. The calculation @PeterDonis was trying to carry out gives $H=(p^2+x^2+i[x,p])/2$, where a factor $i[x,p]/2=-1/2$ is added to the previous Hamiltonian, and thus eliminates the 1/2 "zero point energy" term, which definitely seems to solve many issues for me.

Now that I think about it, we could actually have a little more fun with it and come up with whatever "zero point energy" we like, by noticing that terms involving $(x-ip)(x+ip)/2$ add 0 to the number operator, while terms involving $(x+ip)(x-ip)/2$ add 1, yet classically they are all the same. One could write the Hamiltonian as:
$$ H = c(x+ip)(x-ip)/2 + (1-c)(x-ip)(x+ip)/2 $$
Which of course classically is just p^2+x^2, but when you quantize, you get N+c. So I wouldn't take the harmonic oscillator seriously, to answer the question of @Vanadium 50 .

 

[gggnano]

There are a lot of papers claiming zero point energy is not required to explain that. I actually found tons of discussion in this forum on the subject, just search for Casimir effect physicsforums and you will see. About Lamb shift etc I don't know much.

There are. But don't forget that in Michael Faraday's time no one was able to explain the simple experiment of a dipole over a solenoid and hence the name "Faraday paradox"...then they discovered the electron...

I'm not saying vacuum energy is indeed what Feynman and others claim to be of the absurd values of over 10^60 joules/m3 (thought I'm not complaining if we can use even 0.000000000001% of such energy...) yet I'm hoping for explanations better than "it's an error of measurement", "magnets cause it" or "attraction between atoms causes it".

The pressure is 1.3Gpa which is about the same pressure when the first human-made diamond was synthesized! You get ridiculous values if you decrease it even more to a point I'd be curious what the temperature is on solid matter with such pressure, forget the energy:

https://www.wolframalpha.com/input?i=casimir+pressure+1nm

 

[gggnano]

Check your calculation again, the ladder operators are p-ix and p+ix, both divided by the square root of 2. The number operator N is their product. It is exactly because they don't commute that this result is different from the same result with just p^2 and x^2.

To be more clear:
$$ H = (p-ix)(p+ix)/2 = \frac{(p-ix)}{\sqrt{2}} \frac{(p+ix)}{\sqrt{2}} = a^\dagger a = N $$

By the way, I haven't read your discussion with the other user but the Hamiltonian of a particle in QM can be expressed as:

(1/(2m))*p^2+0.5*m*w^2*x^2

And maybe you're trying to show that it has a root for m different that zero over the real line, that is p is zero and w is zero too...but I'm not sure what's the point since this is a rule that electrons or photons or any real particle should obey?? Remember, virtual particles exist for t < Planck-time and thus can violate everything including photons that have mass.

 

[AndreasC]

By the way, I haven't read your discussion with the other user but the Hamiltonian of a particle in QM can be expressed as:

(1/(2m))*p^2+0.5*m*w^2*x^2

And maybe you're trying to show that it has a root for m different that zero over the real line, that is p is zero and w is zero too...but I'm not sure what's the point since this is a rule that electrons or photons or any real particle should obey?? Remember, virtual particles exist for t < Planck-time and thus can violate everything including photons that have mass.

The point is that @Vanadium 50 said that the matter of the simple harmonic oscillator and its zero point energy should be settled first before tackling something more grandiose, like the entire universe. My conclusion so far in absence of further evidence is that in the case of the oscillator it is unphysical, since it only appears as a consequence of quantizing the classical field in a specific way which is ambiguous and far from unique.

Specifically, I took the Hamiltonian you mentioned and set all the constants to 1 for ease, then showed that when you replace x and p with operators, you can get whatever "zero point energy" you want by making algebraic manipulations to the form classical equation that don't change its essence at all.

 

[PeterDonis]

Check your calculation again

If you get two different answers for the same Hamiltonian by two different methods, one of them must be wrong. The result $H = a^\dagger a + 1/2$ is the one that is accepted as correct in the literature.

The ladder operators are (x+ip) and (x-ip).

(More precisely, there should be a $1 / \sqrt{2}$ factor on each one, as you state later on in your post.)

If that is the case, then we have $H = a^\dagger a + 1/2 = \left( x + ip \right) \left( x - ip \right) /2 + 1/2$, not $H = \left( x + ip \right) \left( x - ip \right) / 2$.

The ladder operators are (x+ip) and (x-ip). My point still stands as long as we look at the Hamiltonian:
$$ H = (x-ip)(x+ip)/2 $$

And if your definition of the ladder operators is correct, then this is the wrong Hamiltonian. See above.

In short, it is not possible for all of the claims you are making to be true, since they are not all consistent with each other. Your original calculation of the Hamiltonian in terms of the ladder operators, which is the one that ends up with $H = a^\dagger a + 1/2$, is the one that is accepted as correct in the literature. So your alternative calculation is making a mistake in one of your claims.