Dark energy questioning Relativity?

[PhysicalAardvark]

TL;DR Summary

Dark energy seems to have been invented because observations don't fit in with General Relativity. So why is the theory itself never questioned?

Forgive my naive understanding of these topics. I have a layman's interest in science and follow all the popular science I can, but I'm certainly not a physicist and even my degree in mathematics is now all but forgotten in the past.

But I have two questions (the other I'll post in another thread) about astrophysics as presented in the popular press that I can't get out of my head. I wondered if you folks could save me a lot of time and probably futile research, as these topics are most likely way beyond my current understanding.

All the reasons for the (proposed) existence of Dark Matter (and Dark Energy) seem to be based on gravitational effects. Galactic clustering. Galactic rotation. The CMB. Large-scale clumping. These observations don't fit with our current understanding of gravitation (General Relativity) so Dark Matter is invented, as this seems to explain things.

Is that a fair summary?

In that case, what about the famous explanation of the Scientific Method by Richard Feynman: "If it disagrees with experiment, [the theory is] wrong. In that simple statement is the key to science."

I buy into the whole 'General Relativity is great, it predicts everything and is the best model we have' etc. shtick, but Newton was completely valid up until someone noticed something funny about the orbit of Mercury and then GPS and then galactic lensing and then black holes crashing into each other and so on.

Something like Dark Matter seems invented despite complete lack of evidence for it and I'm kind-of fine with that. But why in light of the above is the possibility that General Relativity may simply not be correct/sufficient in some cases never discussed at all?

Thanks,

P

 

[Nugatory]

But why in light of the above is the possibility that General Relativity may simply not be correct/sufficient in some cases never discussed at all?

There's an analogy with the observed orbit of Uranus. By the early 19th century Newtonian gravity had proven exquisitely successful at calculating everything from the orbits of the heavenly bodies to the trajectories of dropped or thrown objects. However, it failed to properly describe the orbit of Uranus. Thus, there were two possibilities:
A) Newtonian gravity is not correct/sufficient. We should put our intellectual energy into discovering a new theory that agrees with Newtonian gravity everywhere that Newtonian gravity works but incorporates some as-yet-undiscovered new physics that is unnoticeable near the sun but at greater distances has exactly the right effect on the orbit of Uranus.
B) There's nothing wrong with Newtonian gravity. We've just left something out of our calculations, and we should put our intellectual energy into finding that something.

Phrased that way, it's clear that #A was a wild longshot, not worth much discussion unless and until #B could be excluded. So astronomers hypothesized that there was some undiscovered mass out here, calculated where it would have to be to affect the orbit of Uranus, and then in 1846 pointed their telescopes at that spot... And there was the planet Neptune!
(This is, of course, the same thought process you go through when you pick up a box and find it harder to lift than you expected. Your first thought is not that Newtonian gravity is wrong about the Earth's gravitational pull on the box, it is that there's something heavy that you don't know about in the box)

With regard to dark matter and dark energy (which, despite sharing the word "dark" are completely different things) right now we're at the "finding that something" stage. We can assume that general relativity is wrong, or we can assume that we haven't observed everything that's out there to observe; and the latter looks like a much better bet.

 

[Orodruin]

On the other hand ...

**Edits in bold**
There's an analogy with the observed orbit of Uranus. By the early 20th century Newtonian gravity had proven exquisitely successful at calculating everything from the orbits of the heavenly bodies to the trajectories of dropped or thrown objects. However, it failed to properly describe the orbit of Mercury. Thus, there were two possibilities:
A) Newtonian gravity is not correct/sufficient. We should put our intellectual energy into discovering a new theory that agrees with Newtonian gravity everywhere that Newtonian gravity works but incorporates some as-yet-undiscovered new physics that is unnoticeable near the sun but at greater distances has exactly the right effect on the orbit of Mercury.
B) There's nothing wrong with Newtonian gravity. We've just left something out of our calculations, and we should put our intellectual energy into finding that something

The correct approach was A and led to general relativity.

 

[Dale]

Summary: Dark energy seems to have been invented because observations don't fit in with General Relativity. So why is the theory itself never questioned?

Newton was completely valid up until someone noticed something funny about the orbit of Mercury

Actually, the situation with Newtonian gravity is more similar to GR than you may realize. As the precision of measurements increased there were discrepancies noticed in the orbits of Jupiter and Saturn. Solving Newton’s equations for the known sources resulted in a small failure. So they had two options: modify the equations or modify the the sources.

The equations seemed to work well elsewhere so they tried predicting the sources that would explain the observed discrepancy. This led to the discovery of Neptune and Uranus and eventually Pluto and other objects.

A similar scenario happened with the discovery of the neutrino. A discrepancy in the conservation of momentum and energy was noticed in nuclear reactions. They could abandon the laws of conservation of momentum and energy, or look for unknown sources. The laws of conservation of energy worked well in other scenarios so they decided to look for unknown sources. Looking for unknown sources they found the neutrino.

If the laws of physics that are known to work well in many scenarios have a small discrepancy, the first thing to do is to look for additional sources. This is what the search for dark matter is.

Of course, as you mentioned, sometimes we need to change the theory, like in the case of Mercury. Unfortunately, it is not possible to know in advance whether a given discrepancy is a new-source discrepancy or a new-equation theory.

Certainly, when a new theory is developed we apply those equations to the discrepancy using the known sources. But this presupposes the existence of a new theory. GR was not developed to explain Mercury’s orbit, it arose due to other considerations. Currently we don’t have a viable alternative.

 

[Nugatory]

On the other hand ...

I was considering writing a footnote about Mercury's orbit for exactly that reason - it is both the same and different, in interesting ways.

The big difference is that the incompatibility between special relativity and Newtonian gravity was apparent and troublesome before the discovery of GR - we had two theories and they could not both be correct. Thus, there was reason to pursue the #A line of thought even in the absence of any anomalous observational data.

As the history played out, the anomalies in Mercury's orbit did not motivate general relativity (and it's not at all clear that one could arrive at GR by studying them). At the time there was no particular reason to think that the Mercury problem was related to the theoretical problem of reconciling special relativity and gravity. Instead, the flow went the other way: GR was discovered for other reasons and only then could we say "Wow - so that's what's going on with Mercury".

(Another footnote on the relationship between observation and new theory might be Fizeau's 1850-vintage measurements of the speed of light in a medium. These went unexplained until Einstein discovered special relativity a half-century later).

 

[Vanadium 50]

Summary: Dark energy seems to have been invented because observations don't fit in with General Relativity. So why is the theory itself never questioned?

It's a pity that the first responders here completely swallowed your premise, which is simply not true. Alternatives to GR are developed and considered all the time. There is so much work there needs to be review articles summarizing this work, e.g. Annu. Rev. Nucl. Part. Sci. 66:95-122 (2016) which references one-hundred-and-fifty-four papers.

Before we can discuss why something is true, we need to know if it is true.

It's not.

Something like Dark Matter seems invented despite complete lack of evidence for it

Untrue.

But why in light of the above is the possibility that General Relativity may simply not be correct/sufficient in some cases never discussed at all?

Untrue.

 

[Ibix]

There's also the Pioneer Anomaly, where some of our deep space probes were drifting off course. That turned out to be an unanticipated thrust effect from the vehicle's reactor interacting with the antenna. So a third possibility is some known physics not properly accounted for. No idea what kind of thing that might be in this case, but I mention it for completeness.

It's also worth noting that there are alternative models of (non-quantum) relativistic gravity. It's just that they keep getting ruled out by experiment. I seem to recall LIGO dealt a solid blow to a number of theories that allowed gravitational waves that aren't pure transverse-tensor waves, since the detected signals weren't consistent with other types of wave.

 

[Ibix]

It's also worth noting that there are alternative models of (non-quantum) relativistic gravity.

one-hundred-and-fifty-four papers.

I must admit I hadn't realized that there were quite that many...

 

[PeroK]

Summary: Dark energy seems to have been invented because observations don't fit in with General Relativity.

Your conclusion is false in another sense. GR, in its most general form, works with whatever energy sources the universe has. A simple application of the theory assumes energy from matter and radiation and assumes a zero energy of the vacuum.

But, one explanation for an accelerating expansion is that the vacuum has non-zero energy.

The theory of GR, per se, is unchanged. It's the addition of the vacuum as an energy source that leads to the accelerated expansion, by directly applying the equations of GR.

 

[Vanadium 50]

On the other hand ...

Yes, but the first hypothesis for the orbit of Mercury discrepancy was a new inner planet, Vulcan. Not crazy, since gravitational effects from other planets were 13x larger than the discrepancy: a small body is about the right magnitude. However, nothing was ever found here: people looked from 1843 (first calculation) to at least 1908.

 

[PeterDonis]

"If it disagrees with experiment, [the theory is] wrong. In that simple statement is the key to science."

"Experiment" is not the same as "observation". What Feynman meant by "experiment" here is something more like "observations and experimental data, after the observations and data have been thoroughly checked and we're sure they were all made correctly and that we've correctly accounted for all the observational and experimental errors, and the theoretical calculations have been thoroughly checked and we're sure there aren't any physical effects going on that we don't understand and we're sure that there aren't any other objects or sources out there that we haven't taken into account and we're sure that there aren't any other theoretical effects that we left out but should have been included, etc., etc., etc." If you read enough of Feynman, you will see him talking about all these pitfalls that come into play when you try to compare theory with experiment.

 

[PeterDonis]

All the reasons for the (proposed) existence of Dark Matter (and Dark Energy) seem to be based on gravitational effects. Galactic clustering. Galactic rotation. The CMB. Large-scale clumping. These observations don't fit with our current understanding of gravitation (General Relativity)

No, that's not correct. All of those effects fit in fine with our current understanding of gravity (GR), with the appropriate sources (stress-energy) present, as @PeroK pointed out in post #9. The issue is that the sources we have to include in the model to match the data aren't all visible to us; we can't detect some of them, the ones we call "dark matter" and "dark energy", by any means other than their gravitational effects.

Many scientists are basically OK with that; they view part of what they are doing as just figuring out which sources need to be included in the model to match all the data. If some of those sources aren't detectable currently by other means, that's to be expected; there's no reason why our detection accuracy has to be the same in all regimes. They view including such sources in the model as much more likely to be correct than trying to change the underlying theory of gravity.

Other scientists are not OK with that; they feel that we should only include in our models sources that we can detect by other means. So if those sources can't explain the data, they want to change the underlying theory of gravity.

Both lines of research are being pursued, so, as @Vanadium 50 has pointed out, it's not correct to say alternatives to GR are not being investigated. We simply have not reached the point where either line of research has conclusively been either established or ruled out. This is a perfectly normal thing to have in a scientific field.

 

[PhysicalAardvark]

It's a pity that the first responders here completely swallowed your premise, which is simply not true.
It's not.
Untrue.
Untrue.

Thanks for calling me out for an idiot without citing a single reference I can follow up to confirm this. All I've said is 'untrue' in your opinion, (and I'm surely hoping it is) but I have come no closer whatsoever to being able to believe you. Science isn't based upon derogatory opinions, last time I checked.

 

[Vanadium 50]

without citing a single reference

Annu. Rev. Nucl. Part. Sci. 66:95-122 (2016) which references one-hundred-and-fifty-four papers.

 

[PAllen]

Thanks for calling me out for an idiot without citing a single reference I can follow up to confirm this. All I've said is 'untrue' in your opinion, (and I'm surely hoping it is) but I have come no closer whatsoever to being able to believe you. Science isn't based upon derogatory opinions, last time I checked.

You were provided a reference, in contrast to your OP which contained none. If you don’t know how to check that reference, that is not @Vanadium 50 ’s fault. However, I can concur that his point is obviously true. Modifications of GR to possibly explain small discrepancies go way back before even dark energy ,or dark matter were ever proposed (using a date of circa 1980 as when the modern dark matter proposal started to be generally accepted). The first I personally was exposed to was Robert Dicke (look him up) giving talks on the possibility that scalar-tensor gravity was needed to match solar system observations if the oblateness of the sun was properly accounted for. This didn’t pan out, but he seriously pursued it for a few years, and I heard one of his talks. There is considerable research on modifying GR to replicate the MOND relation to explain galactic rotation curves. For dark energy, you really can’t talk about a modification, because GR has within it a term for dark energy (the cosmological constant), which previously was just set to zero for simplicity. The mystery is then outside of GR - why is the constant so small and positive when various quantum models suggest that its natural size should be enormous in magnitude, and possibly that a negative value should be preferred.

 

[PhysicalAardvark]

Ok; forget my last post. I was tired and couldn't read correctly. I was actually rather surprised to see any responses to my OP, as for a long time there weren't any, and I'd not been notified of any updates.

Thanks for your replies, all; I will endeavour to read and understand them as best as I can in due course. This isn't a subject I expect to understand overnight; but as with the 'Big Bang' question I posted, mainstream media isn't really helpful in getting an answer, so I thank you again for pointing me in the right direction.

 

[Dale]

without citing a single reference I can follow up

Unfortunately there are still many references that are behind paywalls. It makes it difficult for people outside a university to access the best research. One thing that you can do is to go to the abstract of the paywalled article and then get keywords and authors to search for in Arxiv.

 

[Vanadium 50]

Unfortunately there are still many references that are behind paywalls.

But in this case, if you type the reference into Google, the 2nd link is the free version of the article.

I can only point people to the reference. If they don't look, they don't look.

 

[vanhees71]

Unfortunately there are still many references that are behind paywalls. It makes it difficult for people outside a university to access the best research. One thing that you can do is to go to the abstract of the paywalled article and then get keywords and authors to search for in Arxiv.

Another very convenient tool is the "unpaywall plugin" available for various browsers. It links to preprint-versions and other legal open-access sources of articles published behind a paywall:

https://unpaywall.org/

 

[Dale]

Another very convenient tool is the "unpaywall plugin" available for various browsers. It links to preprint-versions and other legal open-access sources of articles published behind a paywall:

https://unpaywall.org/

Oh wow! That is fantastic, I didn’t know something like that existed

 

[PhysicalAardvark]

I must admit I hadn't realized that there were quite that many...

Ok, I think this is actually the 'real' answer to my question. There are alternatives being researched. The issue is with the media. Even in non-science media 'dark energy' and 'dark matter' are plastered everywhere. But the alternatives are never discussed, even in popular science media (that I'm aware of; and I'm quite keen on watching anything in this kind of area).

As someone not even close to the cutting edge of science to know otherwise, this false impression is very strong. To the extent that I didn't even consider looking for an alternative (my bad). Apologies for not exploring this more closely before asking here. It's now readily apparent even with a simple Google search that there are alternatives being researched.

I guess the question I should have asked is 'why are these alternatives never given as much air-time as the dark-{x} theories.' And that's a media question; not a scientific one. I guess 'dark energy' is just way more sexy than 'scalar-tensor-vector gravity' or similar.

On the other hand, it did raise an interesting discussion and raise my self-awareness a little bit, which simply reading a completely incomprehensible Wikipedia article wouldn't have done. So thanks for that.

P

 

[Dale]

I guess the question I should have asked is 'why are these alternatives never given as much air-time as the dark-{x} theories.' And that's a media question

This is, as you say, a question for the media which is not my field of expertise. I don’t know exactly what kind of “behind the scenes” constraints they operate under. But I get the impression very often that they don’t actually understand the papers they write about. As a result they go to sources written by other journalists. This then leads to the effect you noticed: the dominant point of view seems to be the only point of view

 

[vanhees71]

Often I have problems to understand what popular-science writers want to say. I have to look at the scientific papers to get an idea. That's somehow sad, because popular-science writing should inform us about scientific topics we are not experts in and thus about topics we can't understand the scientific papers.

 

[ZapperZ]

Ok, I think this is actually the 'real' answer to my question. There are alternatives being researched. The issue is with the media. Even in non-science media 'dark energy' and 'dark matter' are plastered everywhere. But the alternatives are never discussed, even in popular science media (that I'm aware of; and I'm quite keen on watching anything in this kind of area).

As someone not even close to the cutting edge of science to know otherwise, this false impression is very strong. To the extent that I didn't even consider looking for an alternative (my bad). Apologies for not exploring this more closely before asking here. It's now readily apparent even with a simple Google search that there are alternatives being researched.

I guess the question I should have asked is 'why are these alternatives never given as much air-time as the dark-{x} theories.' And that's a media question; not a scientific one. I guess 'dark energy' is just way more sexy than 'scalar-tensor-vector gravity' or similar.

On the other hand, it did raise an interesting discussion and raise my self-awareness a little bit, which simply reading a completely incomprehensible Wikipedia article wouldn't have done. So thanks for that.

P

But this is the problem with the MEDIA, not the problem with physics and physicists. You are complaining to the wrong people.

And let's be real here. This occurs in many aspect of science as well, not just in this particular case.

But besides that, you also need to be aware that the research surrounding Dark Energy is still in its infancy! Think of the rationality of throwing out General Relativity, a theory that has had so many amazing verification (look at your GPS), for something that we still don't know much of and it is till in its early stages. Would such an action make much rational sense to you?

Zz.

 

[phinds]

mainstream media isn't really helpful in getting an answer

But the alternatives are never discussed, even in popular science media

Well, see, there's your problem. You have the utterly mistaken impression that mainstream media / popular science media actually get things right. They do on rare occasions, but to count on it is foolish, as you are beginning to learn. Pop-sci media in particular is ENTERTAINMENT, absolutely not education.

 

[pervect]

My $0.02 is that there is indeed a lot of theoretical work on other possible theories of gravity, however General Relativity, henceforth GR remains one of the simplest theories of gravity that is compatible with special relativity, henceforth SR,

It's popular in the popular press - and with scientists - partly because of it's relative simplicity. The popular press struggles (and doesn't really succeed very well) in explaining GR to the public - it would struggle even harder (and fail even worse) in trying to describe some of the other more complex theories that are out there.

If you want to go a bit beyond the popular press, and get into the test theories that are more general than GR that have served as a framework for much of our experimental testing, you might look at the PPN formalism. The PPN formalism still makes some theoretical assumptions, it basically considers metric theories of gravity. SR has a metric, if we assume that GR should also, this leads to metric theories. Wiki has a short explanation of PPN at https://en.wikipedia.org/wiki/Parameterized_post-Newtonian_formalism, though it may not emphasize the fact that it's more general than GR and handles any metric theory of gravity.

I'm afraid I can't think of any non-metric theories of gravity offhand, except for Newton-Cartan theory, which isn't a good example because the intent is to go backwards and explain Newton's gravity geometrically, rather than to go forwards beyond GR. Wiki does list a few bimetric theories that I'm not familiar with in it's article on "Alternatives to General Relativity", the current version as of the time of this positing which is at https://en.wikipedia.org/w/index.php?title=Alternatives_to_general_relativity&oldid=960389886.

These theories appear to be examples of Lagrangian field theories, which is another wildly successful framework. From the point of view of Lagnragian field theories, GR is also one of the simplest in the category. This would be another good thing to explore if you want to go beyond the popular press, namely "what is a Lagrangian field theory" and "why are they so popular and powerful".

GR also made a large number of experimental predictions that were unexpected and have been predicted and confirmed later. This is a strong achievement for any theory.

So, it's not the case that nobody is looking for alternative theories. It's more the case that nobody has found one that's really appealing. It's one thing to sit back in one's armchair and to say "this dark matter and dark energy thing is unsatisfying". It's another thing to actually develop a theory that is compatible with all the other experimental results we have to date. People have done so anyway - the usual result though is something that is basically like GR, with a few extra bells and whistles, rather than something fundamentally new.

 

[Dale]

People have done so anyway - the usual result though is something that is basically like GR, with a few extra bells and whistles, rather than something fundamentally new.

And generally the extra bells and whistles actually reduce the predictive power of the theory because they introduce extra variability without fitting the data any better.

 

[romsofia]

Here is a nice paper about a different (toy) theory of gravity called f(R): https://arxiv.org/pdf/0805.1726.pdf

Even if you don't know much math, the introduction section (the 1st 4 pages) gives a nice description and addresses your concerns.

 

[vanhees71]

For me the best semi-popular account of Einstein's theories (special as well as general relativity) is Born's book from the 1920ies (which saved the physics department of my university from bancrupsy and enabled exciting fundamental research like the Stern-Gerlach experiment, which was also substantially financed from funds thanks to Einstein, but that's another story):

https://www.amazon.com/dp/1376156113/?tag=pfamazon01-20

 

[romsofia]

For me the best semi-popular account of Einstein's theories (special as well as general relativity) is Born's book from the 1920ies (which saved the physics department of my university from bancrupsy and enabled exciting fundamental research like the Stern-Gerlach experiment, which was also substantially financed from funds thanks to Einstein, but that's another story):

https://www.amazon.com/dp/1376156113/?tag=pfamazon01-20

Oh wow, that looks like a fun book to read. Never heard of it! (here is a different link for those who want a more readable "preview" for those in the US: https://www.amazon.com/dp/133426127X/?tag=pfamazon01-20)