Discussion of PF Policy on Valid References

[PeterDonis]

Is it time to put this thread out of its misery?

I don't know. Do you think there is no value in this kind of discussion?

 

[PeterDonis]

PF has an entire section on physically meaningless quantum interpretations populated by very smart professional scientists.

Even in that forum, we still apply the rules about valid sources. There is an extensive literature on quantum interpretations, including both textbooks and peer-reviewed papers, so in that respect the QM interpretations subforum is no different from any other of our science forums. (Similar remarks would apply to the Beyond the Standard Model forum, which is the other subforum where the subject matter goes beyond "mainstream" science to areas that many view as more speculative.)

 

[PeterDonis]

Honestly, @PeterDonis, in this thread it just feels like you're trying to win an argument rather than consider other viewpoints.

What I'm trying to do is explain the rationale for PF's policy about valid references. I did not make up that policy; it's right there in the rules:

• Acceptable Sources:
  Generally, discussion topics should be traceable to standard textbooks or to peer-reviewed scientific literature. Usually, we accept references from journals that are listed in the Thomson/Reuters list (now Clarivate):
  
  https://mjl.clarivate.com/home
  
  Use the search feature to search for journals by words in their titles.
  
  In recent years, there has been an increasing number of "fringe" and Internet-only journals that appear to have lax reviewing standards. We do not generally accept references from such journals. Note that some of these fringe journals are listed in Thomson Reuters. Just because a journal is listed in Thomson Reuters does not mean it is acceptable.
  
  References that appear only on http://www.arxiv.org/ (which is not peer-reviewed) are subject to review by the Mentors. We recognize that in some fields this is the accepted means of professional communication, but in other fields, we prefer to wait until formal publication elsewhere. References that appear only on viXra (http://www.vixra.org) are never allowed.

My question to everyone in this thread who has argued anything along the lines of "we should be able to ask questions based on pop science sources" is: apparently you think the above rules should be changed, since those rules do not include pop science sources as valid references. How and why do you think the rules should be changed?

 

[TeethWhitener]

My question to everyone in this thread who has argued anything along the lines of "we should be able to ask questions based on pop science sources" is: apparently you think the above rules should be changed, since those rules do not include pop science sources as valid references. How and why do you think the rules should be changed?

“Traceable” in the policy seems to be open to at least two different interpretations. Plenty of claims in pop sci are traceable to peer reviewed literature. I therefore see no reason for the rules to change. However, you seem to be interpreting “traceable” in some stricter sense, perhaps with an eye to how faithfully represented that literature is by a particular pop sci work. So maybe you want the rules to change to reflect that.

 

[PeterDonis]

Plenty of claims in pop sci are traceable to peer reviewed literature.

Yes, and in such cases, our response is generally to try to redirect the discussion to be based on the peer reviewed literature that the claim is traceable to, at least if the discussion goes beyond a simple question and answer or two. (This assumes that the pop science source reasonably represents the literature; I'll address the case where it doesn't below.)

However, many pop science sources cite no literature whatever. This is particularly disappointing with articles that are specifically discussing some new experiment or research--but without any link to the paper that published the research. In such cases, unfortunately, the response is going to be to ask the OP to find a valid reference.

you seem to be interpreting “traceable” in some stricter sense

No. I'm just pointing out that the traceable cases are rare in comparison to the total number of questions we see based on pop science sources. And in my experience edge cases where the boundary of "traceable" is not reasonably clear are extremely rare.

perhaps with an eye to how faithfully represented that literature is by a particular pop sci work

If we can trace the actual literature, cases where the pop science article misrepresents the literature (which are unfortunately common, even when the pop science article is written by a scientist who should know better) are fairly easy to handle, at least as far as pointing out the discrepancy goes. (The OP of such threads doesn't always receive the news very well, but that's a separate issue.)

 

[AndreasC]

Something can be traceable even if the op doesn't or can't trace it though.

 

[Rive]

How and why do you think the rules should be changed?

I think the source of the source problem is, that the acceptable sources list just does not really fit the Curious Average Joe. Who will 'learn' some popsci first, and then can only expect to get a wall in the face here for the first question.

Not really a good hook for new member retention.

As I see it, while references, sources and starting points should be clearly distinguished by requirements (and regarding references, what we have here is just perfect!), some kind of recognition should be granted for valid questions originating from some (!) popsci sources as starting point (preferably just as a starting point: possibly 'at Mentor discretion' as this approach already exists for other fuzzy cases).

With making it very clear, that popsci questions will result in popsci answers of limited value only, and unless the topic can steer away from the starting point it'll be short lived.
But still, officially recognized, so no one should feel shame for asking.

That's how I think.


So instead of a 'we don't do that' it should be some kind of 'you should move away from that' policy, which - as far as I see it - is actually what's often happening, just unofficially.

 

[vanhees71]

I think it's a kind of a dilemma. On the one hand I like @PeterDonis pretty strict policy on admitting only "papers in refereed journals" and textbooks as valid references in PF. Experience shows that the former are indeed the most reliable sources of valid results of research. Of course, even for them there's no guarantee that there are mistakes in them too since we all are more or less carefull reviewers ;-)). Textbooks are a little less reliable (at least newer ones since nowadays no scientific publisher can afford are true lectorate, but that's another story). In contrast popular-science books or even some webpage or youtube video are usually very unreliable. The reason is that it is utmost difficult to write a good popular-science text without to some extent being unprecise. That's because the only reliable language to express science accurately is the use of mathematics, which you can't do when writing a popular-science text.

On the other hand, popular-science texts are important to inform the interested public about current research. After all much money paying our experiments and the theorists trying to understand them in fundamental research is tax payers' money, and thus the public has a right to be informed about the results. Outreach, nowadays, is thus more and more important also for the funding agencies, and in any proposal for funding we have to include a section about possible outreach activities.

So it's also not a good idea to ban all popular-science texts from discussion in physicsforums, and one should not be too quick with cutting of a discussion, only because the OP cannot deliver a "valid reference". Of course, one can and should point out possible inaccuracies in popular-science texts and also give references to the scientific literature in the discussion.

Last but not least there are also quite reliable (semi-)popular sources like Wikipedia, which is amazingly good (the English version usually somewhat better than the German; other languages I'm unable to judge).

Finally, I don't think we need a change of the general PF rules on "acceptable sources". I only think one should not completely ban discussion of popular-science sources, even if an interested lay man cannot point to a scientific reference to its subject.

 

[TeethWhitener]

I'm just pointing out that the traceable cases are rare in comparison to the total number of questions we see based on pop science sources.

@vanhees71 , @Rive , and @AndreasC above have given pretty good synopses of how I feel here, but I think this quote above is where you and I likely disagree the most. Maybe it’s because we frequent different sections of PF, but I don’t often come across pop sci discussion that can’t be traced fairly straightforwardly to the literature, even if that means I have to do some googling. (I’m leaving aside cases where OP says some variant of “I read somewhere,” where I think asking for concrete sources is acceptable/necessary.)

And yes, the onus isn’t on the responder to find the literature on which a pop sci article is based, but frankly the onus isn’t on the responder to respond at all. If the OP is going to get hit with “go get a real string theory textbook” every time Michio Kaku opens his mouth, that OP is likely not to come back to PF for answers, which makes sense because they didn’t get an answer from PF. They got told to go away.

 

[pinball1970]

I went back to the beginning of the thread, the comment was,
"This is a pop science book. You would be better served by reading actual textbooks or peer-reviewed papers."

I don't see what is wrong with that. It is up to the poster to think about what that means and what to do next.
Realising that popsci does not teach any physics, even though the book is authored by a well known physicist, is not that obvious to a non physicist. (From personal experience)

I do not see the problem explaining that, then directing them where appropriate?
"A good beginners book is ...(insert beginners book here)."
I have seen that lots of times in different forums EDIT: On PF.

 

[vanhees71]

Per se, there's nothing to say against the content of the 1st paragraph, but don't you think it's repelling interested laymen from posting again and/or following the discussion further? Compare this to your own formulation in the last paragraph, which is the much better way of expression the same thing!

 

[Vanadium 50]

I think "what if" is not a very productive direction - we're not going to come up with a flowchart that addresses every possible case. Further, trying to set policy by looking prinarily at edge cases usually just makes a mess.

If someone says "I heard that", it is entirely appropriate for PF - not just the staff - to ask "where?".

If the answer to that is not a very good source, PF. - not just the staff - can say so. "This is a popularization and cuts a lot of corner"s should be an acceptable answer. And yes, I know not everyone accepts it.

On the general issue of popularrizations, if they were as good as textbooks, there wouldn't be any textbooks, Who would bother?

Finally, if you want to argue that threads should be closed by non-participating mentors, because having participating mentors close it is n not a good look ("Shut up, he explained"), I agree. But that's hardly worth a thread on it.

 

[phinds]

My two cents: changing how we deal with pop-sci and references in general would be a TERRIBLE idea. Why? Very simple. This forum is enormously successful for one simple reason. The moderators.

Force the moderators to deal with a lot more crap than they already have to deal with and you start driving THEM away and thus would begin the downfall of this forum as such an excellent site for science.

Also, just my opinion, I don't find the moderators to be overly harsh on ignorant newbies. To the contrary, I find it astounding just how patient they often are.

 

[vanhees71]

Yes, I'd also not change the general rules, which are successfully applied for years. One should only apply them with sufficient flexibility not to deterr new posters by a very harsh reaction to any pop-sci reference (or even on "I've read somewhere, that...").

 

[AndreasC]

I think "what if" is not a very productive direction - we're not going to come up with a flowchart that addresses every possible case. Further, trying to set policy nu looking prinarily at edge cases usually just makes a mess.

If someone says "I heard that", it is entirely appropriate for PF - not just the staff - to ask "where?".

If the answer to that is not a very good source, PF. - not just the staff - can say so. "This is a popularization and cuts a lot of corners
should be an acceptable answer. And yes, I know not everyone accepts it.

On the general issue of popularrizations, if they were as good as textbooks, there wouldn't be any textbooks, Who would bother?

Finally, if you want to argue that threads should be closed by non-participating mentors, because having participating mentors close it is n not a good look ("Shut up, he ecxplained", I agree. But that's hardly worth a thread on it.

I don't see anything wrong with what you are saying here. I don't object with telling people that pop sci is not a good source, etc. But, after that is done, then what I believe is that if the question nevertheless makes sense but op is not advanced enough to understand a highly technical discussion, it is preferable to attempt a level appropriate answer to the extent it is possible, or say that you have no way to answer it in a level appropriate manner and point to the required background, rather than closing the thread or telling them they should read graduate level original papers to have their question answered (which won't happen).

Now, if the question does not make any sense, it's still instructive to tell someone their question doesn't make sense and why.

Or you can not answer at all. Nobody is forced to answer obviously. I just think that telling people to basically come back when they finish their Masters can be kind of frustrating and off putting to some people.

After all, the general public funds research, and some highly theoretical disciplines don't "pay back" very much. They are worthwhile because they broaden human understanding. What good is that however if none of that "trickles down" to anyone but highly specialized experts? And what is a better place for the general public to come in contact with legitimate scientists and ask them questions?

For what it's worth, I don't think changing the general rules is necessary either, I agree with what @vanhees71 said.

 

[russ_watters]

My experience as a moderator of this forum has been that the rare cases where a pop science source actually sparks a good question...

Just note that that's not quite in alignment with what @Tom.G said. What you are saying is clearly true about the signal to noise ratio in threads/on PF, but @Tom.G was talking about engagement. Pop-sci is a common - perhaps the most common - source of engagement between the public and "science". Pop-sci reduces the apparent barrier to entry into science and is what prompts many to come here and ask questions....or even, later, become scientists themselves.

In my opinion our goal should be to bring as many into the fold as possible while maintaining as high of standards as possible. Yes, that's a tough balance/tradeoff. For the record I did think the original response in the thread was pretty benign ("You would be better served...") but I also think that encouraging people to go from zero to a college text instantly has the feel of raising the barrier back up. We may only get five minutes or less of initial attention for such people and offering a fun-sized piece of candy (a factoid) at least to start with is more likely to draw them in (or send them away satisfied with their snack) than suggesting they put on a coat and clear their schedule for a 9 course meal (a college text).

 

[PeterDonis]

I don’t often come across pop sci discussion that can’t be traced fairly straightforwardly to the literature

Bear in mind that you don't see the ones that get deleted by the moderators.

 

[PeterDonis]

I don't think we need a change of the general PF rules on "acceptable sources". I only think one should not completely ban discussion of popular-science sources

These two statements are contradictory. Which do you pick if push comes to shove?

 

[PeterDonis]

In my opinion our goal should be to bring as many into the fold as possible while maintaining as high of standards as possible.

So you are saying PF's goal should change? The "About PF" Insights article currently says:

Our goal is to provide a community for people (whether students, professional scientists, or hobbyists) to learn and discuss science as it is currently generally understood and practiced by the professional scientific community.

That says nothing about bringing as many people into the field as possible.

Plus, what does "bringing as many people into the field as possible" mean? Does it mean we want them to become scientists? Or just PF members?

 

[PeterDonis]

Last but not least there are also quite reliable (semi-)popular sources like Wikipedia

I have already mentioned Wikipedia more than once. A Wikipedia article that footnotes valid sources (textbooks or peer-reviewed papers) for its key claims (which most articles in scientific areas do) meets the "traceable" requirement and is therefore acceptable as a reference.

 

[PeterDonis]

some kind of recognition should be granted for valid questions originating from some (!) popsci sources as starting point

I think the "traceable" language in the rules covers this.

 

[PeterDonis]

if the question nevertheless makes sense but op is not advanced enough to understand a highly technical discussion, it is preferable to attempt a level appropriate answer to the extent it is possible, or say that you have no way to answer it in a level appropriate manner and point to the required background

I think the "traceable" language covers this too, because for a question to "nevertheless make sense", the source it is coming from is going to have a reference to actual literature somewhere. At least that's what I would expect based on my experience.

For cases where the OP clearly doesn't have the background for a higher level answer, we typically lower the thread level if possible, or we explain why that isn't possible. So I think we're handling questions deemed to be valid but with a potential level mismatch the way you describe.

 

[PeterDonis]

if the question does not make any sense, it's still instructive to tell someone their question doesn't make sense and why.

"Does not make any sense" is a very wide category. Yes, some questions will not make sense in a way that can be explained. But most won't. The distinction here is the one Pauli was getting at with his phrase "not even wrong". A question that doesn't make any sense in a way that is wrong might be repairable. A question that is not even wrong, not so much.

 

[russ_watters]

So you are saying PF's goal should change? The "About PF" Insights article currently says:
"Our goal is to provide a community for people (whether students, professional scientists, or hobbyists) to learn and discuss science as it is currently generally understood and practiced by the professional scientific community."

Yes. But in my opinion that statement is narrower than the reality already, and misleading. Even contradictory. The second line implies this is a forum intended for professionals only, and that isn't the case. How does k-12 homework help fit with that second line?

That says nothing about bringing as many people into the field as possible.

Plus, what does "bringing as many people into the field as possible" mean? Does it mean we want them to become scientists? Or just PF members?

The word was "fold". Not caring about our membership/post rate numbers is a good way to kill the forum. In my opinion, too high of a quality bar is pointless if the forum disappears because of it.

I don't want to go much further along this line of discussion until I have something more concrete to propose, and not here - in the private forum.

 

[PeterDonis]

telling people to basically come back when they finish their Masters

Btw, since this sort of comment has been made by several posters in this thread, please note that in the original thread that spawned this one, this was not done. I responded to the OP by pointing out that the book they referenced was a pop science book, and posted a link to the Wikipedia article on false vacuum decay. I recommended reading the sources linked to in that article as well, but the link I posted was to the article (which, as has already been said, meets the requirements for a valid reference since it footnotes its claims to the relevant literature). (I also gave a basic answer to the general question.)

 

[PeterDonis]

I don't want to go much further along this line of discussion until I have something more concrete to propose, and not here - in the private forum.

Fair enough.

 

[PeterDonis]

Something can be traceable even if the op doesn't or can't trace it though.

Traceable means the source has to have references explicitly stated somewhere. If the only way to "trace" something in a pop science source to the literature is a combination of guessing and Googling, that's not traceable in any useful sense.

 

[StandardsGuy]

What if there is no such explanation? What if every possible "pop-sci-like" attempt at an explanation is wrong, or leads to misunderstanding instead of actual understanding?

How many times in the history of science has the current understanding been wrong, only to be changed by future discoveries? Take the Big Bang theory for instance. The recent JWST discoveries have challenged that.

 

[StandardsGuy]

Bear in mind that you don't see the ones that get deleted by the moderators.

As one who has had posts deleted, this is a strong put off. That's why I seldom get on this site. I think it would be better to reply to the person, "this did not contribute to the thread" rather than delete the post and the thumbs up that came with it.

 

[berkeman]

As one who has had posts deleted, this is a strong put off. That's why I seldom get on this site. I think it would be better to reply to the person, "this did not contribute to the thread" rather than delete the post and the thumbs up that came with it.

I did a quick check of 3 such instances -- your posts that were deleted were reported by other members, not cherry-picked by the Mentors. It's always important to try your best to find good sources for assertions in posts in the technical forums.

 

[PeterDonis]

How many times in the history of science has the current understanding been wrong, only to be changed by future discoveries?

Wrong? Almost never once a field becomes a science (i.e., based on experimental testing).

An approximation that gets refined by further discoveries? Lots of times.

Take the Big Bang theory for instance. The recent JWST discoveries have challenged that.

No, they haven't. They have led to quite a bit of discussion about the precise values of some of the parameters, but they have not challenged the framework of the theory itself.

 

[russ_watters]

How many times in the history of science has the current understanding been wrong, only to be changed by future discoveries? Take the Big Bang theory for instance. The recent JWST discoveries have challenged that.

Wrong? Almost never once a field becomes a science (i.e., based on experimental testing).

An approximation that gets refined by further discoveries? Lots of times.

No, they haven't. They have led to quite a bit of discussion about the precise values of some of the parameters, but they have not challenged the framework of the theory itself.

This is, in fact, a built-in flaw in pop-sci; it's over-hyping of new discoveries or even past ones to generate interest. Clickbait.

 

[vanhees71]

"Does not make any sense" is a very wide category. Yes, some questions will not make sense in a way that can be explained. But most won't. The distinction here is the one Pauli was getting at with his phrase "not even wrong". A question that doesn't make any sense in a way that is wrong might be repairable. A question that is not even wrong, not so much.

Questions can neither make sense nor making no sense. They are questions! If a question indicates a misconception of the questioner, one can point that out as well.

 

[PeterDonis]

If a question indicates a misconception of the questioner, one can point that out as well.

My point is that for some questions, "misconception" is too generous a term: they are so far off base that it is not even possible to point out a misconception or reframe the question. That's what the phrase "not even wrong" is intended to convey. There has to be a point at which the only reasonable moderation decision is to delete the thread to keep the signal to noise ratio of the forums acceptable.

 

[vanhees71]

Of course. As I said, I'd keep the rules and the moderation practice as they are.