How do you evaluate product claims?

[Theseus]

OK, I'm not going to tout the wonders of a magnetic bracelet. Rather, this is a survey on scientific evaluation and what you consider is valid evidence. It could apply to lots of things. I just happened to see one of those commercials where you can get your joint pains to go away for $19.95 and it got me thinking (I guess it was a slow night).

I am interested in how you would approach this, that is, if you had any interest at all.
Do you:
1. Immediately dismiss it as pseudoscience and a scam.
2. Find studies that confirm your suspicion that is a pseudoscience and a scam.
3. Objectively look at the studies to see what was discovered.
4. Question people who have used it or try it yourself.

What do you think of people (Arnold Palmer is evidently one of them) who use it and claim that it actually works?

 

[Ivan Seeking]

If it was something that I might really benefit from, I would see if there are any published papers supporting the claims.

Arnold Palmer could just be lying, or it could be the placebo effect. In any case, there are NO qualified studies suggesting that magnetic bracelets work; at least, not that anyone has been able to produce so far. The bracelet people do often take excerpts from completely different applications - very powerful magnets and high frequencies, used under clinical conditions, which can offer some benefits - that have nothing to do with refrigerator magnets worn on the wrist.

 

[Theseus]

I think I would take more or less the same approach though with a product as simple and inexpensive as this I might just skip the studies and try it. Whether it worked or not would bear far more weight than the results of any studies. Of course this applies strictly to being a consumer, producing or marketing the product would be a far more complex issue.

By the way, if I am in pain and the magnet "fixes" the pain, should I care whether it was the placebo effect or something else?

 

[Ivan Seeking]

I think I would take more or less the same approach though with a product as simple and inexpensive as this I might just skip the studies and try it. Whether it worked or not would bear far more weight than the results of any studies. Of course this applies strictly to being a consumer, producing or marketing the product would be a far more complex issue.

By the way, if I am in pain and the magnet "fixes" the pain, should I care whether it was the placebo effect or something else?

True, but then you are subject to wasting money on every scam that comes along. You are also helping to keep the crooks in business. Why would you want to help perpetuate a scam?

It is a near certainty that the magnet claims are bogus, so there is certainly no sense in wasting time on that one.

 

[Theseus]

Yes, I wouldn't want to perpetuate a scam. But if Aunt Sally used the bracelet and it eased her joint pain I probably wouldn't consider it a scam.

A few more points, and I'm not arguing with you here, just trying to get more information:
Should we consider all things a scam until proven otherwise?
While studies are in progress, do we ignore any benefits claimed? Of course claims can be falsified but to consider all claims to be false would require a stretch of the imagination beyond the breaking point.

Remember I am talking about product claims in general, the magnets are just a concrete example. I don't know whether they work or not and don't really care at this point. I am interested in the controversies that seem to often arise between supporting consumers and scientific evaluations.

It is a near certainty that the magnet claims are bogus, so there is certainly no sense in wasting time on that one.

How did you arrive at your near certainty? Was there a study that showed it couldn't work, scientific principles that it violates, etc.?

 

[Ivan Seeking]

You can find studies relating to magnetic field therapy [high power, high frequencies], but I haven't seen any published in a mainstream journal showing that fixed magnets have any beneficial effect beyond the placebo effect. Given that magnets have been around for some time, if they worked, one would expect there to be published papers indicating as much.

Should we consider all things a scam until proven otherwise?
While studies are in progress, do we ignore any benefits claimed? Of course claims can be falsified but to consider all claims to be false would require a stretch of the imagination beyond the breaking point.

The placebo effect is well known, so all untested claims are suspect. Personally, in almost all cases of "breakthroughs", I would want evidence that something works and is safe, before buying it.

Consider some of the herbal stuff that is so popular with health nuts. Some of these have been shown to be dangerous. For example:

Dietary supplements, which are also referred to as health supplements, include herbal products, vitamins, minerals, and any product that is not being marketed as a food or drug (medication). People take dietary supplements with the idea of maintaining or improving their health. Recently, however, my colleagues and I reported in a medical journal a series of seven patients who were referred to us with severe liver injury. They developed the liver damage after taking a health supplement called Lipokinetix® to lose weight. This supplement is a mixture of chemicals that includes an extract from a particular tea. It is said to reduce weight by working to mimic exercise and increase metabolism...

http://www.medicinenet.com/script/main/art.asp?articlekey=20320

I prefer to avoid serving as a guinea pig for crooked marketers and untested claims.

 

[alxm]

First, the same question anyone asks: Is the claim reasonable? If someone claims that rubbing a cat against their head could cure a toothache (nod to Dara O'Briain), then most people would rightly dismiss that as unreasonable. If someone makes a claim with magnets, the situation is different. Most people do not know how magnets work, much less how magnetic fields affect matter/chemistry. This is worth noting.

I happen to be one of the people who does know quite a lot about that stuff, since I'm a chemical physicist. I've spent years studying the subject, and can tell you quite a bit about the Zeeman effect, and other magnetic-atomic/molecular interactions. In that perspective, the claim does not appear particularly reasonable. Because static magnetic fields have almost zero effect on chemistry in general. Even huge magnetic fields (~10 T) have no effect, which is something which is verified millions of times every day in thousands of NMR machines used for chemical analysis every day, for decades, with no weird anomalies discovered so far. Its non-intrusive nature is considered the best feature of NMR. Similarily, no physiological effects of static fields are observed in the thousands of patients being put into the gigantic fields of MRI machines every day (which work on the same principles as NMR). Nor would you expect this, since physiology is ultimately chemistry.

So established scientific theory doesn't provide you with any explanation for why this would work. Do the people selling it have a scientific theory of their own? No, they don't.

Then you look at what evidence there is. Anecdotes like "Arnold Palmer swears by it!" are not scientific evidence, and if the party is presenting anecdotes as if they were evidence, then we have even more of a credibility problem. No-one with a scientific mindset would approach the problem using #4 on OP's list.

Third, you can look at studies. In this case, there are no real studies showing the thing has an effect.

So in summary:
* The sellers have not produced any real evidence this works
* The sellers have no credible theory on why you would believe it works.
* Existing science gives no theory or reason to believe this would work.
* There is no external evidence that it works.

Now let's take into account some circumstances of note:
* To many people, magnets are a mystery with a somewhat 'magic' air about them. They are not capable of evaluating the merit of the claims.
* The people selling the product use testimony as evidence, despite the fact that testimony is not accepted as any kind of scientific evidence in any circumstance.
* They are claiming it helps against joint pain, which is known to be something relatively susceptible to placebo effects.
* The ailment (join pain) is such it's very difficult to objectively quantify. If it was claiming to remove warts from your face, that's easy to test. The question "Do you feel better" is very subjective, and therefore very susceptible to psychological suggestion and cognitive biases.

This all follows the classical pattern of quackery that's been going on for centuries.

Now, speaking personally I would first reach the conclusion that it's not at all a likely claim (and in fact at odds with existing evidence and theory). Then I would take into account the fact that they present no credible theory and no credible evidence themselves, and also the circumstances, and I'd conclude there's no reason at all to look for studies, or even do a study.

For the sake of comparison, let's look at an example of a real scientific claim. Say "Suntanning causes skin cancer, use sunblock". Assuming I don't know this already. I would first look at the fundamental science here. What causes cancer? Fundamentally, it's DNA damage. Can sunlight/suntanning cause DNA damage? I wouldn't know. But sunlight and tanning beds have UV radiation in it. UV radiation is well-known to break chemical bonds, form free radicals and generally wreak havoc on chemistry. So it's at the outset plausible that UV could cause DNA damage and thus cancer. Now, if I go look at the studies, there is in fact a strong correlation (not just some near-margin-of-error thing, but something like a doubling of the cancer incidence). At that point I'd definitely believe it, even if I didn't know the mechanism. As it happens we do now know the mechanism at the molecular level, namely http://en.wikipedia.org/wiki/Pyrimidine_dimers" formation. Finally, if I look at the product claim: Is it plausible that sunblock would alleviate this? This is obviously plausible. All you need is a substance which absorbs and reflects UV light, and there are many of them.

 

[alxm]

Yes, I wouldn't want to perpetuate a scam. But if Aunt Sally used the bracelet and it eased her joint pain I probably wouldn't consider it a scam.

Well then you don't know the scientific method. If Aunt Sally used the bracelet and claimed it eased her joint pain, that would tell you nothing. Because you do not know whether it was the bracelet itself, or whether it was the placebo effect, or whether it was some other factor, or whether she simply started feeling better by herself. For centuries we fed people poisonous mercury compounds as 'medicine'. People swore by it. And they weren't wrong - by your criteria. They did get better. The reason they got better was because they were getting medical attention. But that wasn't known until they started doing proper studies using a control group. Which was the birth of modern medicine.

This is a common fallacy and cognitive bias. Just look at cult members, who invariably claim "I was wasting my life, drinking and partying, but then I found Spaceology and now I'm working a job and happier than ever!". And it's true in the sense that they're better off than they were before. But it's the wrong basis of comparison. They could have found some other way of turning their life around, which quite likely would've been even better for them. Would you advocate joining a cult if it made someone 'happy'?

 

[Theseus]

Thanks, there is a lot of good information here.

If Aunt Sally used the bracelet and claimed it eased her joint pain, that would tell you nothing.

It may not tell you that the thing works but it does tell you something. As you say, you do not know whether it was the bracelet, the placebo effect, etc.

Someone weighing the purchase of a bracelet is most likely experiencing pain for which they seek relief. Currently, we have no way of objectively measuring pain. The best yardstick available AFAIK is the horse's mouth. There are, apparently, questionnaires and profiles available to assist in assessing pain. Now this may not be very "scientific" but on the other hand, a study purporting to evaluate a product for pain relief that discounts any and all "testimony" is skirting the big issue. Unless the study was correlated to actual results I would consider it pretty worthless.

A scientific study that ignores results is like a court system that ignores justice. Assign an advocate to each side and he who plays the best hand wins, truth or justice be damned.

 

[alxm]

It may not tell you that the thing works but it does tell you something. As you say, you do not know whether it was the bracelet, the placebo effect, etc.

No, it tells you nothing. You left out the most important alternative when quoting the text (now there's some confirmation bias right there): That it was nothing at all, that she may have just started feeling better randomly. This happens all the time. You can't say what happened in a single event, because our knowledge of the world is flawed and imperfect. You don't know, and never will know, all the relevant circumstances there. Or to put it another way: You have 'signal' (causal relationships to things we deem significant) and 'noise' ('random' events). You simply can't tell if a single data point is 'signal' or 'noise' if that's all you've got. A single event does not establish a pattern. This is statistics; hard, mathematical fact.

On the other hand we have well-known psychological fact: Our brains are good pattern-finders. They're hard-wired to do it. But this mechanism often fails and leads us to find patterns where there aren't any. We see 'faces' in rocks. We very consistently fail to recognize random behavior. (and conversely, don't succeed very well at producing truly 'random' strings of digits, for instance). And this leads to mis-attributing the causes of events. Add to that our propensity for confirmation bias and you have billions of people walking around with thousands of weird superstitions that simply never had any basis in reality to begin with.

There are, apparently, questionnaires and profiles available to assist in assessing pain. Now this may not be very "scientific" but on the other hand, a study purporting to evaluate a product for pain relief that discounts any and all "testimony" is skirting the big issue.

That's a straw-man argument. There's a huge difference between an individual testifying his personal experiences of a single event, and a controlled scientific study that's based on a large group of individuals reporting their experiences.

One person giving testimony on behalf of a product says nothing. Neither does 100 individuals doing the same. Because what are they comparing to? (nothing, as I already explained) What are you comparing to? What about the people who didn't experience a positive effect, and thus weren't included in the marketing? You have no basis for comparison either. Not to mention ruling out other factors.

A proper scientific study would, for instance, do something such as take three large groups suffering from the same ailments. Give one group your magnetic bracelets, another group identical bracelets that weren't magnetic (placebo), and a third group no bracelets at all (control), and then compare their reported results after some time. (The ones wearing bracelets would naturally all be told that they were wearing the magnetic ones.)

A scientific study that ignores results is like a court system that ignores justice. Assign an advocate to each side and he who plays the best hand wins, truth or justice be damned.

You should really learn more about the scientific method and how science works, and why it works before you go around making statements about how you think it should work. Science does not ignore evidence. Especially not evidence of things that go against our existing theories. In fact, evidence that goes against our existing theories are especially interesting. How else would we improve our knowledge?

Science ignores 'evidence' when it's not actually evidence of anything. Nor does science go out of its way to study possible relationships where we don't have any particular reason to believe there is one. There's no conceivable mechanism for how rubbing a cat against my head would cure toothache, so nobody's going to bother to do a study to find out if such a phenomenon exists. That's not being 'closed-minded'. It's being economical with your time and efforts.

 

[Ivan Seeking]

A proper scientific study would, for instance, do something such as take three large groups suffering from the same ailments. Give one group your magnetic bracelets, another group identical bracelets that weren't magnetic (placebo), and a third group no bracelets at all (control), and then compare their reported results after some time. (The ones wearing bracelets would naturally all be told that they were wearing the magnetic ones.)

One problem with magnetic bracelets: There is no easy way [that I know of] to do a double-blind study. One would have to prevent access to any ferrous materials for the entire duration of the study, for both the researchers [when active in the study] and the test subjects.

 

[Theseus]

One person giving testimony on behalf of a product says nothing.

I have a confession to make ... I don't actually have an Aunt Sally.
I do, however, have an aunt although to my knowledge, she has never tried the Magic Bracelet.
Just to be clear, this was never a point I tried to make.

 

[mugaliens]

I think the best way to evaluate product claims is by first keeping abreast of advances in science across many fronts from a variety of reputable, peer-reviewed sources.

Having a firm grounding in the basic sciences as well as engineering helps, as does practical experience with a wide variety of products.

I am interested in how you would approach this, that is, if you had any interest at all.
Do you:
1. Immediately dismiss it as pseudoscience and a scam.

Nope. I actually do have an open mind.

2. Find studies that confirm your suspicion that is a pseudoscience and a scam.

Nope. Because I have an open mind, I'll look for studies which have something to say about both sides of the issue.

3. Objectively look at the studies to see what was discovered.

Yep, as well as weighing what I read in those studies against what I know of the sciences to see if they hold any water.

4. Question people who have used it or try it yourself.

Nope, as that's anecdotal evidence. The exception is if I've not merely tried something myself, but if I've objectively measured the results. For example, laying down aluminum foil in my attic worked, but I borrowed an electronic radiant thermometer and took two weeks of measurements at various points on my ceiling before and after, as well as external system measurements into interior temperature of the house, external temperature, solar influx, relative humidity...

I know, I get a bit carried away. It's just that if I really want to determine if something is true, I do not rely on personal observation, as that's often misleading, even when we think we're being perfectly objective.

What do you think of people (Arnold Palmer is evidently one of them) who use it and claim that it actually works?[/QUOTE]