Can temperature data be averaged to a greater resolution than the raw data?

[Jobrag]

I've just been looking at a climate change thread (comments on George Monbiot's Guardian column), some one has posted some average temperatures giving figures to two decimal places. Now I doubt that the met office measure temperature to two decimal places so this is probably the result of averaging. To the meat of it, I always understood that you can not average to a greater resolution then the raw data, i.e. if you measure temperature to one decimal place you cannot then quote averaged temperatures to more then one decimal place. Have I misunderstood this for years or is there some dodgy data floating around?

 

[Mark44]

Yes, it's meaningless to end up with computed values that have more precision than the values used in the computation.

 

[Gerenuk]

Actually it is meaningful and it is the point of doing multiple measurements. Obviously if you do 100 measurements of the same quantity and average, you get a much more precise estimate than if you had done just one measurement.

 

[Mark44]

I disagree. If you measure the temperature as 17.6 deg. C, what you are effectively saying is that the actual temp could have been anything between 17.55 and 17.65 degrees. If all your measurements are to one decimal place, a value computed from them cannot be more precise than one decimal place.

There's a difference between accuracy and precision. To use an analogy, if a rifle marksman puts all his shots within a circle of radius 20 cm from the bullseye, his shooting is accurate, but not precise. If a different marksman puts all his shots within a circle of radius 5 cm whose center is 20 cm from the bullseye, his shooting is precise but not accurate. A third marksman who puts all his shots in a 5 cm circle around the bullseye is accurate and precise.

 

[Gerenuk]

I disagree. If you measure the temperature as 17.6 deg. C

That is the point. The average value is not a measured value. It is an estimate for a real value which is obscured by imprecise measurements.

To use an analogy, if a rifle marksman puts all his shots within a circle of radius 20 cm from the bullseye

I'd like to use your analogy of the rifle marksman to explain this. The marksman shoots and someelse sees the hits only - without knowing where the marksman aimed at. If he sees only one shot then he will be very uncertain about the aiming position. If he see many hits he can make a very confident estimate about the position the marksman was aiming for. This estimate will be much more precise than any single of the hits could have told him.

 

[Mark44]

That is the point. The average value is not a measured value. It is an estimate for a real value which is obscured by imprecise measurements.

Clearly the computed average value is not a measured value, nor did I say it was.

I'd like to use your analogy of the rifle marksman to explain this. The marksman shoots and someelse sees the hits only - without knowing where the marksman aimed at. If he sees only one shot then he will be very uncertain about the aiming position. If he see many hits he can make a very confident estimate about the position the marksman was aiming for. This estimate will be much more precise than any single of the hits could have told him.

Suppose I measure the length of something over the course of several days using a stick with no markings on it, and get these measurements: 2, 4, 5. If I take the average of these numbers, I get 11/3 = 3.7 rounding to 1 decimal place. How confident can I be in that .7 in my average?

 

[Gerenuk]

Clearly the computed average value is not a measured value, nor did I say it was.

That is true. However, we were actually talking about high precision numbers that came from a calculation which averaged measurements.

Suppose I measure the length of something over the course of several days using a stick with no markings on it, and get these measurements: 2, 4, 5. If I take the average of these numbers, I get 11/3 = 3.7 rounding to 1 decimal place. How confident can I be in that .7 in my average?

One can actually give an answer here.
I'm not an expert in statistics but here we go:
http://en.wikipedia.org/wiki/Standard_error_(statistics)
So the standard deviation of the mean is 0.7
Which tells me that with a chance of 95% the real mean is between 2.3 and 5.1
Which isn't great, but still better than one measurement alone. Here the last digit is indeed useless.

However if you do many more measurements you might find than the estimated mean is between 3.6849 and 3.6836. In that case you can write that the mean is 3.68 (skipping uncertain digits). That's what the scientists did with many measurements (I hope!).

 

[Mark44]

And these estimated means and other measures are given with a certain probability.

I learned in physics and chemistry a long time ago that if you measure the dimensions of some object and then compute the area, volume, whatever, your computed value should not have more precision than your measured values.

 

[Integral]

And these estimated means and other measures are given with a certain probability.

I learned in physics and chemistry a long time ago that if you measure the dimensions of some object and then compute the area, volume, whatever, your computed value should not have more precision than your measured values.

There is a big difference between errors propagating through a computation (computing the area of a rectangular volume) and taking the average of multiple measurements. I would assume that you can recall that from those same chem and physics classes.

 

[Gerenuk]

I assume that most people take the standard deviation as their "error" which means their result have a 60% probabilities.

If from one value you computed another value then - you are right - you won't be able to improve on accuracy. However if you repeat the same measurement many times, then the accuracy of your estimation does improve with every new measurement.

 

[Integral]

I assume that most people take the standard deviation as their "error" which means their result have a 60% probabilities.

If from one value you computed another value then - you are right - you won't be able to improve on accuracy. However if you repeat the same measurement many times, then the accuracy of your estimation does improve with every new measurement.

Of course this assumes that you have consistent enough measurements to keep the standard deviation small. It is possible to get data that is unusable due to a large st. dev.

 

[zgozvrm]

Yes, it's meaningless to end up with computed values that have more precision than the values used in the computation.

Not true. Take sports, for example. If a basketball player scores 20 points in the 1st game and 19 points in the 2nd, his average after 2 games would be 19.5 points per game. If we didn't allow for greater accuracy than we started with, we would have to say that his average is 20 points per game which, of course, is not accurate.

Suppose, over the next 4 games, he scores 27, 17, 24, and 26 points.
His overall average after 6 games would then be (20+19+27+17+24+26)/6 = 133/6 = 22.16666666, so we would say 22.17 or 22.167 is his average.

 

[Mark44]

Not true. Take sports, for example. If a basketball player scores 20 points in the 1st game and 19 points in the 2nd, his average after 2 games would be 19.5 points per game. If we didn't allow for greater accuracy than we started with, we would have to say that his average is 20 points per game which, of course, is not accurate.

Suppose, over the next 4 games, he scores 27, 17, 24, and 26 points.
His overall average after 6 games would then be (20+19+27+17+24+26)/6 = 133/6 = 22.16666666, so we would say 22.17 or 22.167 is his average.

I'm willing to buy the arguments about repeated measurements, but your example is not germane in this discussion. We can assume that the points made by the BB player are exact and not subject to measurement error, so the computed average can have as much precision as you want.

 

[zgozvrm]

I'm willing to buy the arguments about repeated measurements, but your example is not germane in this discussion. We can assume that the points made by the BB player are exact and not subject to measurement error, so the computed average can have as much precision as you want.

Again, I disagree...

Temperature is an exact measurement as well. Just because the person taking the temperature reading may misread the value (or the equipment is out of calibration), doesn't mean that you aren't working with exact values.

Here's a "more precise" example:

Suppose you take 6 temperature readings, each to the nearest 1/100th of a degree, and add up to 237.39 degrees. Divide that value by 6 and you get 41.23166666...
Therefore the average should be given as 41.2317 or 41.23167 degrees.

In this case, only showing 1/100ths of a degree (41.23 degrees) is not enough precision, since 237.38/6 = 41.23 (exactly) and 237.37/6 is approximately equal to 41.23, as is 237.40/6. We need the extra precision to be accurate about our averages.

As for a basketball player's score being "exact and not subject to measurement error," that is not entirely true: the scorekeeper could miss or forget to count a basket, he could write the score down incorrectly, or he could misinterpret the value of the shot (was it a 3-point basket or a 2-point basket?)

 

[Jobrag]

Good debate so far but reading through this lot has raised another twist. Could it be that if you perform repeated readings with the same instrument then it is reasonable to take the average to a higher order but if you are using a series of different instruments then it is not?

 

[Mark44]

Again, I disagree...

Temperature is an exact measurement as well.

You can't mean this. There is some number that exactly represents the temperature, but there is NO way that we can measure it with infinite precision.

Just because the person taking the temperature reading may misread the value (or the equipment is out of calibration), doesn't mean that you aren't working with exact values.

I'm not talking about operator error here - I'm saying that no such instrument exists that can measure the temperature (or length, or weight, or any other continously distributed quantity) exactly.

Here's a "more precise" example:

Suppose you take 6 temperature readings, each to the nearest 1/100th of a degree, and add up to 237.39 degrees. Divide that value by 6 and you get 41.23166666...
Therefore the average should be given as 41.2317 or 41.23167 degrees.

Each of your temperature readings, being accurate to the nearest .01 degree, is really some number in a band .005 on either side of the nominal measurement. This means that if you measure the temperature as 42.64 degrees, you're really saying that it is somewhere between 42.635 and 42.645 degrees, so there is some uncertainty in the second decimal place. To say that your computed average now has two or three more decimal places of precision -- you're going to have to justify that to make be believe it.

In this case, only showing 1/100ths of a degree (41.23 degrees) is not enough precision, since 237.38/6 = 41.23 (exactly) and 237.37/6 is approximately equal to 41.23, as is 237.40/6. We need the extra precision to be accurate about our averages.

As for a basketball player's score being "exact and not subject to measurement error," that is not entirely true: the scorekeeper could miss or forget to count a basket, he could write the score down incorrectly, or he could misinterpret the value of the shot (was it a 3-point basket or a 2-point basket?)

Sure, that could happen, but it seldom does, because there are other people there who can see the incorrect score on the board, and who are invested enough in the game to let that slide. I think you're missing my point, which is basically the difference between integers (which are discrete) and the real numbers. It's a lot harder to screw up the integer count of something than it is to measure something whose values are distributed along a continuum.

 

[Jobrag]

I can see that if you are taking multiple measurements of the same thing with the same instrument that it might be OK to take the average out to a higher order then the original measurement. But if you go back to the original question, if your taking mutiple measurements using different devices and getting a large spread of results, i.e temperature recordings from around the country, at different times of the year can you then take results to an order better then the original data?

 

[zgozvrm]

Either way, my examples have shown that you need to show MORE degrees of accuracy when dealing with averages, no matter how accurate your original data is. After all, if your not assuming that your original data is as accurate as it can be, then what good is it anyway?

Sure, if I take a temperature measurement to the nearest 1/xth of a degree, one could always make the argument that it could have been taken more accurately to the nearest 1/(x+1)th degree, if we had better, more accurate equipment. But reading only to the 1/xth degree, doesn't take away from the fact that it is as accurate as you can be with what you've got. All it means is that your averages will be taken to less precision than if the original readings were more accurate.

 

[zgozvrm]

Even if you're using different instruments, aren't you assuming that each measurement is as accurate as it can be?

Suppose you took 999 measurements to the nearest 0.01 degree and one measurement to the nearest 0.00001 degree using a different measuring device. This last measurement, being more accurate than the previous ones, increases the accuracy of your average. If you were to round that last measurement to the nearest 0.01 degree, your accuracy of THAT measurement decreases, as does your average.

 

[hamster143]

Any physical variable will have two error bars, statistical and systematic.

The statistical error bar depends on how many measurements you make and on the distribution of results. If you make 5 random measurements of a variable that ranges from -20 C to 50 C, your estimate of its mean on the basis of these 5 measurements will have an extremely high value of statistical error.

The systematic error bar has to do with how you measure the thing. If all your measurements were done with thermometers that are only accurate up to 0.5 C, that means systematic error bar +/-0.5 C, no matter how many measurements there were.

 

[zgozvrm]

The systematic error bar has to do with how you measure the thing. If all your measurements were done with thermometers that are only accurate up to 0.5 C, that means systematic error bar +/-0.5 C, no matter how many measurements there were.

Using this logic, measurements of 42.5 C and 43.0 C (to the nearest 0.5 C) would average to 43.0 C. We can clearly see that the average would actually be closer to 42.75 C in this case. At the very least, you should take the average to the nearest 0.1 C, giving 42.8 C as the average.

 

[Mark44]

Either way, my examples have shown that you need to show MORE degrees of accuracy when dealing with averages, no matter how accurate your original data is. After all, if your not assuming that your original data is as accurate as it can be, then what good is it anyway?

Did you read my previous reply in post #16? You didn't respond to what I said in it.

Sure, if I take a temperature measurement to the nearest 1/xth of a degree, one could always make the argument that it could have been taken more accurately to the nearest 1/(x+1)th degree, if we had better, more accurate equipment. But reading only to the 1/xth degree, doesn't take away from the fact that it is as accurate as you can be with what you've got. All it means is that your averages will be taken to less precision than if the original readings were more accurate.

 

[zgozvrm]

Any measurement from 42.25000000 to 42.74999999 rounded to the nearest 0.5 would result in 42.5

Any measurement from 42.75000000 to 43.24999999 rounded to the nearest 0.5 would result in 43.0

These values would yield sums ranging from 85.00000000 to 85.99999999 and averages ranging from 42.50000000 to 42.99999999

To take the average to the nearest 0.5 would mean that the average is either 42.5 or 43.0 both of which lie on the outskirts of the possible value of the average. 42.75 lies in the middle of that range and is, therefore a better representation of the average.

 

[Mark44]

Instead of measuring things to the nearest .5, let's agree to do things as they usually are done with tolerances. We use decimal fractions, and measurements are made to the nearest 1/10, 1/100, 1/1000, and so on. If we were working with binary (base-2) fractions, then it would make sense to talk about the nearest 1/2, 1/4, 1/8, and so on.

If we measure something to the nearest tenth and get, say, 12.3, what we are saying is that our measurement is 12.3 +/- 0.05.

Suppose we have a collection of metal rods that are roughly the same length. Suppose also that our measurements are accurate to the nearest tenth (of an inch, cm, whatever). We pick out one rod and measure its length as 42.5. Its actual length could be anything between 42.45 and 42.55. Now we pick another rod and measure its length as 43.0, which means its actual length could be anywhere between 42.95 and 43.05.

If we lay the two rods end-to-end, their combined actual length will be somewhere between 85.4 and 85.6. These bounds on the actual combined length don't give us even one decimal place of precision. We could say that the combined length is 85.5 +/- 0.1, but that is less precision than we started with, which was +/- 0.05.

If you take the average of the two rod lengths, you get 42.75 +/- 0.05, which represents a number somewhere between 42.7 and 42.8. As it happens, 42.75 is right smack in the middle of that interval, by there is no justifaction whatsoever for the 5 in the hundredths' place.

 

[zgozvrm]

If you take the average of the two rod lengths, you get 42.75 +/- 0.05, which represents a number somewhere between 42.7 and 42.8. As it happens, 42.75 is right smack in the middle of that interval, by there is no justifaction whatsoever for the 5 in the hundredths' place.

I agree with everything you said here except for your last statement.
The fact that the average lies somewhere in the interval $$42.7 \le X < 42.8$$ is exactly why 42.75 is the best representation of the average. You can show that 42.75 is closer to the actual average approximately 67% of the time.



I included an Excel spreadsheet that simulates this.

If you already have Excel open, close it before opening this document so that recursive settings can be automatically set.

When you open the document, the cursor will be on the yellow box which contains a "1." Change this value to anything other than 1 to allow the program to run. Enter a "1" to reset the counts back to zero.

After changing the number in the yellow box from "1", press "F9" to make the spreadsheet calculate new random values. Hold "F9" down for several seconds or minutes to generate many values and counts.

The top values generate 2 random "measurements" that, when rounded to the nearest 0.1, give 42.5 and 43.0. The actual average of these values is calculated, as is the rounded average (which will be either 42.7 or 42.8), the average of the rounded values (which, of course calculates to 42.75), and the rounded value of this (42.8).

The last three of these averages are compared to the real, actual average by finding the difference between each these averages and the actual average, the lowest of which is highlighted in gray.

After each generation of random values, the appropriate count is incremented showing the number of times each value was closest to the actual average. This is then represented as a percentage.

 

[zgozvrm]

If only the values 42.75 and 42.8 are compared to the actual average, then 42.75 will be closer approximately 87% of the time.

 

[Mark44]

I agree with everything you said here except for your last statement.
The fact that the average lies somewhere in the interval is exactly why 42.75 is the best representation of the average.

All we know is that the average is somewhere between 42.7 and 42.8. If a measurement or calculation is given as 42.75 +/- 0.05, anything between 42.7 and 42.8 is within tolerance. You are making assumptions based on how the values are distributed - I am not.

 

[zgozvrm]

All we know is that the average is somewhere between 42.7 and 42.8. If a measurement or calculation is given as 42.75 +/- 0.05, anything between 42.7 and 42.8 is within tolerance. You are making assumptions based on how the values are distributed - I am not.

What I am saying is that 42.75 is a better approximation of the average between 42.5 and 43.0 than either 42.7 or 42.8.

Granted, after only one reading, this may not be true; 42.7 may actually be the average of the actual values (if they were 42.45 and 42.95), but the OP's original question referred to a climate change, which implies several 100's or 1000's of readings.

 

[Mark44]

If we're talking about the average of the numbers 42.5 and 43.0, yes, the average is 42.75. But if we're talking about values measured to the nearest tenth, then there is no reason to say with any assurance that the average is 42.75. About the best we can do is say how big the band is that contains the average, either as an interval such as [42.7, 42.8] or as a nomimal value with a tolerance, as in 42.75 +/- 0.05.

 

[zgozvrm]

If we're talking about the average of the numbers 42.5 and 43.0, yes, the average is 42.75. But if we're talking about values measured to the nearest tenth, then there is no reason to say with any assurance that the average is 42.75. About the best we can do is say how big the band is that contains the average, either as an interval such as [42.7, 42.8] or as a nomimal value with a tolerance, as in 42.75 +/- 0.05.

I'm not saying that the average is exact, only that by adding that extra degree of precision, we are closer to the exact value over many measurements, which goes back to the original question.

Agreed, we don't know what the exact average is if even one measurement is not known exactly. But, by taking the average of the values you do have, I'm simply saying that you can add that extra decimal place (more precision), as it is likely to be closer to the actual average than if you don't

 

[Mark44]

I'm not saying that the average is exact, only that by adding that extra degree of precision, we are closer to the exact value over many measurements, which goes back to the original question.

Agreed, we don't know what the exact average is if even one measurement is not known exactly.

Every time you measure something, the measurement is inexact.

But, by taking the average of the values you do have, I'm simply saying that you can add that extra decimal place (more precision), as it is likely to be closer to the actual average than if you don't

 

[zgozvrm]

Every time you measure something, the measurement is inexact.

Not true. Counting items is a measurement of quantity.

As for measurements not involving counts ("real" measurements) we can only be as accurate as our measuring devices allow us to be. In most cases, we estimate the last figure. For example, when reading a ruler that has graduations every 0.1 of an inch, we can usually estimate fairly accurately the measurement to the nearest 0.025 of an inch (one quarter of a graduation). Although, measurements from a ruler with 0.020 inch graduations can probably not be estimated to be any more accurate than that (with the naked eye). Whatever our means of measurement, we should only make estimates to the degree that we can guarantee the accuracy. (In the case of the 0.020 inch ruler, it would be ridiculous to try estimate to the nearest 0.005, or 1/4th of a graduation).

So, as long as we can guarantee our accuracy to a given level, we can then take an average of several of those readings at the next higher level of accuracy (one more decimal place).

If I take measurements with my 0.020 inch ruler of 1.26, 3.68, and 2.42, the sum is 7.36, and the average is 2.453. Given that our original 3 measurements are measured to the nearest 0.020 inch, the average of 2.453 is a truer representation of the average than 2.45

 

[Mark44]

When people talk about "measuring" something, they almost never mean this in the sense of counting things.

There is some statistical justification for your extra place of precision, which I believe statdad hinted at many posts ago in this thread - the standard error of the mean. If you take repeated measurements of something with an arbitrary distribution, the distribution of these measurements will be approximately normal, and their standard deviation will be sigma/sqrt(n). This has the effect of bunching them more tightly around the population mean (which we don't know).

 

[statdad]

What I am saying is that 42.75 is a better approximation of the average between 42.5 and 43.0 than either 42.7 or 42.8.

Not true, as has been pointed out - if that were the case, the entire idea of confidence intervals and estimation would be worthless.

If the discussion is about means (I assume that is what "average" is in reference to), the idea is that when repeated samples are taken, the sample to sample variability in means is

$$\frac{\sigma}{\sqrt n}$$

where the numerator is the population standard deviation. If this isn't know, the sample standard deviation is used. The phrase "standardard error" is given to this quantity.

The idea is that gathering a sample and averaging smooths out the variability that occurs from individual to individual, and so statements about the mean of a group are more accurate than statements about individual values.

Caveats: the "root n" decrease is correct for random samples; even a little correlation among the items in the sample can give different results. Hampel & Ronchetti, in "Robust Statistics: The Approach Based on Influence Functions" (think I remember the title correctly: my copy is upstairs) discuss, in the final chapter, some of the real problems that can occur with data when this law isn't satisfied.

 

[zgozvrm]

Then why in a program that randomly selects two values, A & B such that 42.45<= A < 42.55 and 42.95 <= B < 43.05 (which round to 42.5 and 43.0 respectively) to 12 decimal places and then the average M of these is taken by M = (AB)/2 to 13 decimal places, after 10,000 iterations of the program, I show that 85.44% of the time, 42.75 is closer to the actual average M than 42.8?