Compensating for outliers during Standard Deviation Calculation ?

In summary, the conversation discusses the issue of calculating standard deviation for a group of data with gaps in the readings. The speaker provides an example and suggests that ignoring the gaps may not be the best approach. They also mention that the data may indicate a change after a certain point and suggest investigating that instead of ignoring the gaps.
  • #1
Aston08
22
0
I am trying to calculate the standard deviation of a group of data based on a 20 period sliding window. I have run into a bit of a problem in knowing how to deal with gaps up or down in the readings and was wondering what the correct method for compensating for this was.Below is an example of the situation I am trying to compensate for:

102
103
101
105
103
102
103
101
105
103
95
92
94
93
92
95
92
94
93
92Obviously there is a big gap from 103 to 95, but in this particular situation that is not of significance to me and I would like to filter it out if possible as these spikes tend affect the readings that follow.
 
Mathematics news on Phys.org
  • #2
Outliers are indiviual measurements that are significantly different from the rest of the collection. You don't have any outliers here, what you have appears to be two separate populations.

You could measure the SD of this collection of data, but it would be a fairly meaningless number - indeed the mean itself (98) is meaningless.

Your data are trying to tell you that something happened after the 10th sample; I'd suggest you investigate that rather than try and ignore it.
 

FAQ: Compensating for outliers during Standard Deviation Calculation ?

1. How do you identify outliers in a dataset?

Outliers can be identified by plotting the data on a graph and looking for data points that are significantly distant from the rest of the data. They can also be identified using statistical methods such as the interquartile range or z-score.

2. Why is it important to compensate for outliers during standard deviation calculation?

Outliers can significantly affect the value of the standard deviation, making it an inaccurate measure of the spread of the data. Compensating for outliers helps to provide a more accurate representation of the data and prevent misleading conclusions.

3. What methods can be used to compensate for outliers?

There are various methods that can be used to compensate for outliers, such as trimming, winsorizing, or using robust estimators such as the median absolute deviation. These methods involve removing or adjusting the outliers to minimize their impact on the standard deviation calculation.

4. How do you decide which method to use for compensating outliers?

The choice of method for compensating outliers depends on the nature of the data and the research question at hand. It is important to carefully consider which method is most appropriate for the specific dataset and research objectives.

5. Can compensating for outliers completely eliminate their effect on the standard deviation?

No, compensating for outliers cannot completely eliminate their effect on the standard deviation. However, it can significantly reduce their impact and provide a more accurate measure of the spread of the data.

Similar threads

I What is the significance of standard deviation for arbitrary distributions?
Replies
4
Views
1K
Can You Decode This List of Numbers into a Message?
Replies
18
Views
12K
I Finance: does the volatility of a stock follow a certain distribution?
Replies
24
Views
3K
B Chebyshev inequality, confidence intervals, etc
Replies
6
Views
3K
B How to statistically calculate the final value?
Replies
7
Views
817
Calculations using Standard Deviation and Mean
Replies
2
Views
2K
I Error propagation and standard deviation
Replies
3
Views
6K
Calculating Standard Deviation for a Sample Mean
Replies
2
Views
1K
Understanding the meaning of "expected fraction" (Statistics)
Replies
4
Views
2K
How to calculate standard deviation from the delay?
Replies
4
Views
3K

Hot Threads

Recent Insights

Back
Top