Comparing Non-Uniform Data Sets

In summary, the student is trying to figure out a way to display a summary of the results of multiple reviewers on a set of items. There are problems with scaling the data, selecting a single sample of data, and dealing with large numbers of students.
  • #1
JPierce
6
0
I'm not sure my title is very descriptive, but I tried my best. I also hope I am posting this in the right forum. If not, please let me know. (I thought it might be better posted in the social sciences forum.)

I have a project where I am analyzing the results of multiple reviewers on a set of items. I am unsure as to the proper method of normalizing the data.

In essence, here is the problem:

We have a large stack of assignments turned in by students, with one assignment turned in by each student. Teachers analyzed each assignment according to three criteria, which I will call A, B, and C. Teachers values for each of these criteria on a scale from 1 to 5. However, this scale is not linear, so A=4 is not twice as "big" as A=2.

I simply want to display a summary of the results using (say) a histogram. I have no interest in calculating summary statistics because the numerical values for each criterion are purely denumerable -- that is, A = 2.4 (which could correspond to say grade level) is meaningless.

So far, so good. But some of the assignments were reviewed by up to five teachers. Others were reviewed by only one.

So one assignment turned in by (say) Jimmy may have the following reviews from five individual teachers:

A = 3; B = 1; C = 4
A = 3; B = 2; C = 4
A = 3; B = 1; C = 2
A = 2; B = 3; C = 4
A = 3; B = 2; C = 4

Another assignment turned in by Mary may only have A = 3; B = 1; C = 4 as measured by a single teacher.

So, how do we handle the fact that some assignments have more reviews than others? We could just scale up the number of reviews to a common value. In other words, we could pretend that Mary turned in five identical assignments, that is,

A = 3; B = 1; C = 4
A = 3; B = 1; C = 4
A = 3; B = 1; C = 4
A = 3; B = 1; C = 4
A = 3; B = 1; C = 4

Somehow that doesn't seem quite right. And it would foul up the precision of the results.

Another idea is to whittle down the number of reviewers to 1 for each assignment, but I have no good criteria for selecting the one sample to keep.

Any ideas?

If I have left out important info, just let me know.
 
Physics news on Phys.org
  • #2
Most reasonable would seem that every student gets an A, B and C grade and each grade would be the average of the grades that the different teachers gave.

So that you would have:

Jimmy: A=2.8, B=1.8, C = 3.6
Marry: A=3.0, B=1.0, C=4.0
 
  • #3
JPierce said:
I have no interest in calculating summary statistics because the numerical values for each criterion are purely denumerable -- that is, A = 2.4 (which could correspond to say grade level) is meaningless.

Oh, I am not really sure what you mean here, but it seems to indicate that only whole numbers are meaningful. In that case take medians instead of averages. So that you would have:

Jimmy: A=3, B=2, C=4
Marry: A=3, B=1, C=4
 
  • #4
JPierce said:
...We have a large stack of assignments turned in by students, with one assignment turned in by each student. Teachers analyzed each assignment according to three criteria, which I will call A, B, and C. Teachers values for each of these criteria on a scale from 1 to 5. However, this scale is not linear, so A=4 is not twice as "big" as A=2.

I simply want to display a summary of the results using (say) a histogram...But some of the assignments were reviewed by up to five teachers. Others were reviewed by only one.

It sounds kind of similar to the Collaborative Filtering problem made famous by the Netflix Prize, but here if the number of students is not too large it should be possible to present all the data with a few charts.

For example the scores for criteria A could be represented by an Nx5 grid: number the students 1 to N according to some overall rating (such as average total score) and make the colour/brightness of cell (i,j) according to how many reviewers gave student i score j. Then repeat for criteria B and C (so all the data is in 3 grid charts or they could be combined into one with RGB colour mix). I don't know if any packages specifically do this type of chart but it should be possible in Excel using conditional formatting.
 
  • #5
Thanks for all suggestions.

The number of students involved is very large -- tens of thousands.

I will look into the Netflix problem.

Using the median might work.

If anyone has more suggestions, please offer them. I will continue reading responses.
 

FAQ: Comparing Non-Uniform Data Sets

1. How do you compare two non-uniform data sets?

To compare two non-uniform data sets, it is important to first determine the type of data you are working with. If the data is numerical, you can use statistical measures such as mean, median, and mode to compare the central tendencies of the two sets. If the data is categorical, you can use bar graphs or pie charts to visualize and compare the frequencies of each category.

2. What is the best way to visualize non-uniform data sets?

The best way to visualize non-uniform data sets depends on the type of data and the purpose of the comparison. For numerical data, histograms, box plots, or scatter plots can be used to show the distribution and outliers. For categorical data, bar graphs, pie charts, or stacked bar graphs can be used to show the frequencies and proportions of each category.

3. How do you handle missing data when comparing non-uniform data sets?

Handling missing data in non-uniform data sets can be challenging, as it can affect the accuracy and reliability of the comparison. One approach is to remove the missing data points, but this can reduce the sample size and potentially skew the results. Another approach is to impute the missing values using statistical methods such as mean or median imputation. However, this can also introduce bias in the results and should be done carefully.

4. Can non-uniform data sets be compared using statistical tests?

Yes, non-uniform data sets can be compared using statistical tests, but the choice of test depends on the type of data and the research question being addressed. For numerical data, tests such as t-test, ANOVA, or Mann-Whitney U test can be used. For categorical data, chi-square test or Fisher's exact test can be used. It is important to ensure that the assumptions of the chosen statistical test are met before interpreting the results.

5. How do you deal with outliers when comparing non-uniform data sets?

Outliers can significantly affect the results when comparing non-uniform data sets. One approach is to remove the outliers, but this should be done carefully after considering the possible reasons for the outliers. Another approach is to transform the data using methods such as logarithmic or square root transformation, which can help to reduce the impact of outliers. It is also important to report the presence of outliers and the method used to handle them in the analysis.

Similar threads

B Large Scale Data Collection
Replies
9
Views
975
B Experiment to test for causality
Replies
30
Views
563
MHB Calculating statistical values from given data
Replies
1
Views
856
MHB Game Night : Number of combinations
Replies
14
Views
1K
B Subsets of Rational Numbers and Well-Ordered Sets
Replies
2
Views
2K
MHB Proof of a set union and intersection
Replies
1
Views
983
I Computer languages as examples in formal logic
2
Replies
40
Views
7K
I What are the recommended tests for comparing two sets of data?
Replies
6
Views
996
I Understanding 2 equivalent formulations of both data set measures
Replies
9
Views
1K
I Thinking about equality of infinite sets
Replies
5
Views
1K

Hot Threads

Recent Insights

Back
Top