Evaluating similarity between two subsets

  • Thread starter bagatur
  • Start date
  • Tags
    Subsets
In summary, the problem is trying to figure out if two subsets of IDs are different by just a few numbers. The solution is to take a scalar product and divide by the total number of IDs in the two subsets.
  • #1
bagatur
2
0
i have the following problem that i can't figure out.
i have a set ID's which i pseudo-randomly split into 2 subsets A and B. let's say for the sake of simplicity i did it twice only, so i have subsets A&B and A`&B`. the sizes in the different splittings are the same - not sizes of A and B, but sizes of A and A`, and of B and B`.
what i need to know is if there is a standard way of telling how different those distributions are. i just want to make sure that subset A is not different from A` or B` by just a couple of ID's (sizes of A and B differ by 1 only).
 
Physics news on Phys.org
  • #2
Well, if you don't have a distance measure between pairs of ID's--if two ID's are either equal or unequal, no shades of gray--then your problem is simple. Define the similarity between two subsets to be the size of their intersection. If you want to compare subsets of different sizes, one possibility is to define the similarity of A and B to be 2 |A n B| / (|A| + |B|).
 
  • #3
thank you for your reply!
i don't think i was clear about my problem, but what you suggested is something what i tried to do.
ID's are all different from each other (we do clinical research and ID's are identifying numbers of our volunteers).
what i did was assign number 1 to an ID if it is in the subgroup A and -1 if it is in B. then i did the same for the second selection - assign 1 if the ID in the subgroup A1 and -1 if it is in B1.
so i get vectors for 2 different selections, let's name them C and C1, consisting of 1's and -1's.
then i just take a scalar product and divide by the total number of ID's.
in the extreme cases, if A is same as A1 and B is same as B1, we get 1; and if A is same as B1 and B is same as A1, we get -1. the closer this value to 0 is, the more they differ.
now is there some widely accepted name for this kind of calculation? like a similarity or correlation coefficient? or something else?
 

FAQ: Evaluating similarity between two subsets

What is the purpose of evaluating similarity between two subsets?

The purpose of evaluating similarity between two subsets is to determine how similar or dissimilar the two subsets are. This can help in various fields such as data analysis, genetics, and pattern recognition.

What are some methods for evaluating similarity between two subsets?

There are multiple methods for evaluating similarity between two subsets, including Jaccard Similarity, Cosine Similarity, Euclidean Distance, and Hamming Distance. These methods use different mathematical calculations to determine the level of similarity between the two subsets.

How does Jaccard Similarity work?

Jaccard Similarity is a method used to measure the similarity between two sets. It is calculated by dividing the size of the intersection of the two sets by the size of the union of the two sets. The resulting value ranges from 0 (no similarity) to 1 (complete similarity).

What is the difference between Cosine Similarity and Jaccard Similarity?

While Jaccard Similarity only takes into account the presence or absence of elements in two sets, Cosine Similarity also considers the magnitude of the elements. This makes Cosine Similarity more suitable for comparing numerical data, while Jaccard Similarity is better for comparing categorical data.

How can evaluating similarity between two subsets be useful in data analysis?

Evaluating similarity between two subsets can be useful in data analysis by helping to identify patterns and relationships between different data sets. It can also be used for data preprocessing, data clustering, and outlier detection.

Similar threads

Replies
18
Views
1K
Replies
6
Views
1K
Replies
12
Views
6K
Replies
4
Views
1K
Replies
2
Views
4K
Replies
4
Views
1K
Back
Top