Statistics problem: Comparing written work with & w/out use of AI

  • #1
TULC
10
0
I want to compare performance on written work under different conditions, for example with and without the use of AI, according to some specified criteria. Assume the written work is a critical analysis of specific content.

The written work will be scored on a number of dimensions, such as creativity etc. The goal is to gain some understanding - based on a large sample of written samples - of the extent to which AI can improve the written work. This will be a way to develop a benchmark against which we can compare individual written samples. If the correlation b/w individual written work with and w/out use of AI is sig. weaker than expected based on the analysis of a larger sample of written work, then one could argue that this warrants a question: is AI being overused by the individual?

Given the above, would calculating correlation coefficients be a good choice here? I want something simple that can be used with ease by almost anyone. At the same time, I acknowledge the fact that I am manipulating some variables, so a correlational approach may not be ideal. If so, what alternatives, if any, would you suggest?
 
Last edited by a moderator:
Physics news on Phys.org
  • #2
You might have more luck posting this in the statistics group. The moderators might move it if you ask them. (If you post it there yourself they will complain about a duplicate.) You can make this request by hitting the "Report" button and typing it in.
 
  • #3
Thread closed for Moderation...
 
  • #4
TULC said:
I want to compare performance on written work under different conditions, for example with and without the use of AI, according to some specified criteria.
This is not allowed at PF for two reasons: first, we don't allow discussion of personal research; and second, we don't allow discussions based on AI-generated content.

Thread will remain closed.
 
  • #5
Hornbein said:
You might have more luck posting this in the statistics group.
Not this topic, no. See previous post.
 

Related to Statistics problem: Comparing written work with & w/out use of AI

What are the key metrics to compare written work with and without the use of AI?

Key metrics often include accuracy, coherence, readability, creativity, and adherence to guidelines. Accuracy measures the correctness of information, coherence assesses logical flow, readability evaluates ease of understanding, creativity looks at originality, and adherence to guidelines checks compliance with given instructions.

How can we ensure a fair comparison between AI-generated and human-written work?

To ensure fairness, use a controlled environment where both AI and human writers are given the same tasks, guidelines, and time constraints. Additionally, using blind evaluations where reviewers do not know the source of the work can help reduce bias.

What statistical tests are suitable for comparing the quality of AI-generated and human-written work?

Common statistical tests include t-tests for comparing means, chi-square tests for categorical data, and ANOVA for comparing multiple groups. These tests help determine if there are significant differences between the quality metrics of AI-generated and human-written work.

How can we quantify the impact of AI on the quality of written work?

Quantifying the impact involves measuring the differences in quality metrics before and after the use of AI. This can be done using statistical methods to analyze the data and calculate effect sizes, which provide a measure of the magnitude of the difference.

What are the potential biases in comparing AI-generated and human-written work?

Potential biases include reviewer bias, where evaluators may have preconceived notions about AI or human work, and selection bias, where the samples chosen for comparison may not be representative. To mitigate these biases, use blind reviews and ensure a diverse and representative sample set.

Similar threads

Replies
10
Views
2K
Replies
4
Views
2K
Replies
19
Views
2K
  • Poll
  • Science and Math Textbooks
Replies
1
Views
3K
  • Set Theory, Logic, Probability, Statistics
Replies
9
Views
2K
  • Calculus and Beyond Homework Help
Replies
1
Views
2K
  • Calculus and Beyond Homework Help
Replies
1
Views
1K
  • Introductory Physics Homework Help
Replies
2
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
7
Views
2K
Back
Top