Hypothesis Testing for Error Rate at Amazoogle.com

[blackle]

I am not sure whether this falls in this section or the other. But as the calculation of likelihood usually involves calculus, posting it here.

Homework Statement

The Big Boss at Amazoogle.com says that based on long experience, the best model for errors in software is that experienced programmers make errors at a rate of p errors per line of code (say, for p= 0.001), no matter what you do. The salesperson for Ecliptic insists their IDE will save megabucks by reducing the error rate per line to q = p/2 ("... and just for you, this week only, there's a special sale price, don't miss it, blah, blah, ..."). The Big Boss is a skeptic, but willing to experiment, and arranges for m different programs to be written using the new IDE, with lengths n1, n2, ..., nm and the QC department finds, respectively, x1, x2, ..., xm errors in them.

The Boss asks you to analyze and present the results to Management. Being a whiz at hypothesis testing, you know just what to do.

a ) Explain what hypotheses you would examine, what decision rule you would use, etc.

b) Suppose m=3, the programs are of length 2, 4, and 6 thousand lines and the number of errors found were 2, 1 and 5, respectively. How would you summarize the results of the experiment, what recommendations would you make and what are the uncertainties associated with them? Is your null hypothesis more or less likely than your alternative? Is it 5 times more or less likely?

c) After your presentation, the Big Boss says "Why didn't you just average the error rates for the 3 programs: (2/2000 + 1/4000 + 5/6000)/3 and compare that to p?" What will you answer and why?

d) B.B. also says "For the same cost, I could have had more shorter programs, say two dozen of 500 lines each, or fewer longer ones, maybe even just one of 12000 lines. Which would have been better?" What will you answer and why?

The Attempt at a Solution

a) Null Hypothesis (H0): Error per line of code is p
Alternate Hypothesis (H1): Error per line of code is q = p/2

Decision Rule: Accept H0 when error per line of code is greater than q = p/2, greater than 60% of the time

b)

Mth trial error/line Outcome
1 0.001 > q = p/2
2 0.00025 <= q = p/2
3 0.00083 > q = p/2

As approximately 66% of the time , the number or errors per line were greater than q, the null hypothesis is accepted.
As an aside: can we assume that the problem is saying p is 0.001. Or are we supposed to do it devoid of any assumption?

Now we need to write the suggestions and uncertainities?
Can I say, that we need to take a greater number of trials to give more accurate results. As a threshold of both 90% and 70% would be accepted if we use the data above.

Is your null hypothesis less or more likely? What am I expected to answer? My results depending on the threshold I chose - 60% in this case?

Is it 5 times less or more likely?
Okay, so this makes me think that I am somehow supposed to calculate the likelihood. In order to calculate the likelihood, I would need the probabilities right? Like, I believe the equation is
Probability of (Outcome | H0) / Probability of (Outcome | H1)

P(Outcome | H0):
The outcome is > q, < q, > q

However what probabilities do we use to calculate this? I am a little confused?

c)

I have no clue about the answers for both c and d as well. In fact c sounds like a reasonable suggestion to me.

I know this is long, but any help would be appreciated.

Thank you.

 

[chiro]

For d) If you had more programs that were small, the mean should converge to its proper value (regardless of distribution) and the variance should tighten as well, which means you will get a better answer with a lot of trials vs one with less trials (trial in this context is a code segment/file/atomic-block-of-some-sort).

With regards to all of the answers, wouldn't it be easier to work with a single distribution and assume that all of your samples are i.i.d variables from your population distribution?