How Does Least Squares Determine the Mean and Variance?

[zollen]

This problem projects b = (b₁,b₂...,b_m) onto the line through a = (1, 1, 1, ...1). We solve m equations ax = b in 1 unknown (by least squares).

(a) Solve a^T a $\hat{x}$ = a^T b to show that $\hat{x}$ is the mean (the average) of the b’s.
(b) Find e = b - a $\hat{x}$ and the variance ||e||² and the standard deviation ||e||
(c) The horizontal line $\hat{x}$ = 3 is closest to b = (1, 2, 6). Check that p = (3, 3 3) is perpendicular to e and find the 3 by 3 projection matrix P.

Ans(a): Because a = (1,1,1,...1), therefore a^T a = 1 + 1 + 1 +...+ 1 = 1 * m = m
And a^T b = b₁ + b₂ + ... + b_m
So $\hat{x}$ = (b₁ + b₂ + b₃ + ... + b_m) / m = b_avg

Ans(b): Need help..

Ans(c): Need help..

 

[zollen]

Ans(b):
Given $\hat{x}$ = b_avg (from Ans(a))
Therefore
$$e = b - a \hat{x} = \begin{bmatrix} b_1\\ b_2\\ ..\\ b_m \end{bmatrix} ~-~ \begin{bmatrix} 1\\ 1\\ ..\\ 1 \end{bmatrix} ~b_{avg} = \begin{bmatrix} e_1\\ e_2\\ ..\\ e_m \end{bmatrix}$$

$$||e||^2~=~\frac{{e_1}^2 + {e_2}^2 + ... + {e_m}^2}{m}$$
$$||e||~=~\sqrt{ \frac{{e_1}^2 + {e_2}^2 + ... + {e_m}^2}{m} }$$

Ans(c): $$p=Pb~~and~~P= \frac{a * a^t}{a^t * a}$$
$$a = \begin{bmatrix} 1\\ 1\\ 1 \end{bmatrix}$$
But...
$$a^t * a ~=~ \begin{bmatrix} 1&1&1\\ 1&1&1\\ 1&1&1 \end{bmatrix}$$ has no inverse...

Would anyone show me how to calculate the projection matrix?

 

[StoneTemplePython]

Ans(c): $$p=Pb~~and~~P= \frac{a * a^t}{a^t * a}$$
$$a = \begin{bmatrix} 1\\ 1\\ 1 \end{bmatrix}$$
But...
$$a^t * a ~=~ \begin{bmatrix} 1&1&1\\ 1&1&1\\ 1&1&1 \end{bmatrix}$$ has no inverse...

Would anyone show me how to calculate the projection matrix?

I'm not sure what you're trying to do here.

$\mathbf {aa}^T$ is a rank one matrix with trace of 3.

$\mathbf {a}^T \mathbf a = 3$ i.e. it is a scalar.

of course by cyclic property of trace and the fact that the trace of a scalar is that scalar, we have$\mathbf {a}^T \mathbf a =trace\big(\mathbf {a}^T \mathbf a\big) = trace\big(\mathbf {aa}^T\big) = 3$

Projectors (that aren't the identity) are always not invertible. Your projector is simply $\propto \mathbf {aa}^T$ i.e. you re-normalize its sole non-zero eigenvalue to be one. Then as usual, the trace operation gives you the rank of your projector.

A couple of concerns:

I'm not sure why you introduced a $*$ operation, or why you divide by "$\mathbf {a}^T \mathbf a$" in your projector formula without recognizing that it is a scalar -- in general 'dividing by a matrix' doesn't make any sense except for the very special case when it is in fact a non zero scalar that we're talking about. (Hadamard products are excluded from this writeup.)