Optimizing Rank p Matrix V for Symmetric Matrices with SVD Using Frobenius Norm

[doodle]

I have this matrix problem:

Given $R_1, R_2, R_3\in\mathbb{R}^{N\times N}$ are symmetric matrices with rank $p<N$. Their SVD are $U_1\Sigma_1 U_1^T$, $U_2\Sigma_2 U_2^T$ and $U_3\Sigma_3 U_3^T$, respectively. I want to find a rank $p$ matrix $V$ such that

$$J = \|V\Sigma_1 V^T - U_1\Sigma_1 U_1^T\|_F^2 + \|V\Sigma_2 V^T - U_2\Sigma_2 U_2^T\|_F^2 + \|V\Sigma_3 V^T - U_3\Sigma_3 U_3^T\|_F^2$$

is minimized, subject to the constraint $V^T V = I$.

I tried using the trace for the Frobenius norm and ended up with

$$2V (\Sigma_1^2 + \Sigma_2^2 + \Sigma_3^2) - 4(U_1\Sigma_1 U_1^T V \Sigma_1 + U_2\Sigma_2 U_2^T V \Sigma_2 + U_3\Sigma_3 U_3^T V \Sigma_3) + V(\Lambda + \Lambda^T) = 0$$

where $\Lambda$ contains the Lagrange multipliers. I have no idea how to continue from here. Any help would be appreciated.

 

[doodle]

I take it that there is no simple solution here?

In the case where p = 1, the solution for V (when I tried to work it out) is the eigenvector corresponding to the largest eigenvalue of

$$\Sigma_1 R_1 + \Sigma_2 R_2 + \Sigma_3 R_3$$

 

[tacman]

This is a very interesting and challenging problem. The goal of finding a rank p matrix V that minimizes the Frobenius norm of the difference between V\Sigma_i V^T and U_i\Sigma_i U_i^T is essentially equivalent to finding the closest rank p approximation to each of the symmetric matrices R_i. The use of the SVD for each R_i is a good start, as it provides an efficient way to decompose each matrix into its singular values and corresponding orthogonal matrices.

One potential approach to solving this problem is to use the Eckart-Young-Mirsky theorem, which states that the best rank p approximation to a matrix A can be found by truncating its SVD to the first p terms. This means that for each of the matrices R_i, we can find the best rank p approximation by keeping only the first p singular values and corresponding columns of U_i and V_i.

However, this approach does not take into account the constraint V^T V = I. To incorporate this constraint, we can use the Lagrange multiplier method as you have already started to do. This involves introducing a set of Lagrange multipliers \Lambda and solving for V that satisfies the optimality conditions. This may involve finding the eigenvalues and eigenvectors of a matrix involving \Lambda, which can be complex and computationally expensive.

Another approach could be to use an iterative algorithm such as the alternating least squares (ALS) method. This method involves alternating between solving for V while keeping U_i fixed, and then solving for U_i while keeping V fixed. This process is repeated until convergence is achieved. This method has been shown to be effective for finding low-rank approximations to matrices with certain constraints.

In summary, finding the optimal rank p matrix V for symmetric matrices with SVD using the Frobenius norm is a challenging problem that may require a combination of techniques such as the Eckart-Young-Mirsky theorem and the Lagrange multiplier method or the ALS algorithm. More research and experimentation may be needed to find the best approach for your specific problem.