Finding direction(s) of fastest increase/decrease

[CheMech]

Homework Statement

http://imageshack.us/photo/my-images/861/screenshot20111211at928.png/

I am only concerned with part (a) of this problem.

Homework Equations

This question was assigned in a linear algebra class while we were learning about eigenvalues and eigenvectors (I have since finished the class, but since this problem was assigned in the last week of the class before finals, I wasn't able to find enough time to go get help for this question).

The Attempt at a Solution

I learned in a previous calculus course that the gradient vector points in the direction of fastest increase, and that the negative of the gradient vector points in the direction of the greatest decrease. I figured I would use this as a check after solving it using the linear algebra method (as was the way this problem was "supposed" to be solved in this class).

So, I tried finding the gradient:

f(x, y) = 3x² + 6xy - 5y

and...

$\nabla$f = < 6x + 6y, 6x - 10y² >

However, evaluating this at (0, 0, 0) to determine the direction of greatest change yields <0,0>.

The actual answer (the linear algebra way) is that the directions of fastest increase/decrease correspond to the directions of the eigenvectors of the matrix that represents the given quadratic form.* I have a few issues with this:

1) Why do the eigenvectors' directions correspond to the direction of fastest increase/decrease? I learned before that the direction of the gradient pointed in the direction of the fastest increase (and that the negative of it pointed in the direction of the fastest decrease).

2) Looking at part (b) of this question: Based on what I said above, I would say that the angle between the two directions is 180 degrees, because the gradient and its opposite are opposite of one another. However, this turns out not to be the case according to the actual answer stated in 1).Any help at all is appreciated. This question has been bugging me :P* This:
\begin{bmatrix}
3 & 3 \\
3 & -5
\end{bmatrix} is the matrix of the quadratic form given.

 

[Zula110100100]

Don't you still get a vector of <0,0> for lambda = 3, and <0,0> for lambda = -5? I don't really understand eigenstuff, but that is what I got.

 

[CheMech]

I (and my TA) got eigenvalues with the associated eigenvectors as:

lambda = -6 with eigenvector <-1,3>

and

lambda = 4 with eigenvector <3,1>

EDIT: Yeah, just doublechecked with an eigenvector calculator online.

(Also, note how these two directions are perpendicular, which seems to contrast with the "gradient explanation" I mentioned in my earlier post.)

 

[Zula110100100]

Hmm...sorry, I can actually be of no help. As I understood it, with det(A-lambda*I)=0 you would have (3-lambda)(-5-lamda)=0 How do you get the values -6 and 4? not that they are wrong I just don't really get eigenvalues

 

[vela]

$$\begin{vmatrix} 3-\lambda & 3 \\ 3 & -5-\lambda \end{vmatrix} = (3-\lambda)(5-\lambda) - 9$$
You're missing the -9.

 

[vela]

Homework Statement

http://imageshack.us/photo/my-images/861/screenshot20111211at928.png/

I am only concerned with part (a) of this problem.

Homework Equations

This question was assigned in a linear algebra class while we were learning about eigenvalues and eigenvectors (I have since finished the class, but since this problem was assigned in the last week of the class before finals, I wasn't able to find enough time to go get help for this question).

The Attempt at a Solution

I learned in a previous calculus course that the gradient vector points in the direction of fastest increase, and that the negative of the gradient vector points in the direction of the greatest decrease. I figured I would use this as a check after solving it using the linear algebra method (as was the way this problem was "supposed" to be solved in this class).

So, I tried finding the gradient:

f(x, y) = 3x² + 6xy - 5y

and...

$\nabla$f = < 6x + 6y, 6x - 10y² >

However, evaluating this at (0, 0, 0) to determine the direction of greatest change yields <0,0>.

The actual answer (the linear algebra way) is that the directions of fastest increase/decrease correspond to the directions of the eigenvectors of the matrix that represents the given quadratic form.* I have a few issues with this:

1) Why do the eigenvectors' directions correspond to the direction of fastest increase/decrease? I learned before that the direction of the gradient pointed in the direction of the fastest increase (and that the negative of it pointed in the direction of the fastest decrease).

Are you sure that's exactly what you were told? The eigenvectors will point in the direction of the principal axes. In other words, when you rotate the coordinate system so the x and y axes line up with the eigenvectors, the cross term in the quadratic form disappears.

2) Looking at part (b) of this question: Based on what I said above, I would say that the angle between the two directions is 180 degrees, because the gradient and its opposite are opposite of one another. However, this turns out not to be the case according to the actual answer stated in 1).

What you've said about the gradient is correct. I think you're simply misinterpreting what you were told about the eigenvectors, which is causing your confusion.

Any help at all is appreciated. This question has been bugging me :P


* This:
\begin{bmatrix}
3 & 3 \\
3 & -5
\end{bmatrix} is the matrix of the quadratic form given.

 

[Office_Shredder]

What the gradient method does is approximates your function with a linear function, and then points in the direction that the linear function is increasing fastest. This is similar to the one dimensional case: given a graph and a point on the graph, draw the tangent line. This is the linear approximation of the one dimensional function, and the direction that the tangent line is increasing in is the direction the function is increasing in, and the direction the tangent line is decreasing in is the direction the function is decreasing in.

Now suppose that we have a point where the derivative is zero. It doesn't make sense to ask which direction the function is increasing/decreasing in, does it? Well, if it's a local maximum or minimum no, but if you're at an inflection point then yes, there is one direction where the function is increasing and one direction where the function is decreasing. This may seem like an exceptional case but in higher dimensions, you are more likely to have a saddle point just by the nature of the problem. For example in your case the fact that one eigenvalue is positive and the other negative means that you have a saddle point. Then you can use the eigenvectors to discover the directions in which the function has the largest positive and negative concavities. It will be worth re-reading your notes to confirm that your eigenvector test only applies to points where the gradient is zero

A question that might spark some intuition about how the higher dimensional eigenvalues/eigenvectors work: Given that I have a function f(x) and f'(0)=0, and I tell you that f''(0)=0 as well, how can you use the sign of f''(x) to the left and the right of zero to inform you as to whether the function is increasing or decreasing in each direction?

 

[CheMech]

$$\begin{vmatrix} 3-\lambda & 3 \\ 3 & -5-\lambda \end{vmatrix} = (3-\lambda)(-5-\lambda) - 9$$
You're missing the -9.

Further expanding the determinant that you have set up:

$$=-15 + 2\lambda + \lambda^2 - 9$$

So the characteristic polynomial is $$\det(A-\lambda\ I) = \lambda^2 + 2\lambda - 24$$. Setting this equal to zero to find the roots yields:

$\lambda^2 + 2\lambda - 24 = 0$

$(\lambda + 6)(\lambda - 4) = 0$

$\lambda_{1} = -6 \ \mbox{and}\ \lambda_{2} = 4$________

A question that might spark some intuition about how the higher dimensional eigenvalues/eigenvectors work: Given that I have a function f(x) and f'(0)=0, and I tell you that f''(0)=0 as well, how can you use the sign of f''(x) to the left and the right of zero to inform you as to whether the function is increasing or decreasing in each direction?

Well, I'm a tad rusty on this, but couldn't it be either increasing or decreasing on either side of zero? What I mean is $f(x) = x^3$ satisfies both f'(0)=0 and f''(0)=0, and it is decreasing in the negative x direction (moving away from the origin into the negative x direction) and increasing going in the positive x direction. On the other hand, $f(x) = -x^3$ also satisfies both f'(0)=0 and f''(0)=0. However, in this case, f(x) increases going from the origin to the negative x direction and decreases going into the positive x direction.

Are you saying since in this case, the gradient evaluated at the origin is equal to zero, that we can use eigenvectors to determine the directions in which the function has the largest negative and positive concavities (and in this problem, that would be equivalent to determining in what directions it increases/decreases most quickly)?

What happened was I went to my TA's office hours to get help, but had to leave in the middle of the explanation for this problem to get to another class. My friend who was with me said that my TA did some sort of proof of the claim that the eigenvectors point in the direction of the fastest increase/decrease, with the eigenvector associated with the negative eigenvalue pointing in the direction of the fastest decrease and the eigenvector associated with the positive eigenvalue pointing in the direction of the fastest increase. My friend didn't copy the proof down though (D'oh!).

So yeah, I'm not quite sure why this "eigenvector method" would only apply when the gradient is equal to zero. Based on the book used in my class--which doesn't address this specific topic--and what my professor briefly lectured about, all my knowledge is that the eigenvectors point in the directions of the principle axes, and writing the quadratic form in terms of these axes eliminates the xy term from a quadratic form.