Calculus of Variations on Kullback-Liebler Divergence

In summary, the conversation discusses a task given in a machine learning class, where the goal is to minimize the KL-divergence by choosing q to be equal to p, given a fixed p. The person asking for help is struggling to understand the concept of calculus of variations and how to use it in this scenario, but is advised to start by adding constraints to the Euler-Lagrange equations and to try to get optimal q to maximize or minimize the KL-divergence. Some helpful resources are also provided for further understanding.
  • #1
Master1022
611
117
TL;DR Summary
How to use calculus of variations on KL-divergence
Hi,

This isn't a homework question, but a side task given in a machine learning class I am taking.

Question: Using variational calculus, prove that one can minimize the KL-divergence by choosing ##q## to be equal to ##p##, given a fixed ##p##.

Attempt:

Unfortunately, I have never seen calculus of variations (it was suggested that we teach ourselves). I have been trying to watch some videos online, but I mainly just see references to Euler-Lagrange equations which I don't think are of much relevance here (please correct me if I am wrong) and not much explanation of the functional derivatives.

Nonetheless, I think this shouldn't be too hard, but am struggling to understand how to use the tools.

If we start with the definition of the KL-divergence we get:
[tex] \text{KL}[p||q] = \int p(x) log(\frac{p(x)}{q(x)}) dx = I [/tex]

Would it be possible for anyone to help me get started on the path? I am not sure how to proceed really after I write down ## \frac{\delta I}{\delta q} ##?

Thanks in advance
 
Mathematics news on Phys.org
  • #2
Euler Lagrange is what you want, but you also have to worry about the conditions that you have on q that come from it being a probability distribution, namely that the integral is 1 and it's always nonnegative. I think the integral constraint is the important part

http://liberzon.csl.illinois.edu/teaching/cvoc/node38.html

Has some notes on how to add constraints to the euler Lagrange equations.
 
  • Like
Likes Master1022
  • #3
Maybe you should start with fixed p, and then try to get optimal q to maximize or minimize KL.
 

FAQ: Calculus of Variations on Kullback-Liebler Divergence

What is the Kullback-Liebler Divergence?

The Kullback-Liebler Divergence, also known as KL Divergence or relative entropy, is a measure of how different two probability distributions are from each other. It is commonly used in information theory and statistics to compare the similarity between two distributions.

How is the Kullback-Liebler Divergence used in Calculus of Variations?

In Calculus of Variations, the Kullback-Liebler Divergence is used as an objective function to optimize. By minimizing the KL Divergence between a given distribution and a target distribution, we can find the optimal parameters or functions that best fit the data.

What is the relationship between KL Divergence and Information Theory?

KL Divergence is closely related to Information Theory, as it measures the amount of information lost when approximating one distribution with another. It can also be interpreted as the amount of additional information needed to encode data from one distribution using a code optimized for another distribution.

Can KL Divergence be used for continuous distributions?

Yes, KL Divergence can be used for both discrete and continuous distributions. However, for continuous distributions, the integral form of KL Divergence is used instead of the summation form used for discrete distributions.

What are some applications of Calculus of Variations on KL Divergence?

Calculus of Variations on KL Divergence has various applications in fields such as machine learning, signal processing, and image processing. It can be used for tasks such as data compression, feature selection, and parameter estimation.

Similar threads

I Question on Calculus of variations formalism
Replies
2
Views
1K
I A doubt about the multiplicity of polynomials in two variables
Replies
8
Views
2K
I About divergence, gradient and thermodynamics
Replies
4
Views
1K
I Help with some confusions about variational calculus
Replies
22
Views
971
Why are y and y' treated as independent in calculus of variation?
Replies
3
Views
2K
I Solving the Boat Crossing a River Problem using the Calculus of Variations
Replies
33
Views
3K
A Can position and velocity vary independently in Hamilton's Principle?
Replies
2
Views
980
I Can function transformation result in a constant variation?
Replies
9
Views
1K
Isosceles triangle in information theory
Replies
4
Views
1K
I Is My Calculus of Variations Approach Correct?
Replies
12
Views
2K

Hot Threads

Recent Insights

Back
Top