Information theory is the scientific study of the quantification, storage, and communication of digital information. The field was fundamentally established by the works of Harry Nyquist and Ralph Hartley, in the 1920s, and Claude Shannon in the 1940s. The field is at the intersection of probability theory, statistics, computer science, statistical mechanics, information engineering, and electrical engineering.
A key measure in information theory is entropy. Entropy quantifies the amount of uncertainty involved in the value of a random variable or the outcome of a random process. For example, identifying the outcome of a fair coin flip (with two equally likely outcomes) provides less information (lower entropy) than specifying the outcome from a roll of a die (with six equally likely outcomes). Some other important measures in information theory are mutual information, channel capacity, error exponents, and relative entropy. Important sub-fields of information theory include source coding, algorithmic complexity theory, algorithmic information theory, and information-theoretic security.
Applications of fundamental topics of information theory include lossless data compression (e.g. ZIP files), lossy data compression (e.g. MP3s and JPEGs), and channel coding (e.g. for DSL). Its impact has been crucial to the success of the Voyager missions to deep space, the invention of the compact disc, the feasibility of mobile phones and the development of the Internet. The theory has also found applications in other areas, including statistical inference, cryptography, neurobiology, perception, linguistics, the evolution and function of molecular codes (bioinformatics), thermal physics, quantum computing, black holes, information retrieval, intelligence gathering, plagiarism detection, pattern recognition, anomaly detection and even art creation.
Hi. I'm an undergrad majoring in maths/physics from a US university now taking a year abroad at university of Bristol (UOB) in UK. After this academic year, I'll go back to study in my home university for a year and then graduate. The quantum information (QI) researches at the department of...
Hi,
I have a square grid that represents a landscape, each grid cell is forested or non-forested. I am calculating 2 different forest fragmentation metrics. Because there is a finite number of combinations of forest and nonforest cells, there is a finite number of possible values for each...
I want to be a physics major and I am very interested Astrophysics as a final career path. However recently, after reading a few books about it I have become interested in Information Theory and the prospect of Quantum Computers. I have a very strong Math, Physics, and Computer Science...
Let's suppose I have a speech signal with frequency content >300 Hz. I then add noise to this signal, that happens to be somewhere below 300 Hz. I then high-pass filter the signal (300+ Hz) and I have increased the mutual information and seemingly violated the data processing inequality.
Can...
Right guys i am on my 3rd year at uni doing computer network management & design, i have passed everything apart from network design and I am having to take a resit.
I don't understand information theory AT ALL.
They went over it in class but made no sense, i also asked the tutor to...
I present Knit - The new (knowledgeable) http://ca.geocities.com/na2boodie01/KNIT/qkpt.html" .
My rules of engagement are for comments and debatable points that reference the material directly or has an impact on the content of the material, otherwise extraneous comments will be interpreted...
need sum help please:) (information theory)
Hello.
I'm working on an information theory-related problem that involves doing a nasty sum. The problem is this: in a widget factory there is a conveyor belt with N widgets on it, and an unknown fraction \xi = a/N of them are defective. You...
need sum help please:) (information theory)
Hello.
I'm working on an information theory-related problem that involves doing a nasty sum. The problem is this: in a widget factory there is a conveyor belt with N widgets on it, and an unknown fraction \xi = a/N of them are defective. You...
Hi,
Some Qs on information -- something rather new to me. I'm not quite sure what to do with these:
Homework Statement
a) Lack of information is defined as
S_{info} = - \sum_{i} p_{i}log_{2}p_{i}
(p_{i} is the probability of the outcome i). Calculate the entropy associated with the...
There are in fact many "information theories".
Broadly speaking all of these actually define a notion of "information" in terms of a notion of "entropy". In fact, from a basic notion of "entropy", many of these theories define a "conditional entropy" and from this "information".
The most...
Is information theory the crucible of astrophysics in the 21st century? The number and quality of cosmologically related IT papers in the past year is impressive. I am admittedly swayed by this approach. For example:
arXiv:0708.2837
The Physics of Information
Authors: F. Alexander Bais, J...
I have recently been studying Gregory Chaitin's "algorithmic information theory" for a school project. It describes the complexity of mathematical objects by the size of the smallest Turing machine program capable of computing them (in bits). It also defines a "random" object as one with an...
I have to choose between two classes because of a schedule conflict:
CS 575, combinatorics and graph theory
This course is taught by the chief undergraduate advisor and earlier I mentioned to him that I'd be taking the course. So politically it may be a good idea. The course says it...
There, now that I've got your attention:
I started this username about 6 months ago, and i haven't logged on for quite some time. I posted a few things on my journal/blog, or whatever it was back then, and now it seems as though the format for them has changed, and they are no longer there...
Here is the problem I am having trouble with:
Prove that for any cipher that has perfect secrecy, the size of the key space is at least as large as the size of the cyphertext space.
For those rusy with information security, this essentially means proving that for each message p in...