Jan 24
books
Discuss the course texts, all listed on the
resources page:
- Biggs "Codes: ..."
- Shannon's "Mathematical Theory of Communication"
- MacKay's "Information Theory ..."
- wikipedia: information entropy ... and other topics & references there
First is very math-ish; 2nd is great but of limited scope;
3rd is very good at times but wordy and mostly aimed at another topic.
This subject can get very technical (e.g. Cover & Thomas).
overview
- compression
- error correction
- crypto
We'll start with entropy (defining it, calculating it)
and then move to compression (huffman, LZW, etc).
homework
Define & explain :
alphabet, string, message, word
code
uniquely decodable (UD)
prefix-free (PF)
kraft-mcmillan number = sum n[j]/2**j , where n[j] = number of codewords of length j
optimal code & "average word length"
information entropy = sum( - p[i] log(p[i]) ), where p[i] = probability of symbol i
huffman code
source, probability, conditional probability
intuition
In basic physics, entropy of a system is ln[states].
- why logarithm? Answer: we want it to add, for 2 systems, but number of states multiplies.
- what is it, really? Measures how likely a set of states with same macroscopic properties is.
- temperature as measure of entropy with energy
How does this connect with Shannon's entropy?
- First idea : if something can happen N ways, then
ln(N) = - ln(1/N) = - ln(p)
- Second idea: not *total* entropy, but entropy *per_symbol* . So we need to average.
- Definition of average: (make sure this is clear)
mean(x) = sum p(x) * x
H = - sum p_i * ln(p_i)
discussion
Work an example ... perhaps from Shannon's paper.
Start describing Huffman code ... perhaps from someone who's done it before.
... and see how far we get.
