shannon's theorem
... and one of the homework problems

7.8 - an example code with information rate > 0.6

Suppose that the sender wishes to code blocks of size k with words of length n, using a 1-error correcting code.  If it is required to transmit information with an information rate of not less than 0.6, what are the smallest possible values of n and k ?

The parameters are
	n 	number of bits in a codeword
	k	number of bits of information in that codeword
	information rate =ρ≡ (k/n) ≥ 0.6
	r	error correct = 1
	δ	min hamming distance between codewords = 2 r + 1 = 3
For r=1, we need a each codeword to exist in its own neighborhood of (n+1) words, so we have the packing result
	|C| (1+n) ≤ (2^n)
Since we need a codeword for each plaintext block of k bits,
	|C|=(2^k)
Therefore
	(2^k)(1+n)≤ (2^n)
or
	1+n≤(2^(n-k))=(2^(n(1-(k/n))))≤(2^(n(1-ρ)))
or
	(((1+n))^(1/n))≤ (2^(1-ρ))

Note that for any given value of ρ, we can always choose a value of n large enough to make this inequality true.

The picture looks like this.

Plot[{0.6,1-Log[2,(((1+n))^((1/n)))]},{n,1,12},AxesLabel->{"n","ρ"},PlotLabel->"\!\(TraditionalForm\`\*SuperscriptBox[\((1 + n)\), \(1/n\)] = \\ \*SuperscriptBox[\(2\), \(1 - ρ\)]\)"]

The exact value of n where these two curves cross is

FindRoot[((((1+n))^(1/n))- (2^(1-ρ)))/.ρ->0.6,{n,8}]

But of course we need n to be an integer, so it must be at least 8.

First we try n=8 ... which as it turns out won't work.  In that case, k is at least ρ n = 0.6 8=4.8, which means k=5 or bigger.
Then the packing inequality is violated :

(2^k)(1+n)/.{k->5,n->8}

(2^n)/.{n->8}

((2^k)(1+n)≤ (2^n))/.{k->5,n->8}

Another way to see that these values won't work is to look at the Hamming codes, which give the "best" 1-error correcting codes.  With (n,k)=(8,5), we have n-k=m=3.  The hamming code with m=3 has only (2^m)-1=7 independent basis vectors, not 8.  Or if we try to keep m=3 with (n,k)=(7,4), then the information rate of 4/7=0.57 is too low.

The next value to try, then, is n=9.  Then k is at least ρ n = 0.6 9=5.4, so k=6.  Again we check the packing inequality :

((2^k)(1+n)≤(2^n))/.{k->6,n->8}

And again we strike out.  Time to get systematic.

Do[
ρ=0.6;
k=Ceiling[n ρ];
Print["{n,k}=",{n,k}," gives (2^k)(1+n)≤(2^n) is ", ((2^k)(1+n)≤(2^n))],
{n,4,12}
]

So the smallest ones that work are {n,k}={10,6}.

The problem doesn't ask for it, but let's construct such a code, just for clarity.

The generating matrix will therefore have 10 rows and 6 columns.  In standard form, the top 6 by 6 is the identity matrix, and the bottom 4 by 6 must be made of columns with at least two 1's, so that each column has weight 3.  Constructing one of these isn't hard.  For example, here's one with the desired properties :

E=(({{1, 0, 0, 0, 0, 0}, {0, 1, 0, 0, 0, 0}, {0, 0, 1, 0, 0, 0}, {0, 0, 0, 1, 0, 0}, {0, 0, 0, 0, 1, 0}, {0, 0, 0, 0, 0, 1}, {0, 0, 1, 1, 1, 0}, {0, 1, 0, 1, 0, 1}, {1, 0, 0, 0, 1, 1}, {1, 1, 1, 0, 0, 0}}))

The corresponding check matrix, also in standard form, would be

H=(({{0, 0, 1, 1, 1, 0, 1, 0, 0, 0}, {0, 1, 0, 1, 0, 1, 0, 1, 0, 0}, {1, 0, 0, 0, 1, 1, 0, 0, 1, 0}, {1, 1, 1, 0, 0, 0, 0, 0, 0, 1}}))

Is this a hamming code?  No; then H would be 4 rows by (2^4)-1=15 columns.

Is it a perfect code?  No; there are some words which are more than 1 away from any codeword.

Excersise: find a codeword that's more that 1 away.

One answer: {0,0,0,0,0,0,1,1,1,1}.  

How do we know?  Multiply by H.  Result doesn't match any of the columns, so it can't be 1 away.

Shannon's Theorem

What does Shannon's Theorem say about codes like this?  

Well, it doesn't say anything directly about any particular code, like the one in the last example.  But it does connect the idea of "channel capacity" with the "information rate".

To make things concrete, I'm going to do a specific example.

From my earlier notes, if the source probability is p , the channel conditional probabilities are Γ, and the received probabilities are q, then :

p = {p0, 1-p0};
Γ = {{Γ00, (1-Γ00)}, {(1-Γ11),Γ11} };
q = p . Γ;
pANDq=Table[p[[i]] Γ[[i,j]],{i,1,2},{j,1,2}];

For any list of probabilities, the entropy is

H[prob_]:=Total[-prob Log[2,prob]];

And in terms of that entropy, the channel capacity is 
	(bits of info received) / (bits of info sent)
= capacity
= H(p)-H(p|q) = H(p)-(H(p&q)-H(q))
 =H(p)+H(q) - H(p & q)

theCapacity=H[p]+H[q]-H[Flatten[pANDq]];

Now I'm going to assume that the initial probabilities are (1/2, 1/2), and that the Γ matrix is symmetric, with a probability of ϵ for a bit to switch due to an error.

cap[e_]:=theCapacity/.{p0->1/2,Γ00->1-e,Γ11-> 1-e}

Simplify[cap[x]]

A bit of algebra (or just doing the same calculation by hand) can reduce this even further.

capacity[e_]:=1+(1-e)Log[2,1-e]+e Log[2,e];

Plot[capacity[e],{e,0,1},AxesLabel->{"bit error prob","channel capacity"}]

This has the form we'd expect.  If no bits flip, then the capacity =(bits received)/(bits sent)=1.  If all bits flip, information content is the same.  If each bits flip with 50/50 chance, no information arrives.

Error correcting codes are all very well ... but if no information arrives, and the received bits have no connection to the sent bits, we aren't going to correct much.

Shannon's theorem says exactly this :

	If the information rate of the error correcting code is less than the channel capacity, 
	then for any desired rate of mistakes, there is a code which produces fewer mistakes than that.
	
In symbols, with 
	ρ = code information rate = k/n = (bits of plain text)/(bits in codeword)
	γ = channel capacity = (bits of information received)/(bits of information sent)
	n = length of codeword in bits
	k = ρ n = length of corresponding plain text in bits

In mathematical gobbledygook, that looks something like this :

	(γ > ρ)  => ∀m∀p∃ n { (Subscript[C, n])⊆ (B^n) , |(Subscript[C, n])|≥(2^k) , Mistake((Subscript[C, n]),p)≤m}

The idea is that for a fixed information rate, as the code words get longer and longer, the (2^k) different code words can be further and further apart, correcting as many errors as you want, and thus making the overall "mistake" probability vanishingly small ... as long as the errors aren't too likely.  

Shannon's theorem lets you figure out explicitly exactly what "too likely" means, namely that the plaintext/codeword ratio is not bigger than the channel capacity.  In other words, the requirement is that the redundancy put in by the extra bits in the codewords is at least as big as the uncertainty added by the errors.

7.8 - how big an error rate is under Shannon's value ?

So, let's get back to a concrete example.

The code introduced from exercise 7.8 has a known information rate.  What is the corresponding bit error probability from Shannon's Theorem?  Presumbably this code (and other ρ = 0.6 with bigger n values) can do something reasonable with this error rate.

Plot[{0.6,capacity[e]},{e,0,1},AxesLabel->{"bit error prob","channel capacity"}]

eValue=FindRoot[capacity[e]==0.6,{e,0.1}]

For this code, n=10, so each codeword sent has 10 bits.  What are the probabilities of getting a given number of errors?  Well, we get a binomial expansion :

	prob(no errors) = (((1-ϵ ))^10)
	prob(1 error)    = 10 (ϵ^1) (((1-ϵ))^9)
	prob(2 errors)  = (10 9/2) (((ϵ^2)(1-ϵ))^8)
... and so on.

The whole thing looks like this :

Table[{m,Binomial[10,m](e^m)(((1-e))^(10-m))}/.eValue,{m,0,10}]//MatrixForm

Since we can correct 0 or 1 bit errors, the probability of a mistake is ( 1 - prob(0) - prob(1)) :

approxMistakePerWordProbability =1-((((1-e))^10)+10e(((1-e))^9))/.eValue

I should note that this analysis isn't quite right.  

In particular, the model for the source, which was equal probabilities of 0's and 1's, isn't quite what we're sending: we're sending codewords, which are made up of specific patterns of 0's and 1's, so our capacity calculation is off.

To do a more accurate job, we'd need to have a model of the probability of codewords (say, the same probability for each), and then from that and the encoding matrix calculate the probability of 1's and 0's, and then from that calculate the channel capacity.  Any movement of the probabilities p(0) and p(1) away from 50/50 will give us a lower channel capacity for a given symmetric bit error rate.

What happens when the information rate is above the channel capacity ?

Here's what this one looks like for some other mistake rates .  
(All these have the same information rate = 0.6 and n=10.)

mistakeProb[e_]:= 1-((((1-e))^10)+10e(((1-e))^9));

Table[{e,capacity[e],mistakeProb[e]},{e,0.05,0.3,0.05}]//MatrixForm

With these codewords, the probability of a mistake starts to grow pretty quickly as the probability of a bit error increases.