noisy channels

discussion

This is a summary of chapter 5 in "Codes" text, using their notation.

The math is conditional probability and entropy again, but this time with a different interpretation.

We're back to thinking about 1st order Markov again, for now, but with an added "noise" source between the sender and receiver.

	symbol p sent ->  NOISE  -> symbol q received

	p ≡ [(Subscript[p, 1]),(Subscript[p, 2]),...] = probabilities of symbols produced by sender = (1 x N) row vector
	q ≡ [(Subscript[q, 1]),(Subscript[q, 2]), ...] = probabilities symbols seen by receiver = (1 x N) row vector

	Γ ≡[ (Subscript[Γ, ij])] = [ conditional probablity( got j | sent i ) ]
	                = probability of receiving symbol j given that i was sent = (N x N matrix)
	                = prob( q | p )
	where i = row =  sent symbol, and j = column = received symbol
		
	t ≡ [(Subscript[t, ij])] = (Subscript[Γ, ij])(Subscript[p, i]) =prob(get j| sent i) prob(sent i) = prob(get j & sent i) = (N x N matrix)
		= probability(p & q) = [ probability(sent i, got j) ] = probability(i,j)
		= joint probability distribution

We can also define the conditional probability in the other direction
	Δ ≡ [(Subscript[Δ, ij])] = [ conditional probability( sent i | got j ) ]
	                 = [ (Subscript[t, ij ])/ (Subscript[q, j])]  =  N x N matrix

For any probability distribution g that sums to 1, we already have a notion of entropy :
	H(g) ≡  - (∑_( i)) (Subscript[g, i]) (Subscript[log, 2])((Subscript[g, i]))

We now define a conditional entropy.
  
The text uses two similar notations, and shows that two definitions are equivalent.
	H(p | q)  ≡ H(p & q) - H(q) ≡ H( Γ ; p)
	= H(t) - H(q)
	= (∑_( j)) (Subscript[q, j]) H(p | j) = -(∑_( j)) (Subscript[q, j]) (∑_( i)) ((Subscript[t, ij])/(Subscript[q, j])) (Subscript[log, 2])((Subscript[t, ij])/(Subscript[q, j]))
			        =- (∑_( ij))(Subscript[t, ij]) (Subscript[log, 2])((Subscript[t, ij])/(Subscript[q, j]))
Likewise, 

	H(q | p)  ≡ H(p & q) - H(p) ≡ H( Δ ; q)
	= H(t) - H(p)
	= (∑_( i)) (Subscript[p, i]) H(q | i) = -(∑_( i)) (Subscript[p, i]) (∑_( j)) ((Subscript[t, ij])/(Subscript[p, i])) (Subscript[log, 2])((Subscript[t, ij])/(Subscript[p, i]))
			        =- (∑_( ij))(Subscript[t, ij]) (Subscript[log, 2])((Subscript[t, ij])/(Subscript[p, i]))

The "channel capacity" of the source p through the channel Γ is then defined as the difference of two of these entropies :
	(Subscript[f,  Γ])(p) ≡ H(p) - H(Γ; p) = H(p) - H(p|q)
		= bits of information about p available to someone who knows q  .

All this implies (from the definitions of conditional probabilities) a bunch of identities :
	1 = (∑_j) (Subscript[Γ, ij])	for every i	sum of each conditional probability is 1	
	1 = (∑_( i)) (Subscript[Δ, ij])	for every j	ditto
	1 = (∑_( ij))(Subscript[t, ij])			sum of all joint probabilities is 1
	(Subscript[q, j]) = (∑_( i)) (Subscript[p, i ])(Subscript[Γ, ij])		or	q = p Γ	   using matrix multiplication
	(Subscript[p, i]) = (∑_j) (Subscript[t, ij])			p marginal probability from t
	(Subscript[q, j]) = (∑_( i)) (Subscript[t, ij])			q marginal probability from t

Example :
	p = [0.7, 0.3] = [ prob(send 0), prob(sending 1)]
	Γ = [({{(Subscript[Γ, 00]), (Subscript[Γ, 01])}, {(Subscript[Γ, 10]), (Subscript[Γ, 11])}})] =  [({{prob(get 0 | send 0), prob(get 1|send 0)}, {prob(get 0 | send 1), prob(get 1 | send 1)}})] = [({{0.99, 0.01}, {0.02, 0.98}})]
	q = [0.7, 0.3] [({{0.99, 0.01}, {0.02, 0.98}})] = [0.699, 0.301] = [prob(get 0), prob(get 1)]
	t =  [({{(Subscript[p, 0])(Subscript[Γ, 00]), (Subscript[p, 0])(Subscript[Γ, 01])}, {(Subscript[p, 1])(Subscript[Γ, 10]), (Subscript[p, 1])(Subscript[Γ, 11])}})] = [({{0.693, 0.007}, {0.006, 0.294}})]
Everything else is calculated below, including the various entropies.

definitions

H[prob_]:=Total[-Flatten[prob] Log[2,Flatten[prob]]];
H2[pXY_]:=Block[{},
pX=Total[Transpose[pXY]];
Return[Sum[-pXY[[i,j]] Log[2,pXY[[i,j]]/pX[[j]]]]];
];

Γ = {{0.99,0.01},{0.02,0.98}};
p = {0.7, 0.3};
q = p . Γ;
t=Table[p[[i]] Γ[[i,j]],{i,1,2},{j,1,2}];
Δ = Table[t[[i,j]]/q[[j]],{i,1,2},{j,1,2}];

check

Γ//MatrixForm

p//MatrixForm

q//MatrixForm

t//MatrixForm

Δ//MatrixForm

Total[Δ]//MatrixForm

Total[p]

Total[q]

Total[Transpose[Γ]]//MatrixForm

Total[Total[t]]

Total[t]

Total[Transpose[t]]

entropies

Entropy of input

H[p]

Entropy of output

H[q]

Entropy of joint distribution, knowing everything

H[t]

H (q | p) calculated two ways :

1.	Prob(got| sent 0)

Γ[[1]]

Prob(got| sent 1)

Γ[[2]]

Entropy of those two conditionals

H[Γ[[1]]]

H[Γ[[2]]]

weighted average of the conditional entropies

p . {H[Γ[[1]]], H[Γ[[2]]]}

2.	difference of (joint entropy - source entropy)

H[t]-H[p]

H(p | q) = H(Γ; p) calculated two ways :

H[t]-H[q]

q . {H[Transpose[Δ][[1]]], H[Transpose[Δ][[2]]]}

channel capacity

H[p]-(H[t]-H[q])

The interpretation for these numbers then is :
	a) The source p with (0,1) probabilities of (0.7, 0.3), without noise,
	    can deliver 0.88 bits of information per bit sent.
	b) With this noisy Γ channel, which flips (0,1) with (0.01, 0.02) probability, 
	     only 0.78 bits of information per bit sent is possible.