3. Show that the rows of the DCT have zero dot product.

The basic idea here is that any two cosine basis vectors of different frequencies are perpendicular.

dct[[2, All]] . dct[[4, All]]

2.4747*10^-16

This is perhaps less than convincing with Mathematica doing all the work.
The "dot" notation is actually doing this calculation

dct[[2, 1]] dct[[4, 1]] + dct[[2, 2]] dct[[4, 2]] + dct[[2, 3]] dct[[4, 3]] + dct[[2, 4]] dct[[4, 4]]

2.22045*10^-16

Created by Mathematica  (April 1, 2004)