Nov 19
Discuss the assigned reading :
1. NAND perceptron
x1 -> (-2 weight) -> [ ]
[ bias 3 ] -> NAND
x2 -> (-2 weight) -> [ ]
2. sigmoid
output = sigmoid( dot(weight, inputs) + bias)
sigmoid(z) = 1/(1 + exp(-z))
See the attached ipython notebook.
3. problem :
implement the NAND gate using sigmoids in the ipython notebook
4. problem :
Do the 10 outputs to 4 outputs binary layer exercise in textbook
5.
Explore the idea of a "gradient" using a two parameter function like
z = x**2 + 2*y**3
a) Start at some point, say (x,y) = (2,3)
b) How do we find small changes (dx, dy) that make z smaller?
Two answers:
i) numerical trial and error
ii) calculus, (dz/dx, dz/dy) = "gradient" gives downward direction
Discuss dot product & notion of perpendicular
6.
Explore the function that we need to minimize for the handwriting problem :
C(w, b) = 1/(2 n) sum( | y(x) - a |**2 )
where
sum is over x
x is images in the training set
n is length(x)
w is weights (how many? what subscripts?)
b is biases (how many? what suscripts?)
| ... |**2 is vector dot product, i.e. geometric length
y(x) is output of neural net given weights & biases for one image
a is correct output for a given image
7. What is "stochastic gradient descent" ?
8.
Start looking at his code.
- what is this "zip" thing?
- first task : calculate C(b,w), that is, implement forward algorithm.
For fun, see if you can find the gradient by trial-and-error :
make small changes in the weights and see what happens.
How many things to you need to try?
![[paper clip]](/courses/source/wiki_images/paper_clip_tilt.png)
name last modified size