April 17

Questions or comments on anything?

Please do keep working on your own data analysis projects - that's what I think you'll get the most out of at this point.

For today, I figured we'd spend one more session looking at the abstractions that libraries like tensorflow, pytorch, and the book author in chap 19 use for "deep learning".

Next week I suggest we look at chapter 17, decision trees.

deep learning APIs and concepts

Chapter 19 in the text, "deep learning", implements recursively an object-oriented abstraction of the neural net notions that we talked about last time.

This scratch.deep_learning is patterned after the ideas in libraries like tensorflow and pytorch.

With just a little intuition for what's going on underneath, you can (mostly) use these libraries as black boxes to do this kind of modeling.

I'll walk through the text some, show my jupyter notebook that does a simple example and adds a convergence plot, and point towards some tutorials you can look at.

And I've tried to collect some of the code object names and math buzzwords here in one place for you.

deeep learning libraries

• scratch.deep_learning (from chap 18 of our text)
• tensorflow & keras
• pytorch (what Joel Grus, the book author, says he uses)

"scratch" code objects (i.e. abstraction of neural net model)

• Tensor - a bunch of numbers for inputs or outputs or model parameters
• Layer - has inputs, outputs, params, and methods for forwards (inputs to outputs) & backwards (errors to model changes)
• Sequence - layers connected with in a stream of data processing
• Loss - method of evaluating the difference between the inputs and predicted outputs
• Optimizer - method of applying a small step change to the model
• Dropout - a layer that leaves out some neurons during training, to avoid overfitting

some data manipulation functions

• step function
• sigmoid function
• relu : rectified linear unit
• tanh : hyperbolic tangent
• softmax : xs => prob_xs all between 0 and 1 with sum=1.

network step modification methods

• gradient descent : the "downhill" calculus method we described last week
• momentum : continue in same direction, with some corrections; described in text

loss (i.e. error) functions

• SSE : sum of squared errors, as we have done previously
• cross entropy ... maybe discuss this a bit

my stuff

• Joel's deep_learning XOR with a plot
• my info entropy talk ... somewhat off topic ;)

(BTW, the slides for that talk were done with a jupyter notebook ... google "jupyter slides" if you want to know how.)