Deep Learning Course

⇢ Fran├žois Fleuret

You can find here the draft of the materials for the EPFL course EE-559 “Deep Learning” that I will teach next Spring.

These documents are under heavy development, in particular due to pytorch updates. Please avoid to distribute the pdf files, and share the URL of this page instead.

Info sheet: dlc-info-sheet.pdf

We will use the pytorch framework for implementations. You can find below a Linux virtual machine for the practical sessions.

Thanks to Adam Paszke, Alexandre Nanchen, Xavier Glorot, Matus Telgarsky, and Diederik Kingma, for their help, comments, or remarks.

Practical session prologue

Helper python prologue for the practical sessions: dlc_practical_prologue.py

Course material

Course 1 – Introduction and tensors

Thumbnail made from a slide

What is deep learning, some history, what are the current applications. torch.Tensor, linear regression.

Course 2 – Machine learning fundamentals

Thumbnail made from a slide

Empirical risk minimization, capacity, bias-variance dilemma, polynomial regression, k-means and PCA.

Course 3 – Multi-layer perceptrons

Thumbnail made from a slide

Linear classifiers, perceptron, linear separability and feature extraction, Multi-Layer Perceptron, gradient descent, back-propagation.

Course 4 – Convolutional networks and autograd

Thumbnail made from a slide

Generalized acyclic graph networks, torch.autograd, batch processing, convolutional layers and pooling, torch.nn.Module.

Course 5 – Optimization

Thumbnail made from a slide

L1 and L2 penalty. Weight initialization, Xavier's rule, loss monitoring. torch.autograd.Function.

Course 6 – Going deeper

Thumbnail made from a slide

Theoretical advantages of depth, rectifiers, drop-out, batch normalization, residual networks, advanced weight initialization. GPUs and torch.cuda.

Course 7 – Computer vision

Thumbnail made from a slide

Deep networks for image classification (AlexNet, VGGNet), object detection (YOLO), and semantic segmentation (FCN). Data-loaders, neuro-surgery, and fine-tuning.

Course 8 – Under the hood

Thumbnail made from a slide

Visualizing filters and activations, smoothgrad, deconvolution, guided back-propagation. Optimizing samples from scratch, adversarial examples.

Course 9 – Autoencoders and generative models

Thumbnail made from a slide

Transposed convolution layers, autoencoders, variational autoencoders, non volume-preserving networks.

A virtual machine for the practicals

Here is a virtual machine with Debian Linux, miniconda, pytorch and torchvision, MNIST, and CIFAR10 already installed.

This allows you to have a virtual PC completely installed for the practical sessions, running on your Windows, macOS, or Linux computer, without any change to your system.

To use it, first download and install Oracle's VirtualBox, then download:

dlc_amd64 miniconda.ova (3.5Gb)

and open the latter in VirtualBox with

File -> Import Appliance

Note that performance for computation will not be as good as if you install pytorch natively on your machine (which is possible only on Linux and macOS at the moment). In particular, using the VM does not allow to take advantage of a GPU if you have one.

Sharing a folder from the host

You can mount in the VM a directory from the host (that is, the machine and OS where VirtualBox is running), so that you can save files in the VM and access them from your machine.

To do so, in the VirtualBox manager graphical interface go to

Settings -> Shared Folders

and there add a folder whose path is a directory from the machine where VirtualBox is running, and whose name (i.e. the identifier to pass to the mount command from inside the VM) is "share".

Then, in the VM, use the provided command "mounthost" to mount the said directory to /home/user0/host/.

If VirtualBox is running on a linux host, you can add or remove a shared directory from the command line instead of using the graphical interface, respectively with

vboxmanage sharedfolder add "dlc_amd64 miniconda" --name share --hostpath <absolute path>

and

vboxmanage sharedfolder remove "dlc_amd64 miniconda" --name share