Deep Learning Course

⇢ Fran├žois Fleuret

Here are the draft of the materials for the EPFL course EE-559 “Deep Learning” that I will teach next Spring.

These documents are under heavy development, in particular due to frequent changes in pytorch. Please avoid to distribute the pdf files, and share the URL of this page instead.

Info sheet: dlc-info-sheet.pdf

We will use the pytorch framework for implementations. You can find below a Linux virtual machine for the practical sessions.

Practical session prologue

Helper python prologue for the practical sessions: dlc_practical_prologue.py

Course material

Course 1 – Introduction and tensors

What is deep learning, some history, what are the current applications. torch.Tensor, linear regression.

Thumbnail made from a slide

Course 2 – Machine learning fundamentals

Empirical risk minimization, capacity, bias-variance dilemma, polynomial regression, k-means and PCA.

Thumbnail made from a slide

Course 3 – Multi-layer perceptrons

Linear classifiers, perceptron, linear separability and feature extraction, Multi-Layer Perceptron, gradient descent, back-propagation.

Thumbnail made from a slide

Course 4 – Convolutional networks and autograd

Generalized acyclic graph networks, torch.autograd, batch processing, convolutional layers and pooling, torch.nn.Module.

Thumbnail made from a slide

Course 5 – Optimization

L1 and L2 penalty. Weight initialization, Xavier's rule, loss monitoring. torch.autograd.Function.

Thumbnail made from a slide

Course 6 – Going deeper

Theoretical advantages of depth, rectifiers, drop-out, batch normalization, residual networks, advanced weight initialization. GPUs and torch.cuda.

Thumbnail made from a slide

Course 7 – Computer vision

Deep networks for image classification (AlexNet, VGGNet), object detection (YOLO), and semantic segmentation (FCN). Data-loaders, neuro-surgery, and fine-tuning.

Thumbnail made from a slide

A virtual machine for the practicals

Here is a virtual machine with Debian Linux, miniconda, pytorch and torchvision, MNIST, and CIFAR10 already installed.

This allows you to have a virtual PC completely installed for the practical sessions, running on your Windows, macOS, or Linux computer, without any change to your system.

To use it, first download and install Oracle's VirtualBox, then download:

dlc_amd64 miniconda.ova (3.5Gb)

and open the latter in VirtualBox with

File -> Import Appliance

Note that performance for computation will not be as good as if you install pytorch natively on your machine (which is possible only on Linux and macOS at the moment). In particular, using the VM does not allow to take advantage of a GPU if you have one.

Sharing a folder from the host

You can mount in the VM a directory from the host (that is, the machine and OS where VirtualBox is running), so that you can save files in the VM and access them from your machine.

To do so, in the VirtualBox manager graphical interface go to

Settings -> Shared Folders

and there add a folder whose path is a directory from the machine where VirtualBox is running, and whose name (i.e. the identifier to pass to the mount command from inside the VM) is "share".

Then, in the VM, use the provided command "mounthost" to mount the said directory to /home/user0/host/.

If VirtualBox is running on a linux host, you can add or remove a shared directory from the command line instead of using the graphical interface, respectively with

vboxmanage sharedfolder add "dlc_amd64 miniconda" --name share --hostpath <absolute path>

and

vboxmanage sharedfolder remove "dlc_amd64 miniconda" --name share