DL_module

This repository and README contains relevant information regarding the Deep Learning module, please read it all carefully.

The machine learning framework we will use is PyTorch.

Table of contents

1. Objectives
2. Course contents
3. Assessments
4. Primer materials and bibliography
5. Lecture materials
6. Google Colab
7. Teaching team
8. Learning outcomes

Objectives

The objective of this module is to equip you with a solid foundation to understand the basic principles of deep learning (DL).

Despite the humongous size of the deep-learning research landscape, and the impossibility to cover all the topics it encompasses, this course will provide you with all the fundamental concepts, building blocks, and mathematical methods that are used in the majority of modern complex applications, preparing you to develop your professional careers in this field.

Course contents

During the course we will cover the following topics:

1. Introduction to Deep Learning
2. Basic DL concepts (backprop, regularisation, etc).
3. PyTorch primer
4. Feed-forward Networks (FFNs)
5. Convolutional neural networks (CNNs)
6. Probability for DL
7. Generative models:
   • Variational autoencoders (VAEs)
   • Generative adversarial networks (GANs)
   • Diffusion models
8. Recurrent neural networks (RNNs) and long short-term memory networks (LSTMs)
9. Transformers
10. Other DL methods, architectures, and strategies (informative session, not assessed):

The contents will be delivered using jupyter notebooks that contain both theory and practical implementations.

Most days will be structured as follows:

time	session
9:00h-12:00h	theory and implementation
14:00h-17:00h	exercise session

with a few exceptions:

• On Wednesday afternoons we will not have any sessions.

• Other dates that will not follow the morning-lecture/afternoon-exercise structure will be:

session	activity
Thursday 7 Dec (afternoon) & Friday 8 Dec (all day)	coursework 1 release and working time
Thursday 14 Dec (afternoon) & Friday 15 Dec (all day)	coursework 2 release and working time
Thursday 14 Dec (morning)	Q&A and review session before coursework 2

Assessments

The module assessment is based on two courseworks.

COURSEWORK 1	Date and time
Release	Thursday 7 Dec 14:00h
Due (deadline)	Friday 8 Dec 18:00h

COURSEWORK 2	Date and time
Release	Thursday 14 Dec 14:00h
Due (deadline)	Friday 15 Dec 18:00h

Primer materials and bibliography

It is NOT mandatory to read (or view) any of the materials in this section.

Introductory videos

To help prepare for the course, it is recommended to watch these four short (15-20 mins) videos which provide a good introduction to Machine Learning:

1. But what is a Neural Network? | Deep learning, chapter 1
2. Gradient descent, how neural networks learn | Deep learning, chapter 2
3. What is backpropagation really doing? | Deep learning, chapter 3
4. Backpropagation calculus | Deep learning, chapter 4

Probability introductory videos:

1. Binomial distribution
2. Normal distribution
3. Bayes theorem

Bibliography

• Pattern recognition and Deep Learning (Christopher M. Bishop)
• Deep Learning (Ian Goodfellow, Yoshua Bengio, Aaron Courville)

Lecture materials

The table below contains the course materials and their scheduled dates. Lecture slides/notebooks will be released on the morning before the start of the lectures. Two notebooks will be provided for each session (morning or afternoon):

• Codealong/Exercises notebook: Template with some code provided and empty blocks to work on the implementation during the lecture (Practical) or during the afternoon exercises (Exercise).
• Solutions notebook: Practical/Exercise notebook completed with model solutions for the tasks done during the lecture or to be completed during the afternoon exercises.

We encourage you to work on the Practical/Exercise notebook and try to implement the tasks proposed during the lectures, but at the same time we want to provide the delivery of the module contents in a complete format to accomodate different learning styles. In particular, some people prefer to focus their attention on the lecture before attempting to implement code themselves, and this is why we provide solutions beforehand.

[The numbers in the table (01:, 02:, etc), notebook names, Teams' meetings, and other materials, corresponds to the day of the course (from 01 to 15).]

Session	Date Time	Lecture/Exercises	Solutions
Day01 morning - Intro, Colab, and FFNs	Nov 28 09:00-12:00	lecture
Day01 afternoon - Intro, Colab, and FFNs	Nov 28 14:00-17:00	exercises	solutions
Day02 morning - DL_concepts	Nov 29 09:00-12:00	lecture	solutions
Day02 afternoon - DL_concepts (additional material)	Nov 29 not taught	exercises	solutions
Day 03 morning - PyTorch	Nov 30 09:00-12:00	lecture	solutions
Day 03 afternoon - PyTorch	Nov 30 14:00-17:00	exercises	solutions
Day04 morning - CNNs part1	Dec 1 09:00-12:00	lecture	solutions
Day04 afternoon - CNNs part1	Dec 1 14:00-17:00	exercises	solutions
Day05 morning - CNNs part2	Dec 4 09:00-12:00	lecture Probability
Day05 afternoon - CNNs part2	Dec 4 14:00-17:00	exercises	solutions
Day06 morning - VAEs	Dec 5 09:00-12:00	lecture theory VAEs	solutions
Day06 afternoon - VAEs	Dec 5 14:00-17:00	exercises	solutions
Day07 morning - GANs	Dec 6 09:00-12:00	lecture	solutions
Day07 afternoon - GANs (additional material)	Dec 6 not taught	exercise_cGAN exercise_WGAN	solution_cGAN solution_WGAN
Day08 morning - recap, Q&A	Dec 7 09:00-12:00
Day08 afternoon - release CW1	Dec 7 14:00
Day09 all day - CW1	Dec 8 deadline 18:00h
Day10 morning - Diffusion models	Dec 11 09:00-12:00	lecture	solutions
Day10 afternoon - Diffusion models	Dec 11 09:00-12:00	exercises	solutions
Day11 morning - RNNs & LSTMs	Dec 12 09:00-12:00	lecture/exercises slides	solutions
Day11 afternoon - RNNs & LSTMs	Dec 12 14:00-17:00	use morning notebook	use morning notebook
Day12 morning - Transformer & other DL methods	Dec 13 09:00-12:00	lecture
Day12 afternoon - RL (additional material)	Dec 13 not taught	codealong	solutions
Day13 morning - recap, Q&a	Dec 14 09:00-12:00
Day13 afternoon - release CW1	Dec 14 14:00
Day14 all day - CW1	Dec 15 deadline 18:00h

The links in the table will become active as we progress during the course.

Google Colab

All the coding will be done using Google Colab. It is also possible to use your own computer and run the jupyter notebooks locally, if you prefer, but limited support will be available to help you set up your local system.

There will be an introductory session on how to use Google Colab on Day 01 (27 Nov). We will also provide a Google Colab Pro licence for each of you on Day 08 (6 Dec), for that we will have a live session during class to set up the Colab Pro with provided virtual credit cards.

Do not buy any Colab Pro license as they will provided in class

A new google account will be created on the first day of the module, which will be a dedicated account for the course. Do NOT use your existing google account for this because the payment system provided could be extended to any personal payment details you have in your personal account

Teaching team

• Lluis Guasch (email) - module coordinator
• Raul Adriaensen
• Oscar Bates
• Carlos Cueto
• Jack Ma
• Debbie Pelacani Cruz
• George Strong
• Kun Wang
• Hongcheng Xie
• Di Xu
• Weilin Zhang

Learning outcomes

Over the next three weeks, you will be able to go from here:

XKCD 1838

to understanding complex network architectures, how and why they work, and when to use them:

Transformers