Collection of my notes on neural network and practical examples

📓 NN and DL 📓

Theory

Logistic Regression as a Neural Network

1. Logistic Regression as a Neural Network
2. Cost Function
3. Gradient Descent
4. Computational Graph
5. Logistic Regression Gradient Descent
6. Examples of Gradient Descent

Python and Vectorization

7. Vectorization
8. Vectorized Implementation of Logistic Regression
9. Computation of Vectorized Logistic Regressions Gradient
10. Broadcasting in Python
11. Numpy Vectors
12. Justification of Logistic Regression Cost Function

Shallow Neural Network

13. Neural Networks Overview
14. Vectorizing Implementation of Neural Network
15. Activation Functions
16. Derivatives of Activation Functions
17. Gradient Descent for Neural Networks
18. Random Initialization

Deep Neural Network

19. Deep L-layer neural network
20. Forward Propagation in a Deep Network
21. Getting matrix dimensions right
22. Why deep network?
23. Building blocks of deep neural networks
24. Forward and Backward Propagation
25. Parameters vs Hyperparameters
26. What does this have to do with the brain?

Machine Learning Application

27. Train\Dev\Test sets
28. Bias and Variance Tradeoff
29. Basic Recipe for Machine Learning

Neural Network Regularization

30. Regularization
31. Dropout Regularization
32. Other Regularization Methods

Optimization problem

33. Normalizing Inputs
34. Vanishing Exploding Gradients
35. Weight Initialization for Deep Networks
36. Numerical approximation of gradients
37. Gradient Checking

Optimization Algorithms

38. Mini Batch Gradient
39. Exponentially weighted averages
40. Bias correction in exponentially weighted averages
41. Gradient descent with momentum
42. Optimization Algorithms RMSprop
43. Optimization Algorithms Adam
44. Learning rate decay
45. Local optima

Hyperparameter tuning

46. Tuning Process

Batch Normalization

47. Normalizing activations in a network
48. Fitting Batch Norm into a neural network
49. Why does Batch Norm work?
50. Batch Norm at test time

Multi-class classification

51. Softmax Regression
52. Training a softmax classifier

Deeplearning Programming Framework

53. Deeplearning programming framework
54. TensorFlow

📓 ML in Practice 📓

ML Strategy

55. Why Machine Learning Strategy
56. Orthogonalization

Goal Setting

57. Single number evaluation metric
58. Satisficing and Optimizing metric
59. Train/dev/test distributions
60. Size of dev/test data
61. When to change dev/test sets and metrics

Comparing to human-level performance

62. Why human-level performance?
63. Avoidable bias
64. Understanding human-level performance
65. Surpassing human-level performance
66. Improving your model performance

Error Analysis

67. Error analysis
68. Cleaning incorrectly labeled data
69. Build system quickly then iterate

Mismatched training and dev/test set

70. Training and testing on different distributions
71. Bias and Variance with mismatched data distributions
72. Addressing data mismatch

Learning from multiple tasks

73. Transfer learning
74. Multi-task learning

End-to-end deep learning

75. What is end to end deep learning
76. Whether to use end to end deep learning

📓 Convolutional Neural Nets 📓

Theory

77. Computer Vision
78. Edge Detection
79. Padding
80. Strided Convolution
81. Convolutions Over Volume
82. One Layer Convolutional Network
83. Simple Convolutional Network
84. Pooling Layer
85. CNN Example
86. Why Convolutions are useful?

Deep convolutional models: case studies

87. LeNet-5, ALexNet, VGG-16
88. ResNets
89. Networks in Networks and 1x1 Convolutions
90. GoogLeNet: Inception Network

ConvNet in Practice

91. Data Augmentation
92. State of Computer Vision

Object Detectction

93. Object Lcoalization
94. Landmark Detection
95. Object Detection
96. Convolutional Implementation of Sliding Windows
97. YOLO algorithm
98. IoU
99. Non max suppression
100. Anchor Boxes
101. R-CNN

Special applications: Face recognition & Neural style transfer

102. Face Recognition
103. neural style transfer
104. Convolutional Networks in 1D and 3D

Sequence models

105. Sequence Data Examples

RNN

106. Notations
107. Recurrent Neural Network Model
108. Language model and sequence generation
109. Sampling novel sequences
110. GRU
111. LSTM
112. Bidirectional RNN

Word Embedding

113. Word Representation
114. Learning word embeddings
115. Nueral Network Language model
116. Word2Vec
117. GloVe
118. Debiasing word embedding

Sequence to Sequence model

119. Basic Model
120. Picking the most likely sentence
121. Beam Search Algorithm
122. Error Analysis in Beam Search
123. Attention Model
124. Speech Recognition

Generative Adversarial Nets

• GAN
• BiGAN

Applied GAN

• GAN for anormaly detection
  • AnoGAN
  • EGBAD

Summation of Important Papers

• [Word2vec]
• [FastText]
• [Elmo]
• seq2seq
• Transformer
• Universal Language Model Fine-tuning for Text Classification
• [Tranformer XL]
• [Universal Sentence Encoder](https://stomioka.github.io/deeplearning/docs/
• [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding]
• [GPT2]
• [MN-DDN]
• [XLNet]

TensorFlow Basic

• Using Callbacks to control training
• Simple covnet
• MNIST classification example
• Convnet with TF
• Image augmentation

NLP in TensorFlow

• Sentiment Analysis
• Embeddings
• Sentiment Analysis - Embeddings
• IMDB Subwords 8K with Single Layer LSTM
• IMDB Subwords 8K with Multi Layer LSTM
• IMDB Subwords 8K with 1D Convolutional Layer
• Sarcasm with Bidirectional LSTM
• Sarcasm with 1D Convolutional Layer
• Generating Text

Sequence/time series and predictions

• examples
• Common patterns in time series
• Train, validation, and test
• Metrics for evaluating performance
• Moving average and differencing
• Trailing versus centered windows
• Statistical Forecasting
• Time series Forecasting Notebook

Deep neural network for time series

• Preparing features and labels
• Sequence bias
• Feeding windowed dataset into neural networks
• Single layer neural network
• Machine learning on time windows
• Prediction
• More on single layer neural network
• Deep neural network training, tuning and prediction
• RNN for time series

Real-world time series data

• Convolutions+LSTM
• Real data sunspots

Implementations in Python and R

Implementations in R:

1. Neural Network Classification
2. Simple 2 hidden layer network with Keras

Implementations in Python:

1. Building an image recognition algorithm using logistic regression with neural network mindset
2. Building a 2 layer neural network for binary classification problem
3. Building neural network utilities
4. Building a 2 layer neural network and deep learning neural network from scratch
5. Initialization
   • initialize_parameters_zeros
   • initialize_parameters_random
   • initialize_parameters_he
6. Regularization
   • compute_cost_with_regularization
   • backward_propagation_with_regularization
   • forward_propagation_with_dropout
   • backward_propagation_with_dropout
7. Gradient Checking
   • gradient_check
   • gradient_check_n
8. Optimization
   • update_parameters_with_gd
   • random_mini_batches
   • initialize_velocity
   • update_parameters_with_momentum
   • initialize_adam
   • update_parameters_with_adam
9. TensorFlow: Build neural network for multiclassification problem
10. Building Convnet with numpy
11. Building Convnet with TensorFlow
12. Building ResNets with Keras
13. AEDECOD and AETERM similarity with Universal Sentence Encoder with Transformer
14. Objects detection using pre-trained Yolo weights
15. Neural Style Transfer
16. Face Recognition
17. Character level language moodel - LSTM
18. Generate Text - LSTM
19. NLP tasks