Deep Learning Basic

choosing activation fuction

1. While logistic sigmoid neurons are more bi-ologically plausible than hyperbolic tangent neurons, the latter work better for train-ing multi-layer neural networks. Deep Sparse Rectifier Neural Networks
2. tanh vs ReLU

average pooling

1. this blog gives a clear explanation, Global average pooling--
2. alexisbcook's professional explanation
3. an explanation of average pooling with tensorflow codes
4. What are Max Pooling, Average Pooling, Global Max Pooling and Global Average Pooling?

Batch Normalization

1. Batch Normalization: Accelerating Deep Network Training byReducing Internal Covariate Shift

2. 深入理解Batch Normalization批标准化

droupout

1. dropout layer is used to overcome overfitting, refer to:Improving neural networks by preventing co-adaptation of feature detectors by Hinton

Sample/Shuffle

1. In pytorch, the use of WeightedRandomSampler makes the learning more smoothly quicker but the test dataset cannot be used with WeightedRandomSampler(TODO why ) Note: shuffe should be False when using WeightedRandomSampler

Attension

1. Attension in Neural Networks
2. Attention-mechanism
3. Attention based model 是什么，它解决了什么问题？

Time Series forecasting

Time Series in General

1. Financial Time Series Forecasting with Deep Learning -ASystematic Literature Review- 2005-2019
2. Comparison between DeepESNs and gated RNNs on multivariate time-series prediction
3. Multivariate Temporal Convolutional Network--A Deep Neural Networks Approach for Multivariate Time Series Forecasting

LTSM

1. LSTM-MSNet: Leveraging Forecasts on Sets of Related Time Series with Multiple Seasonal Patterns
2. Tensorized LSTM with Adaptive Shared Memory for Learning Trends in Multivariate Time Series
3. DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting

Dataset

1. public bicycle sharing data