Remove dropout from decoder cell state

[richardburleigh]

Fix FP16 stagnation at "OVERFLOW! Skipping step. Attempted loss scale.."

[thepowerfuldeez]

Thank you! That helped me

[candlewill]

but why?

[mychiux413]

It helped me, too.
And I noticed that the tacotron2 only apply dropout on hidden state.

After several studies,
there seems to be no consensus on how to dropout RNN,
and many papers discussed this.

Here is my opinion:
The intuition of dropout is: "Can't rely on any one feature, so have to spread out weight.",
so if we apply dropout on hidden state,
which means we don't want those gates only depend on some specific input features.

But the cell states run directly along the entire chain of RNN, to achieve the long memory behavior. Therefore, if we drop the cell state weights for each recurrent, this seems to mean that we do not want the memory to pass too long?

[Jeevesh8]

@mychiux413 But how would that lead to gradient overflow ?

[chazo1994]

It helped me, too. And I noticed that the tacotron2 only apply dropout on hidden state.

After several studies, there seems to be no consensus on how to dropout RNN, and many papers discussed this.

Here is my opinion: The intuition of dropout is: "Can't rely on any one feature, so have to spread out weight.", so if we apply dropout on hidden state, which means we don't want those gates only depend on some specific input features.

But the cell states run directly along the entire chain of RNN, to achieve the long memory behavior. Therefore, if we drop the cell state weights for each recurrent, this seems to mean that we do not want the memory to pass too long?

@mychiux413 But how about the quality of fp32 model after change the code like this commit ?