Per-layer learning rates and weight decays

[alexbw]

With the advent of hyperparameter tuners like Spearmint and SigOpt, and with recent results such as http://arxiv.org/abs/1502.03492, it seems useful to expose per-layer learning rates. Is this a possibility?

[alexjc]

Advanced training algorithms like Adagrad, Adadelta, RMSProp basically provide per weight adaptive learning... I don't know if pylearn2 provides other ways to tweak learning rates per layer.

It's easy enough to expose if the underlying libraries support it.

[alexjc]

Per-weight layer decay is already supported, learning rates bit harder to do. I'll consider it if there's enough demand.