Differential Learning Rates in PyTorch

GROWING This note is being developed. It is readible but changes are being applied.
til/machine-learning/pytorch/pytorch-differential-learning-rates.md Current Note ID: The unique ID of this note.
#Python #PyTorch #Learning Rate

Using different learning rates in different layers of our artificial neural network.

PyTorch offers optimizer configuration for different learning rates in different layers.

Why Do We Do This

In some models, we need to treat the layers differently. For example, in transfer learning, we could fine tuning the pretrained layers using a tiny learning rate.

In the documentation of pytorch, we find that we can set optimizer parameters on a per-layer basis 1. The example from the documentation is

optim.SGD([
    {'params': model.base.parameters()},
    {'params': model.classifier.parameters(), 'lr': 1e-3}
], lr=1e-2, momentum=0.9)

  1. PyTorchDocs torch.optim — PyTorch 1.10.0 documentation. [cited 30 Nov 2021]. Available: https://pytorch.org/docs/stable/optim.html  ↩︎

Planted: by ;

References:
  1. Pointer2019 Pointer I. Programming PyTorch for deep learning: Creating and deploying deep learning applications. Sebastopol, CA: O’Reilly Media; 2019.
  2. PyTorchDocs torch.optim — PyTorch 1.10.0 documentation. [cited 30 Nov 2021]. Available: https://pytorch.org/docs/stable/optim.html
Similar Articles:
PyTorch: Initialize Parameters
We can set the parameters in a for loop. We take some of the initialization methods from Lippe1. To …
Learning Rate
Find a good learning rate
Documentation
Documenting my data science project using sphinx or mkdocs-material
Cookiecutter
Use cookiecutter to initialize a project
Links to:
Learning Rate
Find a good learning rate

L Ma (2021). 'Differential Learning Rates in PyTorch', Datumorphism, 11 April. Available at: https://datumorphism.leima.is/til/machine-learning/pytorch/pytorch-differential-learning-rates/.