Post
51
New post on Wayne's Blog:
Focal loss gamma for a small language model: the math, the code, and how I use it
I train my small language models with focal loss instead of plain cross-entropy. It is one extra multiplier on each token's loss, (1 - p)^γ, that shrinks the contribution of tokens the model already predicts with confidence. The idea is from the RetinaNet paper (Lin et al., 2017), not mine. This post puts the two side by side in math and in PyTorch, shows the three things I added to make it work for language model training, and ends with how I would suggest setting gamma.
Read the Full Post:
https://wayne.theworkmans.us/posts/2026/11/2026-11-17-focal-loss-gamma-for-small-language-models.html
Focal loss gamma for a small language model: the math, the code, and how I use it
I train my small language models with focal loss instead of plain cross-entropy. It is one extra multiplier on each token's loss, (1 - p)^γ, that shrinks the contribution of tokens the model already predicts with confidence. The idea is from the RetinaNet paper (Lin et al., 2017), not mine. This post puts the two side by side in math and in PyTorch, shows the three things I added to make it work for language model training, and ends with how I would suggest setting gamma.
Read the Full Post:
https://wayne.theworkmans.us/posts/2026/11/2026-11-17-focal-loss-gamma-for-small-language-models.html