Rolling Average vs Gradient Descent with Softmax & Cross Entropy
← Thread:Talk:Rolling Averages/Rolling Average vs Softmax & Cross Entropy
If each Guess Factor bin is considered an output unit before Softmax (logit), and loss is Cross Entropy, then the gradient of each logit is then:
- qi - 1, if bin is hit
- qi, otherwise
If gradient is not applied on logit as normal, but instead applied on qi itself, then:
- qi := qi - eta * (qi - 1) = (1 - eta) * qi + eta * 1, if bin i hit
- qi := qi - eta * qi = (1 - eta) * qi + eta * 0, otherwise
Which is essentially rolling average, where eta (learning rate) equals to the alpha (decay rate) in exponential moving average.
Anyway this analog isn't how rolling average works, as logit isn't qi actually. But what if we replace rolling average with gradient descent? I suppose it could learn even faster, as the outputs far from real value get higher decay rate...
You do not have permission to edit this page, for the following reasons:
You can view and copy the source of this page.
Return to Thread:Talk:Rolling Averages/Rolling Average vs Softmax & Cross Entropy/reply.