REPLY TO HUSZÁR: The elastic weight consolidation penalty is empirically valid

In their recent work on elastic weight consolidation (EWC), Kirkpatrick et al show that forgetting in neural networks can be alleviated by using a quadratic penalty whose derivation was inspired by Bayesian evidence accumulation. In his letter, Dr Huszar provides an alternative form for this penalty by following the standard work on expectation propagation using the Laplace approximation. He correctly argues that in cases when more than two tasks are undertaken the two forms of the penalty are different. Dr. Huszar also shows that for a toy linear regression problem his expression appears to be better. Here, they appreciate Huszar for pointing out the discrepancy between the standard expectation propagation and EWC in the multitask case.