Minimizing memory loss in learning a new environment

Human and other living species can learn new concepts without losing the old ones. On the other hand, artificial neural networks tend to "forget" old concepts. In this paper, we present three methods to minimize the loss of the old information. These methods are analyzed and compared for the linear model. In particular, a method called network sampling is shown to be optimal under certain condition on the sampled data distribution. We also show how to apply these methods in the nonlinear models.