Parameter Dynamics as Indicators of Model Complexity Efficiency and Scope

Abstract

Mathematical models of human cognitive and neurocognitive processes must explain complex phenomena but, at the same time, need to retain parsimony of description. Aside from usual criterions of model selection, such as goodness-of-fit, many other criterions of model selection focusing more on details of parameter dynamics of computational models have been proposed recenty (e.g., Li, Lewandowksky, 1996; Myung et al., 2000). The measurements of model complexity and its implications for understanding the computational complexity of the cognitive (or neurocognitive) systems remain to be important topics of theoretical researches. We extend previous researches on parameter dynamics of digital filters to examine weight sensitivity and interdependence in feedforward networks. Weight sensitivity refers to the effect of small weight perturbations on the network's output, and weight interdependence refers to the degree of co-linearity between weights. A combined measure of the weight space (t), defined as the ratio of weight interdependence to sensitivity, is explored in networks with hidden-unit activation functions of different complexity in the contexts of learning (1) a non-linearly separable bivariate normal classification task, (2) the XOR problem, (3) sigmoidal functions, and (4) sine functions. Simulations show that networks with more complex activation functions give rise to a smaller t and more rapid learning, suggesting that weight sensitivity and interdependence together are indicative of network complexity and are predictive of learning efficiency. These results demonstrate that recent differential geometric methods for ascertaining model complexity may be generalized to analyze the complexity of adaptive learning networks, which are the bases of a variety of computational models of cognition and neurocognitive processes. References: Li, S.-C., Lewandowsky, S., & DeBrunner, V. E. (1996). Using parameter 　sensitivity and interdependence to predict model scope. Journal of 　Experimental Psychology: General, 125, 360-369. Myung. I. J. et al. (2000). Counting probability distributions: Differential 　geometry and model selection. Proceedings of National Academy of 　Sciences USA, 97, 11170-11175.