We investigate the computational role of the feedback from layer VI of the primary visual cortex (V1) onto LGN relay cells. Our modeling is based on recent experimental results on cat and primate visual cortex that has shed light on the net polarity of feedback as a function of receptive field location of an lgn cell relative to that of a V1 column. These results postulate an excitatory feedback effect when the two receptive field locations largely overlap and an inhibitory feedback when the receptive fields are slightly offset, creating a “Mexican hat”. Our numerical experiments suggest that this feedback scheme results in an increase of information transferred from LGN to V1 when the extent of excitatory and inhibitory regions of the Mexican hat filter are within a range similar to what is observed in the experiments. We further set up an information maximization problem to learn the feedback kernel and observe that the resulting kernel has a Mexican hat shape with similar excitatory and inhibitory widths.

No abstract available.

Driven by advances in multi-cell spike recording technology, the statistics of action potential populations are revealing many new details of dynamic signals in the cortex. However, it is still difficult to integrate slow Poisson spiking with much faster spike timing signals in the gamma frequency spectrum. A potential way forward is being sparked by advances in patch clamping methodologies that allow the exploration of communication strategies that use millisecond timescales. The voltage potential of a cell's soma recorded at 20 kilohertz in vivo allows its high resolution structure to be correlated with behaviors. We show that this signal can be potentially interpreted by a unified model that takes advantage of a single cycle of cell's somatic gamma frequency to modulate the generation of its action potentials. This capability can be seen as organized into a general-purpose method of coding fast computation in cortical networks that has three important advantages over traditional formalisms: 1) Its processing speed is two to three orders of magnitude times faster than population coding methods, 2) It allows multiple, independent processes to run in parallel, greatly increasing the processing capability of the cortex and 3) Its processes are not bound to specific locations, but migrate across cortical cells as a function of time, facilitating the maintenance of cortical cell calibration.

Stochastic gradient descent (SGD) has been the core  optimization  method  for  deep  neural  networks, contributing to their resurgence.  While some progress has been made, it remains unclear why SGD leads the learning dynamics in overparameterized networks to solutions that generalize well.  Here we show that for overparameterized networks with a degenerate valley in their loss function, SGD on average decreases the trace of the Hessian. We also show that isotropic noise in the non-degenerate subspace of the Hessian de-creases its determinant. This opens the door to anew optimization approach that guides the model to solutions with better generalization.  We test our results with experiments on toy models and deep neural networks.

The forces that govern how languages assign meanings to words have been debated for decades. Recently, it has been suggested that human semantic systems are adapted for efficient communication. However, a major question has been left largely unaddressed: how does pressure for efficiency relate to language evolution? Here, we address this open question by grounding the notion of efficiency in a general information-theoretic principle, the Information Bottleneck (IB) principle. Specifically, we argue that languages efficiently encode meanings into words by optimizing the IB tradeoff between the complexity and accuracy of the lexicon. In support of this hypothesis, we first show that color naming across languages is near-optimally efficient in the IB sense. Furthermore, this finding suggests (1) a theoretical explanation for why inconsistent naming and stochastic categories may be efficient; and (2) that languages may evolve under pressure for efficiency, through an annealing-like process that synthesizes continuous and discrete aspects of previous accounts of color category evolution. This process generates quantitative predictions for how color naming systems may change over time. These predictions are directly supported by an analysis of recent data documenting changes over time in the color naming system of a single language. Finally, we show that this information-theoretic approach generalizes to two qualitatively different semantic domains: names for household containers and animal taxonomies. Taken together, these results suggest that efficient coding - a general principle that also applies to lower-level neural representations - may explain to a large extent the structure and evolution of semantic representations across languages.

Abstract neurobiological network models use various learning rules with different pros and cons. Popular learning rules include Hebbian learning and gradient descent. However, Hebbian learning has problems with correlated input data and dos not profit from seeing training patterns several times. Gradient descent has the problem of vanishing gradient for partially flat activation functions, especially in online learning. We analyze here a variant, we refer to as Hebbian-Descent, that addresses these problems by dropping the derivative of the activation function and by centering, i.e. keeping the neural activities mean free, leading to an update rule that is provably convergent, does not suffer from the vanishing gradient problem, can deal with correlated data, profits form seeing patterns several times, and enables successful online learning when centering is used.

I will describe our recent work on a deep energy model which brings together kernel density estimators and empirical Bayes least squares estimators. The energy model is the first of its kind in that the learning does not involve inference (negative samples), and the density estimation is formulated purely as an optimization problem, scalable to high dimensions and efficient with double backpropagation and SGD. An elegant physical picture emerges of an interacting system of high-dimensional spheres around each data point together with a globally-defined probability flow field. The picture is powerful and it leads to a novel sampling algorithm (walk-jump sampling), a new notion of associative memory (Robbins associative memory), and it is instrumental in designing experiments. I will finish the talk by showcasing the emergence of remarkably rich creative modes when the model is trained on MNIST. Reference: https://arxiv.org/abs/1903.02334