Capacity analysis in multi-state synaptic models: a retrieval probability perspective

We define the memory capacity of networks of binary neurons with finite-state synapses in terms of retrieval probabilities of learned patterns under standard asynchronous dynamics with a predetermined threshold. The threshold is set to control the proportion of non-selective neurons that fire. An optimal inhibition level is chosen to stabilize network behavior. For any local learning rule we provide a computationally efficient and highly accurate approximation to the retrieval probability of a pattern as a function of its age. The method is applied to the sequential models (Fusi and Abbott, Nat Neurosci 10:485–493, 2007) and meta-plasticity models (Fusi et al., Neuron 45(4):599–611, 2005; Leibold and Kempter, Cereb Cortex 18:67–77, 2008). We show that as the number of synaptic states increases, the capacity, as defined here, either plateaus or decreases. In the few cases where multi-state models exceed the capacity of binary synapse models the improvement is small.

1. The fraction of non-selective neurons above the threshold is not specified in the criterion since it has been controlled in the threshold selection. Moreover, because of the strong inhibition, there cannot be many non-selective neurons above threshold throughout the dynamics.

2. The amount of allowable non-selective neurons above the threshold is not specified in the criterion since it has been controlled in the threshold selection.

Authors and Affiliations

1. Department of Statistics, University of Chicago, 5734 S University Ave, Chicago, IL, 60637, USA

   Yibi Huang

2. Departments of Statistics and Computer Science, University of Chicago, Chicago, IL, USA

   Yali Amit

Corresponding author

Correspondence to Yibi Huang.

Action Editor: Mark van Rossum

Supported in part by NSF ITR DMS-0706816.

Appendix

1.1 A.1 Kronecker product

If A is an m×n matrix and B is a p×q matrix, then the Kronecker product A ⊗ B is the mp×nq block matrix

The Kronecker product is bilinear and associative

• A ⊗ (B + C) = A ⊗ B + A ⊗ C,

• (A + B) ⊗ C = A ⊗ C + B ⊗ C,

• (kA) ⊗ B = A ⊗ (kB) = k(A ⊗ B),

• (A ⊗ B) ⊗ C = A ⊗ (B ⊗ C),

where A, B and C are matrices and k is a scalar. If A, B, C and D are matrices of such size that one can form the matrix products AC and BD, then

This is called the mixed-product property because it mixes the ordinary matrix product and the Kronecker product.

1.2 A.2 Transition matrices and covariances

As the synaptic modification rule is local, the evolution of the state of a synapse is a Markov chain. Summing over the possible firing states of the pre- and postsynaptic neuron, the probability a synapse transits from α _m to α _m′ is

so the transition matrix can then be written as

Similarly, a pair of synapses with a common postsynaptic neuron \((J^{(p)}_{ij}, J^{(p)}_{ik})\) is also a Markov chain, where the state space is the cross product {α ₁, α ₂,...,α _M }×{α ₁, α ₂,...,α _M }. Again summing over the firing status of the three neurons i, j, k, the transition probability from (α _l ,α _m ) to (α _l′,α _m′) is

so the transition matrix is

Suppose the network is initialized at its stationary state. \(\pi_x^{(0)}=\pi\) and \(\gamma^{(0)}_x=\gamma\). To obtain the mean and variance after the first step of learning (Eq. (6)), given \(\xi^{(1)}_i=x\), since the presynaptic neuron is on \(\xi^{(1)}_j=\xi^{(1)}_k=1\), the transition probability for \(J^{(p)}_{ij}\) from α _m to α _m′ is \(q^{x1}_{m m'}\), and the transition probability for \((J^{(p)}_{ij}, J^{(p)}_{ik})\) from (α _l ,α _m ) to (α _l′,α _m′) is \(q^{x1}_{l l'}q^{x1}_{m m'}\). This explains Eq. (6).

Since \(J^{(p)}_{ij}\) is a indicators variable, \(\mathsf{E}[J^{(p)}_{ij}|\xi^{(1)}_i=x,\xi^{(1)}_j=1]\) and \(\mathsf{E}[J^{(p)}_{ij}\otimes J^{(p)}_{ik}|\xi^{(1)}_i=x,\xi^{(1)}_j=\xi^{(1)}_k=1]\) are exactly the p-step distribution of the Markov chains \(\pi^{(p)}_x\) and \(\gamma^{(p)}_x\), and Eq. (7) comes from the Kolmogorov equations.

Equation (8) is straightforward since \(W^{(p)}_{ij}=J^{(p)}_{ij}\mathbf{w}^T\). One can verify that \(\mathsf{Var}(J^{(p)}_{ij}|\xi^{(1)}_i=x,\xi^{(1)}_j=1)=\mathrm{Diag}(\pi_x^{(p)}) -\pi_x^{(p)}(\pi_x^{(p)})^T\) and deduce Eq. (9), where \(\mathrm{Diag}(\pi_x^{(p)})\) is the diagonal matrix with \(\pi_x^{(p)}\) on the diagonal. To obtain Eq. (10) we use:

where the last equality comes from the mixed-product property of the Kronecker product (See Appendix A.1).

1.3 A.3 Decorrelating the synapses

If the four Q matrices are of the form \(Q^{11}=I_{2m}+q_+D_+\), \(Q^{01}=Q^{10}=I_{2m}+q_-D_-\), \(Q^{00}=I_{2m}+q_0D_+\), where \(q_0=\frac{f^2q_+}{(1-f)^2}\), \(q_-=\frac{\tau f q_+}{1-f}\), then

If π is the stationary distribution of P, i.e. πP = π, then π(D ₊ + τD ₋) = 0, which implies πP ₁ = πP ₀ = π. Hence, γ = π ⊗ π of is the stationary distribution of S since

Thus the stationary synaptic covariance ρ defined in Eq. (14) must be 0.

This learning rule only works for single-level coded stimuli, but not for multi-level coded stimuli.

1.4 A.4 Bias in sparseness measurement

In Rolls and Tovee (1995) and Sato et al. (2007) coding level is defined as

where r _i is the firing rate of the neuron to the ith stimulus in a set of n stimuli. Say the baseline firing rate of the neuron is r, and if the neuron is selective to the stimulus, the firing rate is Lr (L > 1), and assume the true sparseness is f. Then on average ∑  _i r _i /n ≈ fLr + (1 − f)r, \(\sum_i r_i^2/n \approx fL^2r^2 + (1-f)r^2\).

Though a→f as L→ ∞, when f→0, a ≈ 1 − f. The relationship between a and f is not monotone and when L = 30, a is always above 0.1 whatever f is (Fig. 11). If the baseline firing rate is subtracted from each r _i , then L will be larger and make a closer to f.

The sparseness measure a (Eq. (36)) versus the actual coding level f for L = 10 and 30

Full size image

Reprints and permissions