CoRe: Community Recurrent SNNs

the walk, spike by spike (tl;dr and abstract)

Vikram Ramavarapu, Himanshu Udupi, Ashish Gautam

Spiking network

Hover over a neuron to see its own synapses; click to pin it.

Walk steps
Between modules
Stimulus-caused
Background

Environment

Tl;dr

Brains are modular and recurrent, but most spiking neural networks for reinforcement learning are stacked feed-forward layers. We introduce Community Recurrent SNNs (CoRe SNNs), recurrent networks split into communities with fixed roles, where the action is the community that an activity walk reaches. We also introduce Map-STDP, a local, reward-modulated learning rule that shapes these communities during training by compressing the network's own activity. On CartPole, learning the modular structure beats the same network without it, and swapping CoRe into a TD-STDP actor-critic improves CartPole and LunarLander while matching Acrobot, with more biologically plausible and potentially more energy-efficient circuits.

Abstract

Biological neural circuits are both modular and recurrent, with dense communities sparsely linked to one another. Spiking neural networks (SNNs) for reinforcement learning are usually feed-forward, and it is unclear how to obtain functional modules in a recurrent SNN or what they buy. In this work, we introduce a class of spiking networks, called Community Recurrent SNNs (CoRe SNNs): recurrent networks partitioned into communities with fixed roles. A controller community receives the stimulus, each action has its own community, and latent communities carry internal state. Spiking activity is then a walk between communities, and the action is the community the walk reaches. All learning is local and three-factor, and the recurrent communities are closer to cortical microcircuits than a feed-forward actor. We compare three regimes for obtaining the communities: dynamic modularity, in which a new learning rule we introduce, Map-STDP, encourages modular structure during learning; partial modularity, in which a stochastic block model (SBM) initialization enforces it partly; and full modularity, in which sealing enforces it fully. On CartPole, across three spiking neuron models, Map-STDP improves learning over the same network without it (+65 to +87 mean return over training), matches full modularity (sealed communities) on two of the three models, and makes the observed spike walk more compressible. Replacing the feed-forward actor of a TD-STDP actor-critic with recurrent action communities improves on the feed-forward network on CartPole (+25 to +45) and LunarLander (+36 to +41 final return), and matches it on Acrobot. How strongly to enforce modularity depends on the task: full modularity helps most on Acrobot, while on LunarLander the dynamic, partial and full regimes perform alike. Overall, these recurrent modular networks can match or even improve upon feed-forward approaches, with improvements in bio-plausibility and energy efficiency.