Google ScholarEvolving Spatially Embedded Recurrent Spiking Neural Networks for Control Tasks
ICANN 2025 (Best Student Paper Award)
A Vasilache, J Scholz, Y Sandamirskaya, J Becker
Results
The following table compares our spatially embedded models to the best performing models (“expert”) from the Farama-Minari repository, trained with PPO, SAC, and TQC, by analyzing their architectures and weight counts. The “RWN” column indicates the ratio of non-zero weights in the spatial SNN compared to the SOTA model, while “RP” shows the relative performance.
| Environment | SOTA Weights | Spatial Weights | RWN (%) | SOTA Performance | Spatial Performance | RP (%) |
|---|---|---|---|---|---|---|
| Swimmer | 9,408 | 137 | 1.46 | 363.7 ± 1.8 | 361.8 ± 1.7 | 99.48 |
| Hopper | 347,392 | 222 | 0.06 | 4098.2 ± 247.7 | 2706.1 ± 73.7 | 66.03 |
| HalfCheetah | 384,256 | 703 | 0.18 | 17641.8 ± 61.9 | 3670.4 ± 1198.3 | 20.81 |
| Walker2d | 359,680 | 694 | 0.19 | 6956.6 ± 15.9 | 2492.0 ± 247.7 | 35.82 |
| Ant | 475,392 | 1,208 | 0.25 | 5846.3 ± 138.5 | 1346.8 ± 35.1 | 23.04 |
Comparing the performance of spatial embeddings, we observed that the 2D spatial embedding generally achieved the best performance especially in less complex environments, which may represent evidence for the similar 2D sheet-like structures observed in the brain, as cortical sheets.
Demonstrations
| Ant | Cartpole |
 |  |
| HalfCheetah | Hopper |
 |  |
A version of the code in this repository was adapted to evolve a neuromorphic controller for a physical inverted pendulum. The following video demonstrates the resulting SNN controller successfully performing the swing-up and balancing task in a real-world setting:
Full video available on YouTube.
The process involved training a network in a finely hand-tuned simulation of the physical environment. Furthermore, to solve the task, a handcrafted solution of two networks was requried: one for swing-up and one for balancing, with the switching point arbitrarily chosen.
The difficulty in adapting the evolutionary aproach for a real-world scenario, prompted me to start looking into sample-efficient algorithms that can be deployed to train on real-world hardware in a short amount of time, which lead me to Model-based Reinforcement Learning.
Sparsity Analysis
Connection Sparsity: Because of the spatial embedding and the distance dependent connectivity, the evolved Spiking Neural Networks (SNNs) is inherently sparse, having over 95% connection sparsity.
Activation Sparsity: Because of the spiking neurons, the networks also have high temporal sparsity, with activation sparsity ranging from 66% to 80%. This means that, on average, a neuron was inactive for the majority of timesteps.
| Environment | Spikes per Step | Activation Sparsity |
|---|---|---|
| Swimmer | (9.40 ± 0.01) × 102 | 80 ± 0% |
| Hopper | (1.12 ± 0.02) × 103 | 77 ± 0% |
| HalfCheetah | (1.47 ± 0.04) × 103 | 72 ± 1% |
| Walker2d | (1.69 ± 0.06) × 103 | 67 ± 1% |
| Ant | (3.68 ± 0.04) × 103 | 66 ± 0% |
Energy Evaluation
The energy efficiency of the evolved networks was estimated by comparing the SNNs on a simulated neuromorphic platform (Intel’s Loihi) against the SOTA baselines on conventional hardware (x86 CPU and ARM).
Synaptic Operations: Due to their sparse activity, the SNNs required fewer synaptic operations per environment step, up to nearly 50x fewer than the dense SOTA models.
Energy Consumption: This reduction in operations, combined with the efficiency of the Loihi platform, resulted in massive energy savings. The SNN on Loihi consumed up to 500x less energy than a SOTA model on an ARM processor and over 5,000x less energy than one on a conventional CPU.
| Environment | Model | Energy (J) | Platform |
|---|---|---|---|
| Swimmer | SOTA | 8.31 × 10-5 | CPU |
| SOTA | 8.70 × 10-6 | ARM | |
| Spatial SNN | (4.91 ± 0.00) × 10-7 | Loihi | |
| Hopper | SOTA | 3.01 × 10-3 | CPU |
| SOTA | 3.15 × 10-4 | ARM | |
| Spatial SNN | (5.63 ± 0.00) × 10-7 | Loihi | |
| HalfCheetah | SOTA | 3.33 × 10-3 | CPU |
| SOTA | 3.48 × 10-4 | ARM | |
| Spatial SNN | (7.79 ± 0.01) × 10-7 | Loihi | |
| Walker2d | SOTA | 3.12 × 10-3 | CPU |
| SOTA | 3.26 × 10-4 | ARM | |
| Spatial SNN | (8.28 ± 0.01) × 10-7 | Loihi | |
| Ant | SOTA | 4.11 × 10-3 | CPU |
| SOTA | 4.30 × 10-4 | ARM | |
| Spatial SNN | (1.41 ± 0.01) × 10-6 | Loihi |
The results hint at a trade-off between task performance and network sparsity. While the evolved SNNs show a performance gap on more complex tasks (largely due to their constrained size), they achieve this with extreme efficiency.
Methods
Our framework evolves Spiking Neural Networks (SNNs) for continuous control tasks using a genetic algorithm. The core idea is to embed the neurons in a 3D space, where the connection probabilities and strengths are influenced by the distance between them. This encourages the evolution of efficient, spatially organized networks.
The SNNs are composed of Leaky Integrate-and-Fire (LIF) neurons and are trained to control agents in various Gymnasium environments. The network’s connectivity is determined by the Euclidean distance between neurons, promoting local connections.
The evolutionary process starts with a population of randomly initialized networks. In each generation, the networks are evaluated, and the best-performing individuals are selected to produce offspring through crossover and mutation. This process iteratively refines the SNNs, leading to specialized and efficient controllers.
The results from the gradient-based optimization approach demonstrate a clear trade-off between task performance, network size, and training stability. While the evolved networks show competitive performance, particularly on less complex tasks, there is a noticeable gap on more challenging environments. This is largely attributed to the constrained network size of 128 neurons and the use of a simple two-layer architecture. Future work could explore larger and more complex network structures to close this performance gap. The key advantage of this method lies in its ability to produce extremely sparse and energy-efficient solutions, making it a promising direction for resource-constrained applications.
Future Work
A key limitation of the current study is the computational cost of the framework. Simulating larger networks, which preliminary experiments suggest would improve performance, is currently infeasible due to the implementation’s inefficiency. For this reason we are working on a JAX implementation of the current PyTorch version to allow for faster experimentation and testing of the approach.
Awards
