Neural Decoding

As the name suggests, the task of neural decoding translates the activity of neurons into motor commands, and is performed by Brain Machine Interfaces. Wireless intracortical Brain Machine Interfaces have the potential to combat the mobility issues of traditional intracortical Brain Machine Interfaces, that arise due to bulky wiring. Despite this, wireless IBMIs still suffer due to limited data rates, and as a result, careful consideration has to be put into model design. For this reason, in the context of the Grand Challenge for Neural Decoding, we targeted hybrid spiking neural networks for embedded neural decoding in wireless iBMIs.

Approach

The dataset consists of recordings of motor cortex and somatosensory cortex recordings in primates, alongside cursor movements on a screen. The dataset is available at https://zenodo.org/records/583331.

When analyzing the videos we observed that the movements can be described as having a “target-locking” behaviour, which means that every time a new target appears on the screen, it prompts a rapid goal directed movement towards it.

We thus hypothesized that in order to predict velocities of this type, maybe what one can do is not predict very fine grained continuous adjustments, but rather more coarse, discrete keypoints and then use these and interpolate the values between them. The advantage of this approachis that it simplifies the machine learning problem by reducing the resolution of the target signal.

Of course, by discretizing the signal, some information is necessarily lost during the process. So in order to chose exactly how coarse we segment, we took the original signal and discretized it to various degrees. We then quantifyied the amount of information loss during the interpolation using the R² score (which has values ranging from –inf to +1, with values closer to 1 being of better quality).

With a 4-step discretization, the reconstruction is almost perfect within 2 decimals. Looking at 8 and 16 step discretization, we see that one can achieve up to a 16x data reduction at a cost of 0.045 R². This suggests that the prediction of a few keypoints along the velocity trajectory can be sufficient for accurately predicting cursor movements, while reducing the number of predicted outputs by a factor of 4, 8 or even 16, which is perfectly suited to the limited data rates available on wireless iBMIs.

Network Architecture

In order to detect the velocity keypoints, we had to reduce and aggregate the signal from the original size, down to a lower dimension. We accomplished this by use of Temporal Convolutional Layers with pooling, allowing us to downscale the data and extract temporal information. We then pass each data point through a recurrent layer, followed by a linear one, to finally detect the desired keypoints. We then apply linear interpolation between these to come back to the original input size.

Recurrent Layer

For the recurrent units we tested 3 different types. For the traditional ANN approach, we looked at the Gated Recurrent Unit (GRU). On the SNN side, we went with the Leaky Integrate and Fire (LIF) neuron. And for the hybrid approach, we combined the two to form the spiking Gated Recurrent Unit (sGRU).

As a qualitative observation from the visual representations, it is evident that on the ANN to SNN gradient, the more closer to SNN we get, the higher the observed sparsity.

Visualization of predicted movement

Finally, we can convert the velocity predictions of our models back to positions and visualize the predicted movement of the cursor across time, while also displaying the incoming neuron activity. The blue star displays the original trajectory, the yellow shows the GRU movement, green stands for sGRU, and red visualizes the LIF model.

As smaller velocity errors accumulate, the overall difference in position builds up over time. Despite this, the general direction and magnitude are being predicted fairly well, even by our lowest performing model, the LIF. This suggests there is potential for real-world deployment of such models.

Follow-up Paper

You can access the follow-up paper on this topic here.

Awards

Paper    LinkedIn   

BibTeX