Multimodal Neuromorphic Sensory Fusion with Hardware-Aware Spiking Neural Networks for Biomedical Applications

alternate text

Motivation:

Our brain seamlessly integrates diverse sensory stimuli - each varying in scale, timing, and intensity - into a unified understanding. Imagine how effortlessly our brain combines the smell of rain, the sound of thunder, and the feel of wet grass to conclude: it’s about to storm. Taking inspiration from bio-plausible data fusion processes, this project aims to implement sensory fusion with neuromorphic Spiking Neural Networks (SNNs) implemented on real hardware [1]. Spiking Neural Networks enacted on novel neuromorphic substrates offer a promising solution by encoding data from multiple sources into sparse, event-driven spike representations, enabling low-power operation. Unlike traditional machine learning systems, which require significant computational resources and struggle to integrate diverse data, neuromorphic systems excel at handling asynchronous, noisy, and diverse inputs while operating on minimal power [2–4]. But here’s the challenge: how do we teach machines to combine data from diverse sensors –like EEG, ECG, motion sensors, temperature, and sound—when the signals differ drastically in scale, format, and dynamics? How can we ensure these systems extract meaningful patterns, identify correlations across modalities, and enable actionable insights in real-time? This project aims to explore data fusion approaches on the performance of a Spiking Neural Network, implemented in hardware-aware systems with all the physical constraints of real-world operation. The ultimate goal will be to validate and optimize a neuromorphic biomedical signal processing pipeline that balances sparse encoding, and high sensitivity by integrating diverse data sources represented by event streams.

The research will be conducted at the Institute of Neuroinformatics (INI, Y55, Irchel Campus ETH/UZH), in the Neuromorphic Cognitive Systems (NCS) group of Prof. Dr. Giacomo Indiveri. The work will be supervised by Olympia Gallou, Dr. Saptarshi Ghosh, and, Dr. Elisa Donati

 

Objectives:

  1. 1. Data Preprocessing and Spike Encoding

  2. 2. SNN Design for Multimodal Sensor Fusion (Simulator and/or on-chip implementation)

  3. 3. Feature Extraction and Classification

  4. 4. Explore Sensor Fusion for an Anomaly Detection Paradigm (optional)

 

Learning Outcomes:

  • - Learn about signal processing, sensor fusion, and neuromorphic interfaces

  • - Gain experience in SNNs

  • - Learn about neuromorphic HW

  • - Explore challenges in deploying neuromorphic systems for wearable applications

 

Prerequisites:

  • - Interest in signal processing, computational neuroscience, and neuroinformatics.

  • - Basic programming skills (Python strongly preferred).

  • - Familiarity with neural dynamics, dynamical systems, and machine learning concepts.

  • - Motivation to work with real-world noisy data and test neuromorphic systems in practical conditions.

This project can be tailored to both semester and master’s thesis projects, offering a valuable opportunity to contribute to cutting-edge neuromorphic solutions for BCI applications.

 

References

[1] Moradi, S., Qiao, N., Stefanini, F., & Indiveri, G. (2017). A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (DYNAPs), IEEE transactions on biomedical circuits and systems, 12(1), 106-122.

[2] M. Sharifshazileh, K. Burelo, J. Sarnthein, and G. Indiveri, An electronic neuromorphic system for real-time detection of high-frequency oscillations (HFOs) in intracranial EEG, Nature Communications, vol. 12, no. 1, pp. 1–14, 2021.

[3] Ceolini, Enea, et al. Hand-gesture recognition based on EMG and event-based camera sensor fusion: A benchmark in neuromorphic computing. Frontiers in neuroscience 14 (2020): 637.

[4] Gallou, Olympia, et al. Online Epileptic Seizure Detection in Long-term iEEG Recordings Using Mixed-signal Neuromorphic Circuits. IEEE Biomedical Circuits and Systems (BIOCAS), 2024.

 

Contact

Olympia Gallou olympia (at) ini.uzh.ch

Dr. Saptarshi Ghosh, sapta (at) ini.uzh.ch

Dr. Elisa Donati, elisa@ini.uzh.ch

back
back