Master or Semester Project: Generative Language Model using Hyperdimensional computing

Background

Large language models based on transformer architectures demonstrate impressive capabilities in generating natural language. However, despite their ability to capture statistical relationships in text, transformer architectures struggle to explicitly represent and reason about syntactic structure. Hyperdimensional computing (HDC), on the other hand, enables the encoding of symbolic rules via specific vector operations, providing a promising approach to representing syntactic structure. This project proposes combining the strengths of these two paradigms by integrating HDC and transformer models for natural language processing.

 

Project description

The goal of the project is to build a generative language model that exhibits explicit, coherent and interpretable syntactic rules while leveraging the impressive performance of transformer models. To achieve this, we will design a hybrid model in which syntactic structures are explicitly represented using HDC vectors and integrated with a transformer model for statistical language modeling.

 

Methodology

You will build on existing HDC models in the literature (see e.g. reference 1) to design an hybrid architecture that integrate the HDC and transformer modules. Experimentation with different approaches will be conducted. The model will be trained on large text corpora, the performance will be analyzed, and the resulting latent representations will be explored.

 

Requirements

Proficiency in Python and knowledge of PyTorch and relevant NLP libraries is essential.

Good knowledge of Machine learning.

 

Contact

- Prof. Timothée Proix: proix@ini.ethz.ch

 

Starting date and duration

This project is currently available as a semester project or master thesis.

 

Related literature

- Hersche et al. (2025). A Neuro-vector-symbolic Architecture for Solving Raven’s Progressive Matrices Nature Machine Intelligence 5,363-375 https://www.nature.com/articles/s42256-023-00630-8

- Calmus et al. (2013). Structured sequence processing and combinatorial binding: neurobiologically and computationally informed hypotheses. Philosophical Transactions of the Royal Society B: Biological Sciences 375(1791) https://royalsocietypublishing.org/doi/10.1098/rstb.2019.0304