NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction
✨ MICCAI 2026 ✨
Wenhao Gao1 Yifan Wang1 Yijia Ma2 Carl Yang3 Wen Li2 Chenyu You1
1 Stony Brook University 2 University of Texas Health Center at Houston 3 Emory University
Wenhao Gao1 Yifan Wang1 Yijia Ma2 Carl Yang3 Wen Li2 Chenyu You1
1 Stony Brook University 2 University of Texas Health Center at Houston 3 Emory University
Figure 1. EEG and audio are partitioned into patches, projected into a shared latent space, processed by a time-conditioned gated Transformer, and decoded through a velocity-based objective for conditional acoustic transport.
## Installation The released code targets Python 3.10, PyTorch 2.5.1, and CUDA 12.4. ```bash conda env create -f environment.yaml conda activate neurosonic pip install -e . ``` Alternatively, install the pinned Python dependencies directly: ```bash pip install -r requirements.txt pip install -e . ``` ## Data Preparation NeuroSonic currently supports two dataset backends: - `eav_input_images`: loads EAV-style prepared per-subject `sub-XX_eeg.npy` and `sub-XX_audio.npy` files. - `paired_index`: loads paired EEG/audio arrays using an index file with `subject_id`, `eeg_idx`, and `audio_idx` arrays. The datasets used in the paper are available from [CineBrain](https://huggingface.co/datasets/Fudan-fMRI/CineBrain) and [EAV](https://zenodo.org/records/13799131). The prepared directory should contain files such as: ```text prepared_eeg_audio/ ├── sub-01_eeg.npy ├── sub-01_audio.npy ├── sub-01_y.npy ├── ... └── manifest.json ``` ## Training The example below follows the EAV setting used by the released training script. ```bash export EAV_PREPARED_DIR=/path/to/EAV/prepared_eeg_audio export OUTPUT_DIR=./outputs/neurosonic_eav bash scripts/train_eav.sh ``` The command trains `NeuroSonic-L` for 400 epochs with AdamW optimization, cosine scheduling, EMA tracking, and online generation evaluation. ## Evaluation Run generation from a checkpoint: ```bash export EAV_PREPARED_DIR=/path/to/EAV/prepared_eeg_audio export CHECKPOINT_DIR=./outputs/neurosonic_eav export OUTPUT_DIR=./outputs/neurosonic_eav_eval bash scripts/evaluate_eav.sh ``` Evaluation writes paired generated and target files named `*_g.npy` and `*_t.npy`. ## Repository Layout ```text neurosonic/ ├── neurosonic/ │ ├── model.py # time-conditioned gated Transformer │ ├── flow_matching.py # conditional flow matching wrapper and ODE generation │ ├── engine.py # training and generation loops │ └── datasets/ # paired-index and EAV prepared datasets ├── scripts/ │ ├── train_eav.sh │ └── evaluate_eav.sh ├── docs/ │ └── index.html └── assets/ ``` ## Citation If you find NeuroSonic useful in your research, or if you use our pretrained checkpoints or the NeuroSonic dataset, please cite: ```bibtex @inproceedings{gao2026neurosonic, title = {NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction}, author = {Gao, Wenhao and Wang, Yifan and Ma, Yijia and Yang, Carl and Li, Wen and You, Chenyu}, booktitle = {Medical Image Computing and Computer Assisted Intervention (MICCAI)}, year = {2026} } ``` Pretrained NeuroSonic checkpoints are available on [Hugging Face](https://huggingface.co/Y-Research-Group/NeuroSonic), and the NeuroSonic dataset is available at [Hugging Face Datasets](https://huggingface.co/datasets/Y-Research-Group/NeuroSonic). ## Acknowledgements This repository includes `third_party/kadtk` for audio distribution metrics and related audio embedding utilities. Please also follow the license terms of third-party components used by the evaluation stack.