Meta FAIR releases NeuralSet, a Python package linking neural data and AI models

Meta FAIR has released NeuralSet, a Python package for Neuro-AI that brings neural data, experimental stimuli and embeddings from modern models into one working pipeline. The project is meant to remove the manual glue between neuroscience tools and the deep learning stack, which has so far slowed large-scale research.

What is the problem

Neuroscience has long had strong specialized tools such as MNE-Python, Nilearn, EEGLAB, FieldTrip and fMRIPrep. But most of that stack predates the deep learning boom and was designed for scenarios where data is loaded into memory in full and each modality is processed separately. That is no longer enough for modern Neuro-AI: researchers need to link brain signals not only to each other but also to text, audio, images and video that pass through models from the Hugging Face ecosystem.

As a result, labs often build homemade pipelines. They clean fMRI or EEG separately, compute embeddings for words, frames or audio separately, then synchronize everything in time by hand, set up caching and rewrite the infrastructure for each new experiment. With public datasets measured in terabytes and continuous stimuli such as speech or video, this approach is not merely inconvenient; it actually slows research down.

How NeuralSet works

The key idea of NeuralSet is to separate the structure of an experiment from heavy data extraction. The package first describes everything that happens as lightweight events, each with a type, a start time, a duration and a shared timeline. These events are assembled into a single Study object built on a pandas DataFrame, so a researcher can filter, combine and rebuild large datasets without loading raw signals into RAM. The approach is also compatible with BIDS datasets, which have become a standard for part of neuroscience research.

• Supports fMRI, EEG, MEG, iEEG, fNIRS, EMG and spikes
• Integrates with text, audio, images and video
• Embeddings can come from Hugging Face models, including CLIP, DINOv2, Whisper, Wav2Vec, GPT-2 and LLaMA
• Static representations can be expanded into time series to synchronize them with neural signals

Extractor components then take over. For neural data, they use established libraries suited to each task: FmriExtractor, for example, relies on Nilearn, while MegExtractor and EegExtractor rely on MNE-Python. For stimuli, the package builds embeddings from modern models and converts them to a common temporal format. The output is a standard PyTorch-compatible Dataset and DataLoader that can plug straight into model training, with no need to rewrite the pipeline for each modality.

Scaling without the pain

Meta FAIR puts particular emphasis on reproducibility and infrastructure. NeuralSet runs extractors in three stages: parameters are validated first, heavy computations are then prepared and cached in advance, and during training the data is lazily pulled from the cache. This matters for expensive operations such as running a large language or multimodal encoder over an entire corpus: representations computed once can be reused in new experiments.

The package also uses Pydantic for strict configuration validation and an exca-based backend for deterministic caching and tracking of computation provenance. If a parameter is set incorrectly, the error surfaces immediately rather than hours into a run. The same code can first run locally on a single subject and then switch to a SLURM cluster with literally one setting.

In the paper and documentation, the authors stress that NeuralSet does not replace MNE-Python or Nilearn but acts as an orchestration layer between mature neuroscience tools and PyTorch. In the paper's comparison, the package was the only solution with full support across all the categories examined, from recording devices to infrastructure capabilities.

What this means

NeuralSet is not another model but an infrastructure layer that could noticeably speed up an entire class of Neuro-AI projects. If the package really does simplify work with multimodal brain data and embeddings from modern models, researchers will have less manual engineering and a better chance of building reproducible experiments at scale faster.

*Meta is designated an extremist organization and banned in Russia.