Meta FAIR Releases NeuralSet — Python Package for Connecting Neural Data and AI Models

Meta FAIR has released NeuralSet — a Python package for Neuro-AI that brings together neural data, experimental stimuli, and embeddings from modern models in a single workflow. The project aims to eliminate manual integration between neuroscience tools and the deep learning stack, which has been slowing down large-scale research.

What's the problem

Neuroscience already has strong specialized tools like MNE-Python, Nilearn, EEGLAB, FieldTrip, and fMRIPrep. But much of this stack was built before the deep learning boom and was designed for scenarios where data is loaded entirely into memory and modalities are processed separately. For modern Neuro-AI tasks, this is no longer enough: researchers need to link brain signals not only with each other, but also with text, audio, images, and videos that pass through models from the Hugging Face ecosystem.

As a result, labs often assemble homemade pipelines: cleaning fMRI or EEG separately, computing embeddings for words, frames, or sounds separately, then manually synchronizing everything over time, configuring caching, and rewriting infrastructure for each new experiment. When dealing with public datasets in terabytes and continuous stimuli like speech or video, this approach becomes not just inconvenient but genuinely slows down research.

How NeuralSet works

The key idea of NeuralSet is to separate experiment structure from heavy data extraction. First, the package describes everything that happens as lightweight events with type, start time, duration, and a common time scale. These events are collected into a single Study object based on a pandas DataFrame, so researchers can filter, combine, and reassemble large datasets without loading raw signals into RAM. This approach is compatible with BIDS datasets, which have already become the standard in parts of neuroscience research.

• Supports fMRI, EEG, MEG, iEEG, fNIRS, EMG, and spikes
• Integration with text, audio, images, and video
• Embeddings can be drawn from Hugging Face models, including CLIP, DINOv2, Whisper, Wav2Vec, GPT-2, and LLaMA
• Static representations can be unfolded into time series to synchronize with neural signals

Next come the Extractor components. For neural data, they use proven libraries for their intended purpose: for example, FmriExtractor relies on Nilearn, while MegExtractor and EegExtractor use MNE-Python. For stimuli, the package builds embeddings from modern models and brings them into a unified time format. The output is a standard PyTorch-compatible Dataset and DataLoader that can be immediately connected to model training without rewriting the pipeline for each modality.

Scaling without pain

Meta FAIR emphasizes reproducibility and infrastructure. NeuralSet uses a three-stage extraction scheme: first parameters are validated, then heavy computations are pre-prepared and cached, and during training data is lazily pulled from the cache. This matters for expensive operations like running a large language or multimodal encoder across an entire corpus: once computed representations can be reused in new experiments.

The package also uses Pydantic for strict configuration validation and a backend based on Dask for deterministic caching and computation provenance tracking. If a parameter is set incorrectly, the error surfaces immediately rather than after hours of calculation. The same code can first be run locally on a single subject, then switched to a SLURM cluster with just one setting change.

In the research paper and documentation, the authors specifically emphasize that NeuralSet does not replace MNE-Python or Nilearn, but serves as an orchestration layer between mature neuro tools and PyTorch. In the comparison from the paper, the package proved to be the only solution with full support across all tested categories — from recording devices to infrastructure capabilities.

What this means

NeuralSet is not another model, but an infrastructure layer that could significantly speed up an entire class of Neuro-AI projects. If the package truly simplifies working with multimodal brain data and embeddings from modern models, researchers will have less manual engineering and a better chance to quickly assemble reproducible experiments at scale.