Tabular Data and Images: Why the Choice of Encoder Matters

Researchers have for the first time systematically evaluated tabular models as encoders in multimodal learning tasks, where a neural network must understand tables and images at the same time. The traditional approach, using a simple MLP for tables, turned out to be suboptimal.

Why is an MLP not enough for tabular data?

Multimodal learning requires a separate encoder for each data type. If a model works with images and tables at once, it needs specialized networks to process each. For images, powerful CNNs or Vision Transformers have long been used; for text, language models. But for tabular data, most researchers simply took a basic MLP.

Tabular data (tables in databases, CSV files, spreadsheets) is a well-known problem area for deep learning. It has been called "the last unconquered castle of machine learning." Better models for working with tables exist, for example specialized architectures based on attention and in-context learning, but they are rarely used as encoders in a multimodal context.

• For the first time, state-of-the-art tabular models have been evaluated as encoders in multimodal tasks
• A simple MLP is the de facto standard in many studies, but far from optimal
• In-Context Learning models are among the best-performing for tabular data

What problem do the best models create?

The main difficulty is that the best tabular models, especially those based on In-Context Learning, need to know the target variable (the label) to process an example correctly. This creates a practical paradox: how do you encode training examples (which have labels during training) and test examples (which have none) in the same way?

In-Context Learning models work by looking at several labeled examples to understand the task. When you use such a model as an encoder for image-table pairs, a mismatch arises. During training the model sees labels; during testing it does not. The study's authors developed methods to solve this problem, adapting several families of In-Context Learning models so that they work consistently in both training and testing.

What does this mean for engineers?

The study highlights that the choice of encoder is often an underrated, critical factor in multimodal learning. Using more powerful tabular models instead of a simple MLP can significantly improve the quality of the whole system.

This matters for real-world applications, where tabular data is often combined with other modalities: customer data in a table plus a product photo in e-commerce, a table of device parameters plus a visualization of its state in industry, a patient table plus medical images in healthcare.

What this means

The results show that tabular data becomes more competitive in multimodal systems when the encoder is chosen well. ML engineers can get better results if they stop cutting corners on the tabular part of the architecture. This is especially relevant in applications where tables and tabular information are critical: fintech, medicine, logistics, finance. The study opens the way to more thoughtful design of multimodal models.