Remote Sensing Foundation Models: How to Adapt AI to Satellite Data

A study published on arXiv as a survey chapter examines how foundation models are applied to Earth observation via satellites and airborne sensors.

What Remote Sensing Foundation Models are

Remote Sensing Foundation Models (RSFMs) are large models trained on archives of satellite and aerial imagery. Unlike traditional computer vision models, which learn from photos of cats and faces, RSFMs look at the planet: cities, fields, forests, clouds, oceans.

Here is an example. Say you want to predict wheat yield in a particular region. Instead of writing a model from scratch for that one task, you take an RSFM already trained on satellite images and adapt it to your task. The model starts working because it already knows how to read satellite data.

Why a general-purpose foundation model does not work

The problem is that satellite data is physically different from ordinary photographs.

• Different sensors, different wavelengths (infrared, red, blue, shortwave range)
• Atmospheric distortion: clouds, haze and aerosols change what the satellite sees
• Low spatial resolution (10-100 meters per pixel, nothing like a smartphone photo)
• Time constraints: a satellite passes over a given spot once every few days, so you have to catch the moment
• Measurement physics: spectral reflectance obeys the laws of physics, not arbitrary patterns in the training data

There is no single universal model that performs equally well across all Earth observation tasks. Benchmarks are contradictory. Model A is better at agricultural prediction, model B is better at classifying building types. That is because each RSFM was built for its own dataset, and transferring knowledge requires fine-tuning.

Examples: when adaptation is critical

Harmful algae prediction. A neural network learns to predict outbreaks of toxic algae in lakes and seas. Spectral characteristics alone are not enough: the model must account for a spectral mask that indicates where in the image to look for blooms. This requires physics-informed masking: the model has to understand not only the patterns but also the physics of why algae appear there.

Choosing monitoring stations. Another task: where should a new weather station for environmental monitoring go? An adapted model trained with reinforcement learning can pick locations that yield the most information about the climate and the state of nature.

What comes next

The next generation of RSFMs should be evaluated not only on standard accuracy metrics (accuracy, F1), but also on:

• Modality-specific adaptation: the model should transfer well to new sensor types and wavelengths
• Physical plausibility: the representations inside the network should match the real physics of satellite measurements, not just patterns in the data

This means that in the future RSFMs will be built with more than machine learning in mind, drawing also on the domain: physics, astronomy and climatology.

What this means

In complex specialized domains (Earth observation, medicine, scientific simulation), foundation models are not a cure-all. A large model trained with billions of parameters is not enough. It needs adaptation to the physics of the task, the specifics of the data and real-world constraints. The next step for AI is not just more parameters, but more physical meaning.