Evaluating Foundation Models Trained with Earth Observation Data

Using and evaluating EOFM¶

In the next portion, we will work through interactive demonstrations of how to use and evaluate EOFM for real domain-specific downstream tasks. Specifically, we will leverage two different EOFMs to generate embeddings for imagery before and after a natural hazard, and determine if these embeddings capture a descriptive pattern. We will also use these EOFMs in a fine tuning capacity to detect the natural hazard. After fine tuning, we’ll calculate the performance scores. At the end, you will have worked through two different methods of leveraging EOFM using two different EOFM base models.

The two models we will work with include:

• Prithvi
• Clay

To begin evaluating the EOFMs, we will start with the Privthi model.

The Prithvi HLS foundation model is an EOFM developed collaboratively by IBM and NASA. It leverages a Vision Transformer (ViT) architecture and is pre-trained on the Harmonized Landsat-Sentinel-2 (HLS) dataset. The model incorporates a Masked Autoencoder (MAE) self-supervised learning approach with a Mean Squared Error (MSE) loss function. Its design is optimized for spatiotemporal analysis, using 3D patch embeddings to encode spatial and temporal features from input data formatted as a sequence of geospatial “videos” (B, C, T, H, W — bands, channels, time, height, width).

The architecture leverages temporal embeddings to model time-series dynamics and spatial embeddings to capture geospatial structure. Spectral inputs to Prithvi include six key spectral bands—Blue, Green, Red, Narrow NIR, SWIR1, and SWIR2—provided as geotiff files in reflectance units. However, it is important to recognize that the use of HLS reflectance data introduces inherent distributional differences compared to native Landsat or Sentinel-2 sources. These discrepancies can stem not only from sensor harmonization techniques but also from inconsistencies across data-serving platforms (e.g., Copernicus Hub vs. Earthdata Search), which have been shown to yield different pixel-level statistics for nominally identical products. Such shifts in input distributions can lead to performance degradation—especially when using frozen encoder weights—and should be carefully considered during training, evaluation, and fine-tuning. Ensuring that model assumptions align with the data provenance is critical for maintaining generalizability and transferability across operational contexts.

It has been fine-tuned for applications such as flood mapping, burn scar detection, and land cover classification. The pre-trained weights and fine-tuning workflows are openly available on platforms like Hugging Face, promoting accessibility and further development by the research community.

Run Prithvi¶

This exercise involves carrying out a complete analysis from beginning to end. The steps include:

1. Selecting a location and time range of interest.
2. Downloading Harmonized Landsat and Sentinel-2 (HLS) imagery for the specified parameters (make sure you have an Earth Data login and .netrc file for this).
3. Loading the model checkpoint.
4. Formatting the data to match the model’s requirements.
5. Running the model on the prepared imagery.
6. Performing PCA to analyze the model’s output (embeddings).
7. Fine-tuning a classifier on top of the model’s embeddings.

Define Location and Date of Interest¶

For this example, we analyze a region in Pakistan that experienced a monsoon flood. We will generate embeddings for the area with the model and analyze any relationships within those.

Retrieve Data from the STAC Catalog¶

Using the specified location and dates, let’s obtain Harmonized Landsat and Sentinel-2 (HLS) imagery with stackstac. We’ll parameterize our search query so as to only retrieve the desired STAC items for analysis.

Create a Bounding Box for the Area of Interest¶

We’ll generate this bounding box in the projection of the dataset to ensure appropriately sized image chips.

Retrieve the imagery data¶

Review the Downloaded Imagery¶

The imagery dataset contains eight pre-flood images (one of which is cloudy) and 14 post-flood images.

Load the Model¶

Now that we have the imagery prepared, the next step is to load the model for analysis.

Format Band Pixel Data for the Model¶

Now we will transform the imagery stack into the format required by the model. Prithvi uses short time series of three images, “triples,” as input. Due to limited data, overlapping dates are used to create 19 triples: the first spans dates 1 to 3, the second covers dates 2 to 4, and so forth.

Execute the Model¶

Now we pass the prepared datacube to the model to generate one embedding vector for each image.

Analyze the Embeddings¶

Now we will run a simple analysis to find any patterns in the model’s understanding of the data. PCA is applied to reduce each embedding to a single value. The embeddings, as you’ll see, appear to be grouped into three categories within PCA space: pre-flood images, cloudy images, and post-flood images.

Embeddings for Each Time Step¶

The model also generates embeddings for individual time steps. These embeddings are extracted for each input triple and visualized using PCA, revealing similar clustering patterns.

Fine-Tune the Model¶

Now we will train a classifier head for detecting flood events on top of the embeddings.

Clay¶

The Clay Foundation model is another open source EOFM. It also utilizes a Vision Transformer (ViT) architecture with a masked autoencoder approach. The encoder of the model consists of several layers, each with multiple attention heads, allowing it to capture complex spatial and temporal relationships in the data. The self-attention mechanism enables it to understand both local and global features within satellite imagery. The decoder is structured similarly but with fewer layers and attention heads, and it is optimized for efficient reconstruction during the pre-training phase. The architecture uses an MLP (Multi-Layer Perceptron) layer for feature extraction.

The model is trained on a large dataset that combines multi-modal geospatial data from Landsat 8 and 9, Sentinel-2 L2A, Sentinel-1 RTC, NAIP and LINZ. A sampling strategy carefully curated to cover a variety of land types was implemented, ensuring diverse geographical and seasonal representation.

The architecture and data inputs make Clay highly flexible.

For more details, you can visit the official Clay Foundation website or the documentation website.

Let’s get the necessary code for Clay.

Load the model¶

We now have the data to analyse, let’s load the model. But first, we need to rename the bands to match what’s expected in the config file.

Prepare Band Metadata for Model Input¶

This step is the most technical so far. It involves processing the imagery stack and converting its data into the format required by the model. This includes normalizing the latitude/longitude and the imagery’s date values.

The Clay model is designed to accept any combination of bands in any order, even from different platforms. However, the model requires metadata about each band, including its central wavelength and normalization parameters. This information allows the model to standardize the data and correctly interpret each band based on its wavelength.

For Sentinel-2, we can extract these details from the model’s metadata file. For other platforms, a custom approach may be needed to provide this information. For simplicity here, we utilize the Sentinel-2 parameters for our HLS imagery.

Convert Band Pixel Data to Model Format¶

This step involves transforming the imagery stack into the format required by the model. It includes normalizing the latitude/longitude and the imagery’s date values.

Combine Metadata and Pixels¶

All required inputs, including metadata and transformed pixel values, are compiled into a single dictionary.

Execute the Model¶

The datacube is passed to the model, which generates one embedding vector per image.

Analyze the Embeddings¶

A straightforward way to analyze the embeddings is by reducing each to a single value using Principal Component Analysis (PCA). This involves fitting a PCA model to the embeddings and performing dimensionality reduction. The resulting PCA space reveals three distinct groups: earlier images, cloudy images, and post-flood images, each occupying a different range within the PCA space.

Fine-Tune the Model¶

Finally, a classifier is again trained on the embeddings to identify flood events.

Change Detection Heatmap (Pre- vs. Post-Flood Distances)¶

We can generate a heatmap to visualize pairwise distances between embeddings from pre-flood and post-flood periods. Each cell’s value represents the distance between a specific pre-flood and post-flood embedding.

We will also compute the average distance. A high average distance suggests that the model identifies significant differences between pre- and post-flood conditions. This could reflect meaningful changes in the landscape.

As for the heatmap, pattern consistency is key to look for. Uniformity in the heatmap (similar distances across all pairs) indicates a consistent model response to changes. Non-uniformity might suggest localized variations. Smaller distances between certain pairs might highlight areas unaffected by the event or areas with similarities across both periods.

Notice that the row impacted by the greatest distance includes the cloudy image at index 8. This makes sense as the cloudy image will bear greater dissimilarity to the other embeddings, both pre- and post-flood.

It might be helpful to look once more at the images chronologically ordered after visualizing the graph above.

Discussion¶

How do I know when EOFMs are the right tool?¶

EOFMs are particularly suitable when:

• You have limited labeled data
• Your task requires generalizable spatial or temporal reasoning
• You want to transfer models across regions or sensors
• You seek to build multi-task pipelines (e.g., classification + segmentation + anomaly detection)

They are less appropriate when:

• You have abundant labels and simple classification tasks
• Computational cost is a constraint
• Model interpretability is the top priority

How does the use of EOFMs compare to the performance of other approaches?¶

Zero-shot¶

This means of use is addressed in the chapter. Zero-shot performance (without task-specific training) gives a baseline for how well the pretrained EOFM has generalized from pretraining to your task domain. This is a strength of Prithvi and CLAY in particular, as they can encode rich geospatial patterns directly from input data.

Fine-tuning¶

Fine-tuning should be benchmarked alongside zero-shot usage in practice when using a specialized dataset, or when training for downstream tasks. Fine-tuning can enable domain-specific adaptation and often improves accuracy, particularly in edge cases or for features not well represented in pretraining. This comes with increased computational cost and overfitting risk if data is limited.

Simpler methods¶

It’s essential to benchmark EOFMs against simpler baselines like:

• Classical remote sensing indices + Random Forest
• SVM
• Spectral-only MLPs
• Convolutional Neural Networks

These comparisons help quantify whether the added complexity of a foundation model is justified for your task.

Other feature reduction techniques¶

Other unsupervised methods like pure PCA, autoencoders, or temporal summaries may be compared to EOFM embeddings when used as input to downstream classifiers. This allows us to understand whether EOFMs truly encode more discriminative or transferable features than conventional feature engineering pipelines.

How does the performance of different models compare relative to each approach’s resource consumption?¶

What kinds of interpretability can be done?¶

• Attention maps (e.g., from ViT-based EOFMs like Prithvi and Clay)
• Similarity search using embedding distances (semantic matching)
• Visualization of embedding spaces (UMAP/t-SNE for EO tasks)

For Prithvi and CLAY, embedding-based analysis (e.g., clustering, retrieval, similarity) is particularly powerful, though attribution methods are still emerging.

How can I use the information above to ensure I choose the right method?¶

Choosing between methods depends on:

• Label availability (EOFMs are strong when labels are scarce)
• Task complexity (fine-grained or abstract tasks benefit more from EOFMs)
• Computation available (EOFMs are heavier; classical methods are leaner)
• Performance delta (if EOFMs outperform simple methods only marginally, simpler methods may be preferable)

How can we stratify train, test, and validation to ensure proper evaluation?¶

• Spatial stratification: sample by region or tile, carefully to avoid spatial leakage
• Temporal stratification: ensure held-out time stamps in test set
• Environmental conditions: vary climate, terrain, and seasonality
• Stratified sampling: ensure class balance in all sets (important for rare class detection)

Avoid random pixel-based splits—these can overestimate performance due to spatial autocorrelation.

How can we evaluate at both the tile and map level to ensure proper continuity and qualitative “reasonability” of the output?¶

• Tile-level metrics: per-tile accuracy, F1, confusion matrices
• Map-level coherence: visually inspect maps for spatial discontinuities, fragmentation, or unreasonable transitions
• Temporal consistency: ensure predictions are temporally smooth where expected (e.g., gradual seasonal change)

Use moving window accuracy checks and expert review of map artifacts. EOFMs can occasionally “hallucinate” features if the embedding has been poorly grounded.

What to do when one’s label set is limited?¶

• Use EOFM embeddings as features: they encode high-level patterns without requiring supervised training
• Leverage weak supervision: use coarsely labeled data
• Few-shot fine-tuning: tune only the last layer with a handful of labels
• Self-training or pseudo-labeling: train on confident predictions
• Domain adaptation: try transferring from well-labeled regions to unlabeled ones.

Conclusion¶

In this notebook, we have demonstrated how to evaluate two EOFMs on a real historical event. Through these examples, the goal was to illustrate how EOFMs can be used to generate embeddings from relevant satellite imagery, observe salient patterns within the data, and subsequently expedite classification of an event by way of fine-tuning. This has been a practical series of tasks one might use to evaluate the efficacy of an EOFM for their unique problem.