Evaluating Machine Learning Models in Remote Sensing for Sustainable Development Goals

Lay Summary

In a world facing climate change, biodiversity loss, and urbanization, global efforts like the Sustainable Development Goals (SDGs) are crucial. Monitoring progress toward these goals is challenging, and to achieve accurate data on a large scale, we must look to innovative technology. Machine learning models combined with remote sensing provide a powerful way to track environmental and societal changes from space.

This post walks through a meta-analysis (a study that combines results from multiple studies) I conducted for my masters thesis, which focused on understanding how well machine learning models perform when applied to remote sensing data for the SDGs and which study features affect the performance. The Active Analysis Dashboard also gives you the opportunity to add data (see contribute data page). The analysis automatically updated using this data.

Key Findings from the Study

Machine Learning Model Accuracy: On average, machine learning models in this field deliver an impressive 90% accuracy in classifying and predicting outcomes, such as land use, crop types, and forest cover, as seen across 20 studies and 86 results. It is important to note that this is a small sample size and only looked at results from published papers — this accuracy would almost certainly be lower if I also included unpublished work.

Figure 1: Reported overall accuracy colour-coded by SDG goal. Individual outcomes shown as points and mean overall accuracy represented by triangles.

Figure 2: Map of researched locations

Selecting features

One key aspect of my analysis was multi-model inference, a technique used to compare multiple possible models to identify which study features most influence machine learning performance. In this study a total of 31,298 models were fit. By evaluating these combinations, the method determines which features consistently appear in the top-performing models, highlighting their importance.

Figure 3: Model-averaged predictor importance plot

Factors That Influence Performance: the majority class proportion i.e. the class imbalance in data (where one type of data is far more frequent then another) and the use of ancillary data (additional non-remote sensing data like information on ground temperature), were found to significantly affect model performance. Interestingly, the study found no major difference in performance between different types of machine learning models such as neural networks and tree-based models.

Figure 4: Bubble plot showing the observed effect size, overall accuracy of the individual studies plotted against a the proportion of the majority class. The color of the points is unique to each study, with the lowest overall actuary from each study labeled with the first author and publication year.

It is not surprising that the overall accuracy is high when the majority class proportion is large because the model can perform well simply by predicting the majority class most of the time. Which leads to inflated accuracy values that do not necessarily reflect good performance across all classes.

Observed vs. Predicted Accuracy: When using a model that include majority class proportion, ancillary data, and the use of remote sensing indices (the best fit model I have) the predicted accuracies tend to be overestimated.

Figure 5: ADD

Assumption of Normality

The normality assumption refers to the expectation that the distribution of the overall accuracy (effect size) in the studies follows a normal (bell-shaped) distribution. This assumption is important for the validity of certain statistical models.

In practice, the observed accuracy values tend to be skewed. To address this skewness and make the data more suitable for analysis, transformation techniques are applied. In my analysis, I used the Freeman-Tukey (FT) transformation. These transformations help stabilize the variance and make the distribution more symmetrical. The FT transformation, in particular, performed better in this case, although normality was still not perfectly achieved. In the following plot you can see the distributions of the effect size and their corresponding variance.

// Initialize the data
df = transpose(my_data_ojs);

viewof plotType = Inputs.select(["Effect Size", "Variance"], {label: "Select Parameter"});

// Dropdown to select the transformation
viewof transformation = Inputs.select([
  "Raw Proportion", 
  "Freeman-Tukey double arcsine transformed", 
  "Log transformed", 
  "Logit transformed", 
  "Arcsine square root transformed"
], {label: "Select Transformation"});




// Create a reactive dataset based on the selected transformation and plot type
transformedData = df.map(row => {
  if (plotType === "Variance") {
    if (transformation === "Log transformed") {
      return { ...row, value: row.PLN_vi };
    } else if (transformation === "Freeman-Tukey double arcsine transformed") {
      return { ...row, value: row.PFT_vi };
    } else if (transformation === "Logit transformed") {
      return { ...row, value: row.PLO_vi };
    } else if (transformation === "Arcsine square root transformed") {
      return { ...row, value: row.PAS_vi };
    } else {
      return { ...row, value: row.raw_vi }; // Original data (variance is not applicable here)
    }
  } else {
    // Plot effect sizes (default behavior)
    if (transformation === "Log transformed") {
      return { ...row, value: row.PLN_yi };
    } else if (transformation === "Freeman-Tukey double arcsine transformed") {
      return { ...row, value: row.PFT_yi };
    } else if (transformation === "Logit transformed") {
      return { ...row, value: row.PLO_yi };
    } else if (transformation === "Arcsine square root transformed") {
      return { ...row, value: row.PAS_yi };
    } else {
      return { ...row, value: row.OA_reported }; // Original data
    }
  }
});

// Plot based on the selected transformation and plot type
Plot.plot({
  round: true,
  color: {legend: true},
  marks: [
    Plot.rectY(transformedData, Plot.binX({y2: "count"}, {x: "value"})),
    Plot.ruleY([0])
  ], 
 // y: {
   // domain: [0, 45]  // Adjust y-axis based on your data
//  }, 
  x: {
    label: plotType === "Variance" ? "Variance" : "Effect Size"  // Label changes based on plot type
  }
});