Getting a Dataset Ready for Modeling
In this article, we'll go from a public, raw dataset to a dataset ready for model development on OlmoEarth. This process is applicable to any type of dataset, public or private. I'm using an open source one so you can follow along at home and try yourself!
Note that while this tutorial walks through a very specific use case, the lessons are applicable for any dataset that doesn't fit OlmoEarth requirements.
Building a Minnesota land cover model
Cropland data: USDA NASS Cropland Data Layer, 2023
This dataset is downloadable directly from the State of Minnesota's Geospatial Commons. My goal is to train a per-pixel classification (segmentation) model that predicts a land cover class for every pixel.
I'm going to use the training data requirements as a guide.
Why this dataset?
- Available for multiple years, so I can train on 2023 data and test on 2025 data
- Negative values already included: I don't need to add any new negative values, my dataset already contains them. Your model will learn that the entire world can only be one of the classes in your training set. So if I train my model on corn, soybeans, and wetlands, it will classify everything including water and urban areas, as one of these three categories. Non-crop classes, such as water and built, mean this pitfall is already covered for me.
Initial challenges that make my dataset not yet uploadable
- Format: My data is a raster stored in a GeoPackage: a grid of pixels, each holding a crop code. Studio only accepts vector data (points or polygons) as CSV or GeoJSON, so I need to sample labeled points from the raster rather than convert the file directly.
- Coordinate system: My data is in UTM zone 15N, not WGS84. The points must be reprojected before I can upload them.
- Label: Each pixel holds a number, such as 1 for corn or 5 for soybeans, rather than a class name. Because the values are numbers, Studio will suggest treating them as measurements, which would train a regression model. I'll replace each code with its crop name from the USDA Cropland Data Layer (CDL) class list and make sure the field is set to Category when I import it.
These are all changes I can make natively in QGIS or another GIS application!
More existential challenges
- Date: My data represents all of 2023, with no specific observation date. Using the growing season as each label's date range helps keep my model from trying to classify corn in a season when it isn't growing.
- Too many classes: My map has 53 different classes, some of which are extremely rare. With very few examples of a class, the model can't learn it, so I will need to merge similar classes. See Merging Classes for how I grouped them and why.