Clay Foundation Model: An open source AI model for Earth
clay-foundation.github.io
clay-foundation.github.io
> Clay harnesses AI, satellite images, and other spatial data to organize information about what’s happening in precise locations around the world. We give Clay millions of satellite data and use the latest AI tools so it can supervise itself learning about Earth through those images. As it learns, we benchmark how those skills improve its capacity to do important tasks like creating land cover maps, detecting crops or burn scars, or predicting carbon stock.
The page on "location embeddings" [1] seems interesting:
> In CLAY, the encoder receives unmasked patches, latitude-longitude data, and timestep information. Notably, the last 2 embeddings from the encoder specifically represent the latitude-longitude and timestep embeddings.
Perhaps it could figure out the latitude, longitude, and orientation from a satellite photo (geolocation).
[1] https://clay-foundation.github.io/model/clay-v0-location-emb...
- Their training process is a self-supervised MAE to recover the original satellite images and some additional metadata.
- They're trying to then use this for plastic pollution detection, deforestation, mining, etc.
I'd liken this to a llama-base. It's exposed to a lot of satellite data, but to do something useful really has to be fine-tuned (or clustered in the case of unsupervised embeddings). No word yet on the performance against these specific downstream tasks.
> Clay v0 is a self-supervised modified vision transfer model trained on stacks of Sentinel-2, Sentinel-1 & DEM data. It is trained as a Masked Autoencoder (MAE) to reconstruct the original image from a masked image.
> Each data entry is a stack of 10 bands of Sentinel-2, 2 bands of Sentinel-1 & 1 band of DEM data. The model is trained with 3 timesteps of data for each location, with a total of 1203 MGRS tiles globally distributed, each of size 10km x 10km. The data was collected from the Microsoft Planetary Computer.
> The model was trained on AWS on 4 NVIDIA A10G GPUs for 25 epochs (~14h per epoch) in December 2023.
Also useful: https://clay-foundation.github.io/model/model_embeddings.htm...
> The embeddings file utilizes the following naming convention:
> {MGRS:5}_{MINDATE:8}_{MAXDATE:8}_v{VERSION:3}.gpq
> Example: 27WXN_20200101_20231231_v001.gpq
MGRS is Military Grid Reference System. I believe 5 characters corresponds to about 60 miles by 60 miles.
So presumably the result is an embedding vector representing e.g. a specific spot on earth around a specific 3-year period, such that you can compare that embedding vector with other points to get an idea for what's similar and maybe categorize them against known vectors for things like had-a-lot-of-deforestation?
I'm having trouble visualizing benefits of using ai as opposed to traditional data analysis for these purposes: - Track changes in forest cover - Allocate funding for environmental justice
Pleasant landing page - really enjoyed!
The FAQ claims: "Clay provides key infrastructure like pre-computed raw data analysis, apps, an API, cutting-edge models, and benchmark tools."
... but I cannot find most of these. EDIT: Found something - the Product section does, after enormously patience-taxing scroll jacking, show off a few model capabilities.
(Getting that text required using the inspector, their FAQ page - you know, the page with the information you want to be commonly known - breaks copy and paste.)
They have funding from the Radiant Earth incubator, so they've been able to convince someone of this, but it must have been with better material than this website. Maybe they should share that one :-)