> Clay harnesses AI, satellite images, and other spatial data to organize information about what’s happening in precise locations around the world. We give Clay millions of satellite data and use the latest AI tools so it can supervise itself learning about Earth through those images. As it learns, we benchmark how those skills improve its capacity to do important tasks like creating land cover maps, detecting crops or burn scars, or predicting carbon stock.
- Their training process is a self-supervised MAE to recover the original satellite images and some additional metadata.
- They're trying to then use this for plastic pollution detection, deforestation, mining, etc.
I'd liken this to a llama-base. It's exposed to a lot of satellite data, but to do something useful really has to be fine-tuned (or clustered in the case of unsupervised embeddings). No word yet on the performance against these specific downstream tasks.
The page on "location embeddings" [1] seems interesting:
> In CLAY, the encoder receives unmasked patches, latitude-longitude data, and timestep information. Notably, the last 2 embeddings from the encoder specifically represent the latitude-longitude and timestep embeddings.
Perhaps it could figure out the latitude, longitude, and orientation from a satellite photo (geolocation).
[1] https://clay-foundation.github.io/model/clay-v0-location-emb...