Scoping Clay V2: Governance, License, and Features #369
Description
We’ve been exploring and testing what Clay model v2.0 could be.
This issue is meant to start an open discussion in its direction. The aim is to continue to be as transparent as possible while recognizing that those committing resources will also want to shape the model.
Why v2.0?
Clay v1.5 has shown strong adoption, impact, and services are being built on top of it. However, there are structural gaps we cannot bridge without doing a new version. A v2 effort should only happen if we can guarantee to deliver a step-change, not just incremental improvements. v2 is not about annual releases, nor to drive fundraising —it’s seeking stability, sustainability, and focusing where we can drive the most impact. Specially considering the resources needed to train a model from scratch.
This does not invalidate work done with v1.5, and we should seek how v1.5 embeddings relate to v2, so users can leverage e.g. text to v1.5 via v2.0 if possible.
I also want to be explicit, from day one I only pushed to make a Clay model because we saw a gap big enough to matter. There was so much frontier AI that could be applied to EO. Clay imo still (sadly?) offers something no other transformer model does (open or even closed with free outputs and a clear SLA). IMO Clay v1.5 stands alone being:
- fully open data, open source, open license,
- input agnostic.
Sadly, we do not have a peer-review paper for v1.5 nor I think it should be a priority above making it operationally optimized and used. We are bandwidth constrained and a peer-review process is intensive. We very much welcome support to make it happen.
Pillar 1: Governance
Who decides what is Clay v2.0? Must balance stakeholder needs with available funding and resources.
Complication: some co-founders of Clay are also co-founders of the for-profit Lgnd, which is expected to contribute significant resources from several key people. Lgnd is keen to help increase the shared understanding and maturity of geo-embeddings, and are in a position to do so through Clay in the most open way possible.
Proposal: a more visible Clay Board.
Pillar 2: License
No changes proposed—remain fully open:
- Code: Apache 2.0
- Weights: Apache 2.0
- Docs: CC-BY
Anything else we should address here?
Pillar 3: Features (Technical Direction)
Baseline priorities I can see:
- Clay must remain (and expand as possible) operationally focused and input agnostic (omni EO modal). Add non-wavelength data like DEMs.
- Embeddings-centric. v1.5 evolved from a fine-tune mindset. We've moved on towards an embeddings-centric mindset.
- Time. Clay should be temporally sensitive. Time AND space should be both important. V1.5 approached time in practice almost like "frames on a movie", where users had to embed snapshots.
- Text bridge. Build a bidirectional text modality. Ideally leverage existing models to avoid the cost of training a full text model. Should we do dense time (like AEF?).
Open questions:
- Should we add weather as optional input.
- Should we add non-nadir inputs, like drone images.
- What other recent learnings should be added? (e.g. from diffusion, Tessera, AEF, ... )
- Chips, projections and overlaps are messy. What can we do?
- Should it be a transformer? vs. e.g. Monty approach?
Some implications I can see:
- Since it's an embeddings model, use smallest possible decoders; and consider “Matrioska” vectors and efficient types (int8/int16).
- Maximize context window for attention mechanism; minimize embedding number of dimensions (1024 might be too many).
Next Steps
Gather input from the community on governance, licensing, and features. What resonates or concerns you about the proposed pillars? What’s missing? If you are already a v1.5 user, what would help you? What innovations or gaps should Clay v2 address to be worth the effort?