Before reading the paper's obliquely mentioned in the article: How is this different from embeddings?
After reading the papers: How is this different from embeddings?
After reading the papers: How is this different from embeddings?
Here, they're talking about multiplying, dividing and permuting vectors. Multiplying combines concepts, adding creates a superposition.
They also mention randomly selecting embeddings for the concepts in mind. So my guess is that instead of one-hot encoding the classifier, they instead use random encodings on the output, and are working to give those encodings desirable properties.
I would also hazard a guess that the random vectors they choose are close to zero in most components.
The algebra makes it possible to encode sets, key/value associations and sequences to build a knowledge base, and the dot product provides a similarity measure for querying the base.
IIUC the key is that for large space dimensions, any two random vectors (say with uniform distribution over {-1, +1}^d) are almost guaranteed to be near- orthogonal.
This makes it easy to add new items to the base (by sampling a new random vector and updating the base using algebraic operations with other items), yet the amount of noise introduced by near-orthogonality remains controlled and can be filtered out to keep the algebraic structure working as the base grows.
Honestly it seems a bit too good to be true, I'd be very interested to see what are the tradeoffs in practice.
If anyone can link a favorite review paper on this approach, you'd be doing us a favor.
Edit: one of them does link to a preprint version: https://arxiv.org/abs/2203.04571
> […] our proposed neuro-vector-symbolic architecture (NVSA) [implements] powerful operators on high-dimensional distributed representations that serve as a common language between neural networks and symbolic AI. The efficacy of NVSA is demonstrated by solving the Raven's progressive matrices datasets. Compared to state-of-the-art deep neural network and neuro-symbolic approaches, end-to-end training of NVSA achieves a new record of 87.7% average accuracy in RAVEN, and 88.1% in I-RAVEN datasets. Moreover, compared to the symbolic reasoning within the neuro-symbolic approaches, the probabilistic reasoning of NVSA with less expensive operations on the distributed representations is two orders of magnitude faster.