X-Transformers: A fully-featured transformer with experimental features
github.com
github.com
For the non-DL crowd, Transformers are a tsunami in deep learning for the past few years. They are topping benchmarks in many subfields. I do research professionally and this work is amazingly useful for people like me.
You may have seen the recent splash of MLP-Mixer for image classification using perceptrons (https://arxiv.org/abs/2105.01601#google) but FC-heavy or FC-only nets have popped up here and there, never quite going away, and slowly getting better initializations & residual layers making them ever more trainable. I have a bibliography of some links I've noted over the years about them: https://www.gwern.net/notes/FC
I've wondered if MLPs will be another example of the Bitter Lesson - the additional flexibility and power was their undoing until data/compute caught up and 'grad student descent' figured out how to use them right... If in the next 5 years everyone gets on the "MLP Is All You Need" train, I will be only mildly surprised.
https://github.com/MrMEEE/ironhide
Optimus Support for Linux Through VirtualGL - PPA version also available
my branch of the original bumblebee project..
Likewise for this project, all you need in the README is one line that's like:
X-Transformers is a re-implementation of Machine Learning Transformers that has been built based on experimental Arxiv papers
Would you say the same about an experimental programming language based on a bunch of recent PLT/CS research?
That's pretty much what this is, but for machine learning. It's not meant to be "for the public", it's effectively research.
A researcher releasing work publicly on github is presumably doing so too spread the ideas.
Sharing a link on Hacker News is effectively taking it out of context and sometimes it's up to us to add that context back in. The author doesn't owe it to us.
There are like 6 different definitions of what a "transformer" is and none of them fall under "machine learning". If you have a README, then that implies its intended for others to read, and so it should be made with that in mind. If I happened to be interested in machine learning, this would be a project that I would bookmark for further use -- except I wouldn't know to bookmark it because they didn't bother to include a total of two words ("machine learning") that give context of how to parse the word "transformer".
The fact that the owner didn't provide that context is not excused by the fact that the repository owner decided to implement a handful of research papers. It absolutely is the responsibility of the authors of projects, and the authors of README files, to provide context about what something does. The entire point of the README is to clear up ambiguity about what something is. You might as well say, why not include a README? People who are using the project know well enough how to run it!
Paid products, public releases, etc. are different and are treated differently.
From me? Or are you lumping me in with all the "nay-sayers", despite the fact that I'm trying to offer criticism of a problem that's pretty common in computer science and outside it?
There is always context and sometimes you don’t have it. Do you expect everything to be written in English for your benefit?
But sure, it would be nice to clarify things for random passers-by so they know it’s not the droid they’re looking for. It’s just not required to do this in every Github repo.
I'm not even sure what you mean to say here. Is asking that the README contain a single two letter phrase reference to the subject matter, the same as expecting the entirety of the world to be written in english?
Quite literally all I've stated is that if you're making something that's clearly intended to be public-facing, it's worth taking the exact ½ of a second to allow people who can't read your mind to understand what context they should read something in?
You seem not to understand that people have different comprehension levels and the one you’re in isn’t necessary the most important one to the author.
I disagree, it would be enough for a non-technical audience, since it would give enough information for them to do *research* and find out more information.
I personally disagree that volunteers owe you anything at all, much less a README, and that we should be thankful for all they do to help as is.
But for real, go post an issue! not trying to play devil's advocate, honestly! Your voice is not heard when complaining anywhere but the repository itself this is open source software and everyone should have a voice!
Some of lucidrains other projects include the line "This is an implementation of thing with link" which I feel you'd prefer.
Like others, I too encourage you to open an issue to do the same here instead of leaving it till "after the fold".
Make a PR with this in the README.md. You still should - in the worst case it's rejected, but it starts a dialogue with the maintainer that can be referenced later unlike this comment which will be likely be buried and tough for others to find in the future.
X-Transformers is a re-implementation of Machine Learning Transformers that has been built based on experimental Arxiv papers
A concise but fully-featured transformer, complete with a set of promising experimental features from various papers.
About X-Transformers, it is a very great piece of engineering that implemented almost of all possible improvements in transformers. But according to my experience and Phil himself, only the Feedforward GLU and RoPe (Rotary Positional Embeddings) works (or to be fair, they show improvements in more general use-cases)
def f(x):
for _ in range(3):
x = g(Wx + b)
return x
Transformers are a modification to that architecture - using different multiplications, additions, and non-linearities. both of these are general in the sense that if you have enough of them, they can approximate any function. The ones used for transformers empirically do well on a lot of machine learning problems, particularly where data has a sequential nature.