Simulation? Like fluid dynamics. I heard that was CPU intensive.
72 karma · joined September 3, 2018
Attribution is important but it's impossible to know all the literature that is out there and even when you've tried your best to seek out prior arts there is always a possibility that you've missed something or not understood the connection with something previously read. Heck even when something is similar you may not cite it since it's so old the relevance is no longer there for a working research paper.
I've always thought that Schmidhuber was toxically uncharitable - what is the point of accusing and blaming? If someone has a similar idea to mine I'd celebrate them, seek to collaborate and encourage. Wouldn't that mean your research program is getting advanced for free! Heck you should thank them for doing your work for you.
[1] https://aclanthology.org/2020.acl-main.698/ [2] https://www.annualreviews.org/doi/pdf/10.1146/annurev-lingui...
The primary use case is: I am a researcher in nlp where speed of prototyping is key. I work in an environment where research fragments are primarily jupyter notebooks. So needing to diff notebooks is typically reviewing my own changes when modifying my and others research sketches. Since its helpful to see how code changes.
What really resonates with me is what others have said which is I need to run cells that take 2-6 hours to compute so recomputing cells is annoying... I dont love notebooks for their messy state which cause obvious problems that are very annoying.. and I am not an advocate for notebooks for production for this reason but the flexibility of computing stuff and having that persist and doing downstream prototyping makes notebooks amazing! Markdown and latex in there is also really helpful.
The secondary use case is PRs but... typically reviewing others research code isnt at the granular level of notebook riffs across a few commits so it deosnt come up often.
Sales fixes everything and is the hardest thing.
If sales can execute then there is no problem. It seems obvious but it's really not and people make the business about the mission and product - that stuff is secondary seriously secondary.
As a founder and early employee - you need to know if you can execute on sales or not and make it clear to both yourself and your coworkers.
Sales isnt something you can learn to do my biggest lesson is i cant do it no matter how hard I tried. If your employees / boss / cofounder are not closing sales you seriously need to get someone who can as your companies top priority.
Closing means money in the bank and nothing else. Paper is meaningless.
A contradictory learning is the best startups I've been at are ones where folks are ideologically fanatical with mission.
The one in pytorch itself isnt so bad either.
Be nice to see some examples of decoding in your repo (forgive me if I dont see it). I remeber when i first implemented a transformer from scratch generating sequences using greedy or beam search after train/testing turned out to be harder then I thought but turns out I made a mistake with teacher forcing in the beginning so BOS tokens were meaningless to the decoder lol
I typed in d-serine, bromantane, nsi189, modafinil, phenibut with no results...
This is called the compensation hypothesis at least in phonology (how we speak a language)
Pixels per character is certainly an interesting dimension of complexity and I would encourage the author to try to get this ready for publication at SIGMORPHON or something like that!
See https://oxford.universitypressscholarship.com//mobile/view/1... for more on complexity
There is a good reason to reject if you are not using a standard dataset. How can you compare the results of two approaches, to say natural language inference of whether one sentence entails another, without results being tested on the same dataset?
I think one thing overlooked in the conversation is that many papers start with a standard baseline and then use another dataset to establish additional results.
In my experience in nlp also, journals and conferences tend to establish datasets of their own when they make a call for submission. Often these are called the "shared task" track. ACL has operated this way for decades.
Many researchers are interested in the effects of scale on natural language understanding tasks and generalization. One of the surprising results of large language models is their performance on GLUE or the like without needed specific training indicating they are extremely good at generalization. In the NLP community we do want to understand how far we can push this since generalization is an important aspect of modeling how we understand language. The fact is we dont know how much we can push this , and that very fact is very interesting to me and many others. I think many researchers involved in this would acknowledge that this research program may not provide direct and interpretal insight into many language tasks... but that is sort of missing the point.
I am also skeptical that you would have initiatives like DistillBert and other attempts at smaller models that are as effective without the larger ones.
Also the amount of money required for training these large scale models is available to researchers through public research funding. Researchers in other disciplines regularly have million dollar research programs.
I do wish the author of this article had taken the time to present the research program of scaling large models clearer as it reads as a shameless ad for huggingface (use canned models instead of pushing for the limits of research) which doesn't really apply to research in NN design anyway..
The scite extension also works with connected papers so you can see that info there as well.
Disclaimer: I work on scite
I imagine that would be both cheaper and less overhead (in ansible yaml) ect.