1,275 karma · joined December 3, 2021
When there is an order of magnitude improvement in hardware, the AI labs will figure out an algorithm to best take advantage of it.
during pre-training, there is never an incentive for the model to say "I don't know" because it would be penalized. the model is incentivized to make an educated guess
large transformer models are really good at approximating their dataset. there is no data on the internet about what LLMs know. and even if there were such data, it would probably become obsolete soon
that being said, maybe a big shift in the architecture could solve this. I hope!
If you were to search for billions of parameters by brute force, you literally could not do it in the lifespan of the universe.
A neural network is differentiable, meaning you can take the derivative of it. You train the parameters by taking finding gradient with respect to each parameter, and going in the opposite direction. Hence the name of the popular algorithm, gradient descent.
yes you can
"You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications"
``` const myButton = button({ onclick: e => console.log(e), textContent: "Click me" }) ```
"You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License."
to be clear, Apache 2.0 requires you to give a copy of the license in a derivative, but does not require you to apply Apache 2.0 to a derivative. the term you are probably looking for is copyleft, which Apache 2.0 definitely does not have
That kind of stuff would be great to train on. As long as the answer says something like "If you abide by x framework, then y"
If instead you only need $1e3 to build it, it doesn't have to make as much money. I could just fund it out of pocket