434 karma · joined September 9, 2020
Interested in generative models, programming, music and philosophy of mind.
https://github.com/angusturner/
But I would not be surprised if this was trainable on a commercial GPU at home within that time. But I think another important trend that we are seeing is that you don't need to train these models from scratch.
Open-source "foundation models" means that you can usually get away with the much easier task of fine-tuning, as to not throw away / re-learn everything that these large models have already fit.
Edit: I initially said 2-5 years, but on more reflection this does seem optimistic (for training from scratch).
As of about a week ago this tech runs on consumer GPUs. The weights have been downloaded 100s of thousands of times, and fine-tuning / modifying is possible.
Training from scratch is about $500k still, but it will only get cheaper and easier.
I think spaces that effectively moderate AI art content will be successful (or not) based on these same factors.
It won't depend on some brittle technology for predicting if something is harmful or NSFW. (Which, incidentally, people will use to optimize/find NSFW content specifically, as they already do with Stable Diffusion).
e.g. the one line of code in Stable Diffusion that predicts if stuff is NSFW, can be inverted to generate only NSFW stuff.
I tend to agree with OP that there is no technical solution to this problem.
I really feel like "reading lots of books" is a better analogy. (Of course its not perfect, since these systems can scale way past the number of books a human can read).
Like, isn’t it kind of wild to think that the average person could have access to models that have learned from all art ever made…
I’m also encouraged by the fact that some of the best versions of these systems are open source and run on a commercial GPU.
Of course there is nothing that necessitates letting these systems do what humans do.
What I am saying is that I am allowed as a human to ingest copyrighted material, learn from it, form mental abstractions and use those learnings to generate new art. Why then, would I be forbidden from externalising some of that process into an artificial system that operates on similar principles.
The trained model obviously does not have agency. It is a creative and computational tool, much like any other digital tool or product.
And I do think “learning” is the correct metaphor here. There are good reasons to believe that the brain is doing something conceptually similar to generative statistical models. i.e compressing inputs (/percepts) into some kind of abstract representation space. Even if we don’t understand the specific details yet.
As far as I’m concerned, it makes no sense to prevent training on copyright data. If I use a trained model to substantially reproduce parts of a copyrighted work, feels like the issue is largely with me and what I do with the content.
These models are just tools after all.
Likewise, if google decides it wants to preserve its service for students and researchers, then I hope that people chewing up resources for malicious/questionable use-cases are booted off.
Would be interested to learn more about this approach.
Tether keeps printing billions of dollars with no oversight or regulation. They lied about it being backed by cash reserves, and now make a much softer claim that its backed by "some cash" and other assets (other assets include more crypto IIRC).
Basically, it feels like tether is inflating the price of BTC, and the exchanges all have a vested interested in continuing to pretend tether is worth something. If there was a run on tether (i.e. everyone tries to exchange for fiat) it could be catastrophic for all of crypto.
(Disclaimer: I am not a financial expert, and have just picked this up from reading about tether over the last few months).
Suddenly the promise of "trust-less" makes no sense, and we are just left with poor performance and immutability of human errors.
There's just too much magic going on, which is the same thing that makes me reluctant to write Vue or Svelte (after reading this)
For the first time in my life I have non-technical friends asking me how to invest in cryptocurrency, and I have to explain to them the real risk that there is a huge speculative bubble right now.
I am seeing the DeFi people reinventing all sorts of synthetics, derivatives and highly leveraged trading instruments in the absence of any regulatory oversight and can't help but wonder when there will be a reality check.
Even more worrying is the prospect that we don't get a price correction soon, lulling more retail investors and institutions to expose themselves to BTC. And when the fallout does happen it will hurt the real economy too (you know, the one that actually produces goods and services with value).
Crazy.
I mean, it does in the sense that local pixels are strongly correlated and a binary tree will captures this. In fact if you add weight-sharing to the K-tree model you can recover 1D convolution with a stride and kernel of 2.
But is this really the right operation for images? Why fixed kernel of 2? I think capsules or some other vector-based operation would make more sense. Perhaps with a learned or dynamic connectivity pattern.