We have powerful preprocessing blocks for images: Strong computer vision capabilities predates LLMs by several years. Image classification, segmentation, object detection, etc. All differential and trainable in same way as LLMs, including jointly.
To the best of my knowledge, no team has shown really high scores by adding in a image preprocessing block?
Funnily enough a lot of the challenges in this particular case is related to PyTorch and CUDA being native code (mostly in C++). Of course combined with the fact that pip is not really adequate as a native/C++ code package manager.
Perhaps if C++ had a decent standardized package manager, the Python package system reuse that? ;p
This might be said in jest. But does everything have to be for world domination? Is the guy not allowed to have actual hobby projects? That go just where he fancies, including potentially nowhere at all really...
That is a hilarious bodymod. And considering that is is possible to get rough brain activity indicators in a non-invasive manner, something like it could actually be made.
This happens to me a lot. Almost once per session now, and not even when things are complicated. The model also thinks it has done the changes. So it seems a UI/state bug, not on the model side.
Happily using gevent for our backend (IoT+ML) since 2019. Was very glad when I saw it is still being well supported by recent SQLAlchemy and pscycopg releases.
Uber market cap makes places it in the top100 in the world, whereas Lyft is around 1/25th of Uber in market cap, and not even top1000. I would consider that a success...
That is basically as much winner-takes-it-all one can realistically get in a global market. Cases where the top is just 5x the runner up would still be very winner oriented.
Hope it stays long enough to build up serious electricity generation, storage and distribution. Cause that has a lot of productive uses, and has historically been underdeveloped (in favor of fossile fuels). Though there will likely be a squeeze before we get there...
It is being played like a winner-takes-it-all right now (it may or may not be such a market). So it is a game of being the one that is left standing, once the others fall off. In this kind of game, speeding more is done as a strategy to increase the chances of other competitors running out of cash or otherwise hitting a wall. Sustainability is the opposite of the goal being pursued... Whether one reaches "AGI" is not considered important either, as long as one can starve out most competitors.
And for the newcomers, the scale needs to be bigger than what the incumbents (Google and Microsoft) have as discretionary spending - which is at least a few billion per year. Because at that rate, those companies can sustain it forever and would be default winners. So I think yearly expenditure is going to be 20B year++
Good reporting would present the key parts of the article up top, but invite you to read the full story if interested.
The lead paragraph and the inverted pyramid are key concepts developed in the field of journalism.
Not less efficient. But the impact on resources usage is still higher.
Of course the impact in terms of positive effects is also higher. So the cost/benefit may also have gone up.
I think that before that will be the PCB-ification, where you can order online and be batched in with others on a standardized process, for a reasonable price. This is actually starting to become possible, for example via TinyTapeout you can get tiny chips taped out for a few hundred USD.
Yeah many have their favorite gripes with $TheOthers, and hoping that the AI will solve them. It won't - not because AI isn't good, but because the people involved will quickly adapt. Likely making new annoying behaviors, and probably many will adapt to find new things to be annoyed at also.
Forced I don't know... But of course the financial incentives are very strong, as in many categories the Chinese brands have remarkable and sometimes astonishing value-for-money.
But for a small business, the cost of these tools might be quite low relative to manpower anyway, so paying 2x might not be a big deal. We got 8 Prusa machines at our local hackerspace, and 10 at previous startup lab I was at.
It would be good if the difficulty of getting patents would go up by a factor of 10. To get less of them in volume, and less bullshit ones. Should also throw out a bunch of the existing bad ones.
The TinyML niche, doing ML on microcontroller grade hardware (usually for sensor data analysis), is close in terms of constraints. Models measuring in kilobytes of RAM/FLASH.
But usually missing the flair and showmanship of demoscene, and the just-because-we-can attitude. So I agree, need more demoscene style!
As the maintainer of an open source library in this space (emlearn), I would be interested in contributing to such an effort.
GDDR6 8Gb spot (DRAMExchange) is now around 2.6 USD, down from 3.5 USD in summer 2023, and 6 USD in summer 2022? Last year has been pretty flat though!
Automation can be useful and valuable (economically) even if not intelligent.
Heck from a big picture view of solving a problem (say to manufacture something), then a solution/process/workflow etc that requires less intelligence may be the preferable one - if such a solution can be found, that is. It can be expected to be cheaper, more robust, repeatable.
This was a very important realization for me. That many times when I felt small cravings, the main thing I needed was not something sweet but water.
Before I would drink milk, lemonade etc. But now I just drink water. Helped to regulate blood sugar better during the day.
GPU compute per dollar has been on a pretty steady curve of around 10x per decade. In ML for computer vision we were also able to make models around 10x as efficient per 10 years.
I think with these two factors combined, mapped to LLMs, we will be able go match the performance of say Sonnet 4 on a 2000 USD workstation well within 10 years from today.
"LLM" as well, because coding agents are already more than just an LLM. There is very useful context management around it, and tool calling, and ability to run tests/programs, etc. Though they are LLM-based systems, they are not LLMs.
Seems like we may get local, open, workstation-grade models that are useful for coding in a few years. By workstation-grade I mean a computer around 2000 USD, and by useful for coding I mean around Sonnet 4 level.
Current cloud based models are fun and useful, but a tool that is / will be so core to the developer experience, I want to be able to run locally.
No, the open source code I make is there to be used freely and for all.
LLMs do not change this for me.
I think that open source libraries (but maybe not applications) may be even more relevant now than ever. More application code will be written, and there is still a need for correct and reliable components.