imranq

What do you mean by edge? There are already many apps that use LLMs and diffusion models locally on even a mac M1, which means iPhones and tensor chips aren't far behind

The second point is really important I think - the internet has skewed our perceptions of what is real so much that it warps our own sense of worth. Also the demise of religious institutions have forced people to judge themselves by their contributions to the economy - which is a pretty bad game to play given how many winner-take-all markets there are.

We need to teach students about well-being and when we have too much internet and not enough community.

Check out: https://www.insitro.com/

They have an automated robotics powered research lab

Attention is basically routing, these other routing schemes put a less fine-grained choice for the model, which potentially makes it easier to train

Reading the paper, the connection to compilers is more of an analogy rather than a direct technical link.

The authors propose using an LLM to reframe the task as high level psuedocode, and then reason on that code on the specific details of the task

No compilers were used or compiled - no real code was generated or executed. Its just the idea that a programming language syntax has good structure to process details, and a way to interpret some of the results. Many of the other comments here seem like they didn't read the paper at all and are reacting to the headline

I think 99% of users aren't trying to train their own LLM with their data

Shouldn't this be LoRA training?

GPT-5 will likely only see improvements in deductive reasoning like in math, I think it there are diminishing returns if all GPT-5 is a larger transformer than GPT-4

Dang this is super cool - here is the github link for anyone to see how it works: https://github.com/saharan/works/tree/main/life

Thanks for the summary

This won't work alone for ML. The matrix sizes are huge with billions of parameters. Doing light based calculations for that seems impossible with this technique.

Potentially you could break apart a big matrix into smaller matrices for these chips to process

The methods are incredible, but it seems like the text is ordinary and nothing to write home about

Unless you can figure out AI that is available everywhere, all the time (even in the most remote jungle with no power), there will always be value in making humans smarter

At this point are academic papers simply peer-reviewed blog posts?

I think Gemini Pro is in bard already? So that's what it might be. A few users on reddit also noticed improved Bard responses a few days before this launch

LLMs are comprised of just three elements

Data

Compute

Algorithms

All three are just scratching the surface of what is possible.

Data: What has been scraped off the internet is just <0.001% of human knowledge as most platforms cannot be scraped so easily, are in formats that are not in text like video, audio, or just plain old pieces of paper undigitized. Finally there are probably techniques to increase data through synthetic means, which is purportedly OpenAI's secret sauce to GPT-4's quality.

Compute: While 3nm processes are approaching an atomic limit (0.21nm for Si), there is still room to explore more densely packed transistors or other materials like Gallium Nitride or optical computing. Not only that but there is a lot of room in hardware architecture to allow more parallelism and 3-D stacked transistors.

Algorithms: The transformer and other attention mechanisms have several sub-optimal components to them like how arbitrary the Transformer is in terms of design decisions, and quadratic time complexity for attention. There also seems to be a large space of LLM augmentations like RLHF for instruction following and improvements in factuality and other mechanisms.

And these ideas are just from my own limited experience. So I think its fair to say that LLMs have plenty of room to improve.

Super cool! I like the fluid movement with the robot

100%, if we had real personalized AI companions that actually had an idea of your knowledge state then summarization would be a game changer.

Is the vision model directly reading the screen and therefore also reading the Vimeo tags? It might be more effective to export the DOM tags and the associated elements as a Json object that is fed into chatGPT without using the vision component

One problem with summarization that I think is overlooked is that it relies on the context of the reader. A lot of summarization assume some ambiguous level of context, but it would be way better if you knew exactly where the readers coming from and used that knowledge to perform a summary

Is it possible to know the minimum model size / data set size it takes to train a model given certain efficiency parameters (latency, etc.)?

While SD is pretty interesting, I'm curious what do people use it for? Outside of custom greeting cards and backgrounds, it's not really precise enough for conceptual art nor is it consistent enough for animation.

I think interpretable is a overloaded term. If you use RAG, sure you can look at the retrieved text and understand what is being used, but at that point you're just interpreting the retriever.

LLMs need to be interpreted so that they can be edited have their biases understood in a systematic way. However just as people aren't "interpretable" these algorithms are not going to be able to display their inner workings with 100% confidence. It's going to remain probabilistic, which might be fine for the majority of use cases. I think we're coming from an age where everything was 100% interpretable because we knew what was going on inside the machine (e.g. in a knowledge graph).

There needs to be some definition of what we want to achieve with interpretability for us to understand what standards we need to keep.

From https://www.lockheedmartin.com/en-us/news/features/history/b...

"[The plane] flew coast to coast [in the U.S] in 67 minutes"

Being able to go from NY to California in an hour seems pretty great

Courses that can help people work with genAI are sorely needed. At the same time, hardware/software integration with these technologies is really lacking. There hasn't been an iPhone moment for generative AI yet

I found this tool to be interesting to play with and seems to work pretty well assuming you stay in the same position: https://gazerecorder.com/

I'm guessing a combination of projection mapping, built in lighting, and some crowdsourced data will get accuracy to very usable levels

This is the book that got me into mathematics as a high schooler

Just take a badge reader back with you and badge in at home

Maybe this increased prudishness is coming from the kinds of queries they are seeing come in...

Seems like a lot of confirmation bias on this thread, but I've found google search to be fine for most of the search queries I've used. The real threat seems to be chatGPT which has been a great timesaver for information that needs to be customized

I've skimmed through the Landau books and while they are technically excellent, they lack a sense of feeling and motivation which makes it hard to self study from. It felt like a slog across random topics with no application