LeanderK

of course you're aiming for zero. But you can't just wish it, there are constraints that arise from our reality. It's all a story of risks and costs, from crossing the street, driving a bicycle to using electricity. You can even die from the flu. In the end dying is unfortunately part of living and in some way unavoidable. A little bit of risks repeated a lot goes a long way.

Calling my comment sophistry is a little bit dramatic I think ;) especially considering you started this argument by twisting my statement about 16 dead kids being fine

public policy should be rational and evidence based

I don't think so. I would have said the same about other statistics. In the end it defies my intuition, which is interesting given that the possibility of getting run over is pretty present in every day life of many children. You have to keep in mind that zero is pretty much impossible because the risk is present all long as we drive, so we should aim for zero but won't get to zero.

> 16 dead kids

of course every preventable death is too much and we should be aiming at zero...but 16 doesn't seem that much? I would have guessed more for sure.

I think common in whole germany, but maybe not ubiquitous. I certainly remember them well and in my imagination they were giant!

Yes, I wouldn't say it's impossible, but that's just a hard limit from the architecture. MCTS would have to terminate after a few iterations. And the number of steps is not that big, just massively parallel.

that would be the optimal rule :) We usually optimise to improve and learn better and better rules, hopefully approximating the optimal rule after some iterations. There's still a gap here, but hopefully it can be closed by improving the models, training algorithms etc.

fair. It's not proven that a solution exists for our models but I don't see much that leads me to believe it's impossible. I know GPT is not reliable but there's also really not much done to improve reliability. Its open research but certainly interesting. Most approaches I know are developed on way smaller datasets, models and usually in a computer vision context.

I think current LLM architectures are limiting the strategies it learned. MCTS requires recursion but GPT is always executing a fixed number of steps. Allowing language models more flexibility by a variable number of steps, for example through recursion of the model, would eliminate this hard bound but they are harder to design and train. We have just been able to train GPT-sized models.

by trying to learning the general rules that to explain the dataset and minimise its loss. That's what machine learning is about, it's not called machine memorising.

Disclaimer: I am not really into robotics.

I think the limiting factors is the interface between ML models and robotics. We can not really train ML models end to end since since to train the interaction the model needs to interact, limiting the data size the model gets trained on. And simulations are not good enough for robust handling of the world. But I think we are getting closer.

this is absolutely not a fundamental limit but simply a hard challenge. Approaches exist and it is an active field of research where we do make progress.

I think the task of predicting the next word can be misunderstood. The better you want to be the more you have to "understand" how the previous words interacted. From the style of writing to the current topic discussed, the task gets increasingly complex if you want to be really, really good. How could the next sentence start? Will the author end the sentence here or keep going? These questions are very complex.

This does not mean that we humans might predict all the time, in fact I would argue that LLMs only predict during training. They generate otherwise. We might also learn by trying to predict. I can imagine babies doing it.

I don't understand the comment at all, it has nothing to do with nepotism or favoritsm....like at all?

The situation would be different if the typesetter would also be grading the work. Also, an apprentice is not one of the "well-connected successful friends" for most people.

You seem to argue that the taking the opportunity is wrong because others do not have it, by definition since it's a personal, not work-related connection. It could not have been accessed by the others since it's not a generally available opportunity. But you seem to forget that in the end it's really not about producing the most equal thesis but the best possible. It's not a standardised test but a first try at creating meaningful insights. Comparing it with others is important for the grade but that's not what it's about. It could also be ungraded. It's about the work itself. Taking the opportunity is right.

> The anecdote reinforces the now dominant view that networking is everything

this is not even a logical conclusion. The thesis is not about being good very good typeset. You have to write it first.

training ML model to code is a very interesting challenge. I am surprised by GPTs ability to code, given that it, as I understood it, has basically no tools at the ready. I am convinced that it is way harder to code without debugging and other interactive features both for a human and for a machine. Keep in mind that GPT could not have learned to simulate the code internally given its fixed runtime.

I think ML models need to learn how to interact with our tools (compiler, debugger etc.) to really be effective at coding. That's hard.

super cool. Past ML progress told us that for most problems for which we manages to get sort-of okay performance, we saw a dramatic subsequent increase in accuracy. The moment we have some idea how to solve it, we usually have quite a few paths forward to increase accuracy. I bet that understanding code will be similar as soon as we focus on it. It's an interesting problem. So far our models are not really coding, which would involve actually running stuff.

EDIT: but GPT4, GPT5 and GPT6 will still get stuff wrong. But so do humans. I don't think the existence of errors is a problem, we just have to get accurate enough to get useful.

I was very worried that a fundamental limitation of current AI-approaches is the massive amount of data. Many interesting applications only have very small datasets. Of course there are still many problems where we have this fundamental problem, for example interpreting sensor data that is very domain specific etc.

But after playing around with ChatGPT I think it really opened up new use-cases because it has some "common sense" through its extensive training. As long as your domain is text it really is quite flexible and was able to perform simple tasks and transforming it into structured output, so you can later consume it and feed it into another system. I think this is really revolutionary, the ability to describe simple tasks as you would describe it to a human coupled with a structure on the output, like a simple json with some defined keys. An interface between unstructured and structured so to speak.

A fun, small example from my life: I live in a smaller city and I am constantly a bit annoyed by the amount of work it takes to keep up what's going on culturally. The small independent-cinema communicates mostly via instagram, the small jazz-venue has a news-letter it sends every week and the art-museum has a website etc. Where's some live music this week? I would really like to have some recommendation ever week what's going on. Just to test it, I pushed the text into ChatGPT and asked it for a json to give me a list with a date, heading and a two sentences describing the event. I could have just connected it to a website and published it there and also let ChatGPT generate me an instagram post every week with the summary. The result was quite good. This was just plain impossible before and would have to be done by hand.

For bigger cities you have various news-sources and magazines covering the culture life, but here in my 80K-city it's obviously not profitable.

I don't know where this comes from because this is literally wrong. It sounds like chomsky dismissing current AI trends because of the mathematical beauty of formal grammars.

First of all, it's a black-box algorithm with pretty universal capabilities when viewed from our current SOTA view. It might appear primitive in a few years, but right now the pure approximation and generalisation capabilities are astounding. So this:

> It sees goat, lion, cabbage, and looks for something that said goat/lion/cabbage

can not be stated as truth without evidence. Same here:

> it's not assigning entities with parameters to each item. It does care about things like sentence structure and what not

Where's your evidence? The enormous parameter space coupled with our so far best performing network structure gives it quite a bit of flexibility. It can memorise things but also derive rules and computation, in order to generalise. We do not just memorise everything, or look up things into the dataset. Of course it learned how to solve things and derive solutions, but the relevant data-points for the puzzle could be {enormous set of logic problems} where it derived general rules that translate to each problem. Generalisation IS NOT trying to find the closest data-point, but finding rules explaining as much data-points, maybe unseen in the test-set, as possible. A fundamental difference.

I am not hyping it without belief, but if we humans can reason then NNs can potentially also. Maybe not GPT-4. Because we do not know how humans do it, so an argument about intrinsic properties is worthless. It's all about capabilities. Reasoning is a functional description as long as you can't tell me exactly how we do it. Maybe wittgenstein could help us: "Whereof one cannot speak, thereof one must be silent". As long as there's no tangible definition of reasoning it's worthless to discuss it.

If we want to talk about fundamental limitations we have to talk about things like ChatGPT-4 not being able to simulate because it's runtime is fundamentally limited by design. It can not recurse. It can only run only a fixed number of steps, that are always the same, until it has to return an answer. So if there's some kind of recursion learned through weights encoding programs intercepted by later layers, the recursion depth is limited.

As I see it, we do not really know much about how GPT does it. The approximations can be very universal so we do not really know what is computed. I take very much issue with people dismissing it as "pattern matching", "being close to the training data", because in order to generalise we try to learn the most general rules and through increasing complexity we learn the most general, simple computations (for some kind of simple and general).

But we have fundamental, mathematical bounds on the LLM. We know that the complexity is at most O(n^2) in token length n, probably closer to O(n). It can not "think" about a problem and recurse into simulating games. It can not simulate. It's an interesting frontier, especially because we have also cool results about the theoretical, universal approximation capabilities of RNNs.

you mean this http://karpathy.github.io/2012/10/22/state-of-computer-visio...? Very funny to revisit. How primitive our tools were in comparison to now is astounding. It feels like the first flight of the Wright Brothers vs a jetliner. Imagenet was the new frontier. Simpler times...

at this stages ranting about assigning probabilities is not reasoning is just dismissive. Mentioning its predictive character doesn't prove anything. We reason and make mistake too, even if I think really hard about a problem I can still make an mistake in my reasoning. And the ever occurring reference to training data just completely ignores generalisation. ChatGPT is not memorising the dataset, we have known this for years with more trivial neural network. Generalisation capabilities of neural network has been the subject of intense study for years. The idea that we are just mapping it to samples occurring in the dataset is just ignoring the entire field of statistical learning.

> $0.12 per 1k completion tokens is high enough that it makes it prohibitively expensive to use the 32k context model.

this is a lot. I bet there's a quite a bit of profit in there

It's been a bit since I've been out of touch with the autonomous driving space. Was there significant progress the last 3 years? What are the current challenges?

what about Bosch or Siemens? They still product things like dishwashers or vacuums.

I sometimes really hate how text wraps in browsers. It really limits beautiful designs. What is the current state...are there javascript libraries that could layout text? Or are the primitives not exposed enough to control the spacing and line breaks?

Every time I use tex I just marvel how well it typesets text and fight with floating figures :)

We are not building things for the technology sake. The user comes first, usability and accessibility is paramount. That's where we work around. History has shown is that it's easier to teach the browser new tricks than to make native easier to use. I want to:

go to some website => solve problem => finish

without extra steps. As a user I usually really don't care, I just am not interested in the problem and I am already spending too much time with stuff I am not interested in.

I can imagine that China also has massive infrastructure and a manufacturing environment built up over the last years that may become increasingly hard to replicate in the US. I bet there some "critical mass" for high-volume manufacturing that's needed, if you don't count subsidies. Even if it's all robots, you still need suppliers etc.

I really have no electrical engineering knowledge, so this question might be naive. Given that we are approaching a hard limit (the size of the atom), I wonder wether we could branch out into (true) 3d? So far everything is 2D and if I think of the brain I picture it as an interconnected 3d-blob, which just seems much more sensible.

you can't be serious, germany is a very innovative country. Like who would label as innovate except the US? For example, it creates a LOT of patents, second behind only the US. It is a leading country for industrial research of various kinds, from the mechanical industry to material science. It is the foreign country with the biggest presence of industrial research for american companies, if you want to make the comparison (a fact I remember, so take it with a grain of salt). As a recent high-tech example one of the two mRNA vaccines for covid was created in germany from BioNTech, and one of the other leading mRNA company besides Moderna is also in germany. Granted, it's not performing well in the number of startups created and the software industry is not as strong, but other indicators are doing very well. Also, basic research in comparison to industrial is very competitive, a lot of papers get produced and cited every year. Germany invests a lot of the GDP back into R&D and is one of the leading countries by this metric.

Germanys economy is just not as consumer driven anymore, as many former industry giants crumbled from the competition form asia, so you don't really interact much with it. But if you look at the metrics instead of personal impressions they have their strengths. You don't really export so much by producing bad, non-innovative products.

yeah, german here. I know schufa exists but right now it's really barely relevant. I don't plan to take on large debt or something (I think that's what they are used for?) and I've never used it so far.

While ChatGPT is a real technological breakthrough because I think we, for the first time ever, crossed a line that made having conversations with AI Chatbots useful, it also was inevitable. Year by year we crawled forwards, improving bit by bit. I think we just underestimated the tipping point, at least I would have never thought something of the level of ChatGPT to be possible. It will only improve from here, competition will increase and AI NLP will become that much more common.