iandanforth

Meta: This has to be one of the most aggressively blocked pages I've encountered as it refuses to render any content if you have an adblocker on (uBlock at least) and resists several forms of archiving.

"As a last resort we tried machine learning ..."

- Hilarious that a cutting edge solution (document embedding and search) from 5-6 years ago was their last resort.

- Doubly hilarious that "throw more AI at it" surprised them when it didn't work.

Nope, Gemini Advanced with Deep Research. New mode of operation that does more "thinking" and web searches to answer your question.

Let's say that Google is already 1 generation ahead of nvidia in terms of efficient AI compute. ($1700)

Then let's say that OpenAI brute forced this without any meta-optimization of the hypothesized search component (they just set a compute budget). This is probably low hanging fruit and another 2x in compute reduction. ($850)

Then let's say that OpenAI was pushing really really hard for the numbers and was willing to burn cash and so didn't bother with serious thought around hardware aware distributed inference. This could be more than a 2x decrease in cost like we've seen deliver 10x reductions in cost via better attention mechanisms, but let's go with 2x for now. ($425).

So I think we've got about an 8x reduction in cost sitting there once Google steps up. This is probably 4-6 months of work flat out if they haven't already started down this path, but with what they've got with deep research, maybe it's sooner?

Then if "all" we get is hardware improvements we're down to what 10-14 years?

I built this with twilio, STT, TTS and some glue a while back. Having a phone number to call to chat with GPT-4 was fun, but laggy and error prone. I primarily used it in the car for hands free discussions. I look forward to giving this new option a try!

As it's only tangentially mentioned in the article people should know that in many (most?) places insurers can't raise prices commensurate with risk making "nonrenewal" their only option.

These regulations have reasonable origins because as a mandated product it's pretty tempting to price gouge, but there's no exception for circumstances where the price really should be 3x the historical cost.

Passkeys are a terrible idea. They are security theater and a disaster for users waiting to happen.

Imagine you're on vacation and have lost your phone. You want to go to a cafe and log into a chat app, an email service, whatever to contact your family. In the world that passkey advocates want this is impossible via the passkey flow. If you can't authenticate via a primary device that contains your private key, you're f-ed. Service providers know this so of course they will provide recovery mechanisms. (Not consistent recovery mechanisms of course, each will have their own convoluted and likely to be broken ones). If the recovery mechanism allows for knowledge based recovery (challenge questions) then you're basically telling people that they need N passwords rather than one password. Maybe that challenge question is just a long password (recovery key). Maybe it requires access to another system, which you likely don't have in this circumstance. So you're either back to having passwords, or you're f-ed. Security theater or a disaster.

A service only has value if I can access it. I should be able to sit down at any computer in the world with the knowledge in my head and get access to any online service to which I subscribe.

Boringness classifier! Pretty cool because this implies the large models already know what is useless and what isn't.

I find it interesting how far linguistic, and experienced based approaches have fallen out of fashion. Humans don't read character by character, even if we can it's not a standard operating mode. We have word stems and understand modifications by endings. Tokenization doesn't replicate this experience (seriously, look at the tokens that appear in LLM vocabularies), nor does character or byte encoding. Humans have multiple ways to parse words. You can grok a full sentence, read a phrase, read word by word, or sound out a new word character by character. Very few papers explicitly claim that a method is good because it replicates the way a human would perform a task, or perceive the world.

I suspect as LLM reliance increases we'll want to align the models to our experience more closely. I further suspect this will make the errors that models make more comprehensible.

I like the idea, but it seems like such a cliche how poorly represented native peoples are in this project. If you're a historian and cartographer is there a more obvious failure that everyone knows about? It's like storing your passwords in plaintext for services, so many people have made the error, and its so famous that making this error again seems like deliberate incompetence.

Totally counts though!

TIL a "rolling tray" is a tray designed specifically to aid in rolling joints.

I love the idea! Anyone here from MS/Github who could integrate this into Github sponsors? That way you could "Donate to Open Source" and see the allocation distribution without having to do all this work.

If you go this route consider this extra hack. Caffeine acts as a direct reinforcer. You can buy caffeinated chocolates and dispense those. This will augment the power of the treat.

What's weird about student/amateur rocketry is how, at a certain point, knowledgeable people have to say "I'm not allowed to help you." If you get too good at this hobby you run straight into arms treaties.

This is a frustratingly bad article.

The primary argument is hedonistic. The author is arguing that the state of mind created by reading books is what's valuable, and not the content.

This infuriating for me. This is like writing an article in defense of pistachio ice cream. The author has a sensation they enjoy that they want more people to enjoy. I would have trouble coming up with a more trivializing case for physical books. You might as well just talk about the joy of the smell of old books. It's pleasurable, unique, and completely missing on the internet.

The author fails to connect that pleasurable sensation to anything meaningful and so can be easily dismissed.

Whereas other writers, ones the author quotes even, have pointed out how long form content trains concentration, short and long term memory, and critical thought, this author fails to convince that books are anything more than a warm blanket for the mind.

Re: The photos case: I have the camera app bound to a double click of my power button. I find it to be incredibly useful to have a way to immediately open my camera and Intenty doesn't override that flow.

How does Everbank offer rates above the fed rate? Wealthfront offers good rates but they track the fed rate pretty closely.

Nice! My favorite challenge was avoiding the glowing white dots along the path.

Total defense duty is pretty hard core. 16-70 obligatory service in time of war.

They could spin it off and then set up the same kind of pay-for-default-search deal that Mozilla has. This might put just enough distance between the two orgs to satisfy the DOJ without actually changing much.

A few important things to remember here:

The best engineering minds have been focused on scaling transformer pre and post training for the last three years because they had good reason to believe it would work, and it has up until now.

Progress has been measured against benchmarks which are / were largely solvable with scale.

There is another emerging paradigm which is still small(er) scale but showing remarkable results. That's full multi-modal training with embodied agents (aka robots). 1x, Figure, Physical Intelligence, Tesla are all making rapid progress on functionality which is definitely beyond frontier LLMs because it is distinctly different.

OpenAI/Google/Anthropic are not ignorant of this trend and are also reviving or investing in robots or robot-like research.

So while Orion and Claude 3.5 opus may not be another shocking giant leap forward, that does not mean that there arn't giant shocking leaps forward coming from slightly different directions.

Does this send code via your servers? If so, why? Nothing you've described couldn't be better implemented as a local service.

Could this tool get a command from the LLM which would result in file-loss? How would you prevent that?

The last bit about using this same technique for sensors is pretty cool. Ultra-sensitive microphones or touch sensors would be pretty awesome.

The probabilistic sampling of tokens does not naturally produce introspective evaluation of confidence, it enforces a highest probability token selection (in greedy sampling). The paper that you linked demonstrates that if a separate evaluation phase is allowed then a model can decide with some accuracy whether its previous statements were true. This is not the behavior we want out of a system as it involves 1. Production of potentially misleading output 2. The identification of factual statements within that output 3. The classification of each of those statements and 4. Restatement of the original output without factual errors. The research area that I am advocating for would aim to prevent 1 not mask it.

Not that this invalidates your comment, but why do you put yourself through ads? Block them.

I hope this dataset is used to probe the wrong answers much more than try to get all the answers correct. I don't need LLMs to know everything, but I do need them to know what they don't know. That's not a capability that comes naturally to a probabilistic sampling of tokens.

The ideal 'progress' on this benchmark is for a model to remove incorrect answers and replace them with I don't know answers. Even if it hurts the correct answer count a bit I'd gladly make that tradeoff for a model that hallucinated far less often.

This reminds me of mirror box therapy for phantom limb syndrome. Amputees sometimes feel intense pain in the limbs that they have lost. This is measurable activity in the brain. Unfortunately all of the feedback mechanisms for that activity have been removed with the limb. By showing people a mirrored version of their remaining limb. And then stimulating that limb in a way that would remove pain say massage or touch or unclenching a cramped clenched phantom hand, the patient gets the feedback to the part of their brain via their visual system that there's nothing wrong with that limb. This activates inhibitory circuits that would otherwise be inaccessible.

In this therapy a combination of visual and audio input as well as external control over the behavior of the Voice allows for a feedback mechanism which does not exist otherwise.

First thing I tried and was disappoint.

The key bit I didn't understand at first was what happens if the two groups of attention learn the same thing; because their attention masks are subtracted from one another if they both output similar values the attention across the board will drop to zero and this will lead to high loss. So the only way to reduce loss is if they learn to attend to different things. One of the simplest strategies they could learn (and this paper claims that they do) is for one group to focus on relevant context and the other to focus on irrelevant context. Thus one group learns the noise and the other the signal (it's not this cut and dry but is a useful simplification for understanding IMO).