Apple’s new Transformer-powered predictive text model
jackcook.com
jackcook.com
Whereas GPT2 hallucinated random stuff about the US government. A text prediction model should predict what the user wanted to type, so if you evaluate the models based on that, GPT2 actually performed horribly, since the user showed zero intent in talking about the US.
Or maybe I wanted to express myself in the timeless words of the poets:
"A day, a day of glory! A day that ends our woe! A day that tells of triumph. Against our vanquished foe!"
"Rose is a rose is a rose is a rose. Loveliness extreme. Extra gaiters. Loveliness extreme."
"A-well now, everybody's heard about the bird, everybody's heard about the bird, About the bird, the bird, bird bird bird, Haven't you heard about the bird? Don't you know that the bird's the word?"
The default goal everyone is assuming is spitting out the longest correct sequence possible.
But in reality the mental cost of a wildly wrong prediction is much higher than the mental cost of a slightly wrong one, so what you'd train the model for is sequences of a few words at most being with higher confidence.
Most people can/will tune out slightly wrong words especially as they get a feel for what autocorrect is good and bad at.
If you unleash the full range of tokens GPT 2 can normally output, you'll constantly be blasting out words they didn't expect.
—
The fact your long sequence prediction got better doesn't matter because the UI is autocomplete not "auto-write": they're still expecting to drive, and a smart but noisy copilot is worse than a dumb but lazy one in that case.
I wouldn't be surprised if they trained the model to an effective context window of just a few hundred tokens with that in mind
Too many people are focused on technology for technology’s sake.
"Today is a good day for you to be able to do it more than i just a few weeks to get a new."
For actual input prediction, you probably want the temperature to be zero. But even a model as good as GPT-3 becomes very boring and repetitive with those settings.
Interestingly the Neural Engine version of the model does not take a temperature input, but it does output the raw embeddings in addition to the logits.
If you take an iOS simulator, turn off "Settings > General > Keyboard > Predictive", reboot it and then watch the console logs as you turn that switch back on, you'll see the "kbd" process load the models out of that bundle.
Would also be cool if they trained their own copilot for Xcode, given their obviously enturmes code base in Swift and Obj-C.
phi-1.5 is 1.3 BILLION, this one is a lot smaller at 34 million.
My typing and spelling accuracy is horrendous to the point I use autocorrect on my laptop. For me, it’s an important accessibility feature. I also use dictation for single words.
Anyway, I'm not saying you shouldn't use whatever works best for you but one size doesn't fit all.
It took a few months to get used to it, but I developed typo-correction muscle memory pretty quickly (RIP 3D Touch, it was awesome for text editing). Plus I think that the copyedit re-read that I give to things I write on my phone is helpful above and beyond just catching typos. So be it if I make the occasional typo!
I got a little reprieve with swiping text, but still, autocorrect should be much better.
- kids use one level of slang
- older ages groups using another (and it might mean something quite different)
- communications for professional wouldn't use slang and would use formal messages
- different cultures/regions will use different phrases for the same thing (and some colloquialisms are quite sophisticated like Cockney rhyming slang)
- on twitter when out of space you might then shorten words
The thing that really 'riles me up' / 'get on my wick' / 'annoys me' / 'is immensely frustrating' / 'is an opportunity for a sales feature' [trying to demonstrate the different phrasing depending upon audience] is when the auto correct 'fixes' grammar incorrectly. It needs to be clever enough to realise the correct use of there/their/they're and its/it is/it's.
I'll leave with a relevant poem: https://allpoetry.com/poem/4010351-They-re-There-with-Their-...
That said, as a user, it is nice to see LLMs used in a small, discrete, undeniably useful way like this. No flashy promises, nothing new for me to learn- it’s just autocorrect, but better.
My word is “bureaucrat”. No matter how many times I look it up, I can never remember how to spell it.
I always try some variation of “beurocrat”, find that spell-check is of no use whatsoever, paste it into Google, and get the proper spelling instantly.
For example, here's a function from llama.cpp that applies repetition penalty: https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp...
Here's the one from transformers: https://github.com/huggingface/transformers/blob/0a55d9f7376...
To summarize how they work: you keep some number of previously generated tokens, and once you get logits that you want to sample a new token from, you find the logits for existing tokens and multiply them by a penalty, thus lowering the probability of the corresponding tokens.
https://platform.openai.com/docs/api-reference/completions/c...
See presence_penalty and frequency_penalty.
Sampling techniques is one of important arts of LLMs, you'll can find a lot of papers on them.
In general, smaller are more prone to repetition, but you can get caught in it even with larger models.
Or, if you're okay with a smaller LLM, go here, set temperature to zero and enjoy repetition: https://transformer.huggingface.co/doc/distil-gpt2
That post-processing infrastructure can use all kinds of mechanisms to prevent loops and induce more useful output. With the most basic simply systems simply refusing to select any output token that already in the input, to more complex stochastic process that explore the tree of possible outputs, to find branches whose overall result is improved by choosing less optimal immediate steps.
The vast majority of what make Chat GPT different to simpler GPT-3 models is this complex post-processing phase that allows designers to push and pull on the behaviour of the pre-baked static model underlying the chat interface.
My guess is that they have some kind of state-aware sampler, and they know if they are in the table, etc. Because then you can sample in a much better way. Just like with grammars, but grammar itself is probably not enough.
Sampling and tokenization are IMHO the biggest open problems. We have something which kind of works but it's nowhere close to be perfect.
There was an article/paper showing that GPTs (whole family) quickly get confident in looping if there is anything loop-like in the window. So basically, as soon as it loops once, it will never get out of it.
Note that sometimes loops are desired, like with docstring in the code, always starting before the function definition, and other structural things.
So far I prefer iOS because the predictive text above the keyboard (in iOS 16 too) and in particular the speech to text feature has been so good!
The only egregious error that I have encountered has been confusion around the phrase “into” when I mean “in two”. An understandable mistake.
I don't think you could do any better unless you were from Cupertino. With an Irish accent, my experience has been... okay with Apples text to speech, it's certainly better than others, but I still have to make a conscious effort to enunciate quite differently to my normal speech.
OpenAI's whisper has really impressed me though, and transcribes almost everything perfectly, even if I throw in phrases or words from other languages part way through the conversation.
Anecdotally, I've heard autocorrect is much better on iOS 17. But I won't update until the general release next week.
That’s how autocorrect has worked since day one, it was just using a simpler heuristic-based prediction rather than a machine learning model.
The solution badly needed isn't a complicated black box model when we know a simple one is good enough.
In the last couple of weeks using the beta I’ve found myself fighting it more than ever. After years of typing by tapping at the screen, I’ve switched to swiping the words in because it seems to be more reliable.
I can never press on underlined words that are the last word before a linebreak while typing.
It’s been broken for years. YEARS!
That was when they first changed to a machine learning-based model. I’m hoping this one is better.
And I’d like to add that the editing experience is also dire and getting worse.
Just trying to put the cursor where I want it to fix an error earlier in the sentence in ios17 is a massive pain in the ass.
You tap somewhere and it continually selects entire words - I just want to put the cursor there dammit!
Also, getting the copy/paste menu to show takes a very long time
I recently realised how much of a difference it makes when you type quickly as I got an extra work phone with stock android I have to use occasionally.
There is also a feature of this app I use only occasionally, but when I do its great. It let's you type by moving your finger between letters in one continously motion. When one describes it seems rather weird, but it's really one of the best features of the app. You can write really fast this way, the only difficulty it has is with very uncommon words.
FYI, this has been included in stock iOS keyboard for a while now.
Source: I have one :p
No one credible, least of all a guy like me who had been an enthusiastic amateur for a few years, is saying the stuff hadn’t gotten pretty wild recently.
But unprecedented? Nah.
Same with proper nouns. If I deliberately start a word with a capital letter it’s probably a proper noun and I know how it’s spelled, don’t correct me.
Fix this shit, Apple, it’s the one part of iPhone I really hate.
Is this what progress looks like?
Like, sure, don't support the language, but at least give me a Google Translate link then.
You can also add Czech as a language under Settings -> General -> Language & Region -> Add Language. This will let you switch your keyboard to Czech when typing and enable the language on websites and stuff. You can also switch the whole phone to Czech if you like, just by moving Czech to the top of the list of preferred languages.
Come on Apple, you KNOW you don’t support cycling directions in Ireland - stop getting my hopes up every time I ask for directions. Just only show the button when you’re in a geo that supports the feature.
They managed to fit a full QWERTY keyboard on the screen yet made it completely useless.
I think the Czech government needs to intervene. I believe that's what happened with getting Welsh into Windows too?
I don't think companies like Apple listen to anything but the numbers.
But also, Apple doesn't do much research. MS, however, has always had an active NLP research and support program, if only because of the importance of Word. But Apple depends on the research available from third parties.
Without putting too fine a point on it, I can tell you from direct experience that while it might seem that purely statistical methods are sufficient for predictive text, one of the problems is that you're building those models (whether using neural nets or pure stat models) from a body of text that is fraught with peril and the details matter.
There's really basic stuff like profanity and other offensive constructs (see all the jokes about google suggestion auto-complete where sexist, racist, and otherwise offensive suggestions get tons of attention), then there's the problem of source languages often having a clear shift in acceptable completions for entirely different reasons.
The euphemism treadmill in US English are an obvious and continuing expense, but consider whether USSR-era Czech texts, which are substantial, would be a safe bet in training a completion system.
This isn't to say a human element doesn't have a ton to offer! Just to say that we aren't necessarily engineering a bland world of pablum, either.
E.g. try asking an LLM to name a real, non-famous person. The internet and it's training corpus is full of regular people, but you won't have much luck - they're statistically too uncommon to remember.
Source? A MS exec said that creative and precise are GPT4 but balanced is a not (or not 100%): https://twitter.com/MParakhin/status/1693579775590224097
So, I don't agree that there's insufficient data for it to remember a random person. This was obviously a conscious decision, probably in response to situations like this one last spring: https://www.businessinsider.com/chatgpt-ai-made-up-sexual-ha...
I really look forward to using GPT to help me throw together RFCs, documentation, announcement letters, daily standup write ups and other artifacts like that that prevent me from getting actual work done.
Think of it like spellcheck : the vast majority of the time it produces desired results, but if you really want to type bjPvc9fQ, you certainly can.
If the suggestion doesn't match what you were already planning on saying, you just ignore it.
The human desire to be original and authentic is always going to be stronger.
(It's much less effort to ignore it and keep typing your original thought, than it is to think about it, compare with what you were going to say, decide its version is better, and then accept it.)
1) there is a sort order to them. And we don't know how the 'recommendations' work. What is a recommendation? Why is Google recommending something over something else?
2) it bogs down creativity. You end up not thinking and accepting the suggestion as 'good' enough.
"When can I expect to get $workproduct?"
"Here's when I can commit to getting you $workproduct"
"What's the estimated date for $milestone?"
"Here's the project plan for $initiative"
"I can't make this meeting, can you be sure to record it?"
I welcome any tool that can predict what I want to type and does it for me. I'm not sure if it's my imagination or not but Outlook and Teams seem to have gotten better in this regard. I'll take more of that.
This is why I stopped using copilot autocomplete in my IDE. Once you see a suggestion, you can’t make your brain un-see it.
Authenticity didn't start to become a thing people cared about until about the early 1990s, and it didn't blow up and take over the mainstream culture until the 2000s and 2010s.
Prior to the 1990s, there was a lot of interest in professionalism. People spoke and wrote in overly formal, jargony ways that they perceived as being a marker for competence in some specialized professional domain. Being too honest/authentic would have been seen as unsophisticated and lower class.
Just pointing out that what people strive to emulate can change over time. Once LLM writing becomes commonplace, it will probably become trendy to write in the exact opposite way that an LLM does!
In your lifetime, but…
Another common theme in the philosophy of both Dostoevsky and Descartes is the idea of authenticity. Dostoevsky believed that people must live authentically, in accordance with their true selves, and that this is essential for their happiness and well-being. Descartes, on the other hand, argued that the path to knowledge and certainty begins with the rejection of all previous beliefs and the adoption of a completely new and authentic perspective.¹
And of course, the U.S. counterculture movement of the 1960s was deeply preoccupied with authenticity:
You note three themes underlying the American hippie experience: authenticity, individualism and community. Why did these concepts stand out, and what did they mean in the context of the hippie/counterculture movement? / W.R.: Although hippies often disagreed about beliefs and practices, they shared a desire to be authentic. Members of the counterculture condemned mainstream society for being conformist, rule-driven and uptight. Authenticity meant “doing your own thing.” Because freaks distrusted both society and government, individual decisions were applauded as the most authentic.²
¹ https://vocal.media/geeks/freedom-authenticity-and-the-mind
² https://www.washington.edu/news/2015/08/17/uw-historian-will...
Self-Reliance, by Ralph Waldo Emerson (1841): https://en.wikipedia.org/wiki/Self-Reliance
I think and hope the pendulum will swing back eventually, but my guess is that it’s still got some way to go before that.
I personally don't find them useful for quick / short / informal communication like email, or at least not yet.
I’ll accept the risk of blandness, if it means that written communication finally becomes “easy” for me to participate in.
I'm much more open to using transformers as a better auto-correct, where it's one word at a time and uses the first letter as a filter. Especially on a tiny phone keyboard on the go.
On the other side of that token, the average language abilities of the average American office worker are pretty low so I’m assuming they view this as an enhanced AutoCorrect and they appreciate it because it makes them look less dumb.
I agree with your point though. And to answer your last question, unfortunately I think the answer is “yes”.
On the other side of that token, the average language abilities of the average American office worker are pretty low so I’m assuming they view this as an enhanced AutoCorrect and they appreciate it because it makes them look less dumb.
why?
This problem can be solved without text prediction by building a personal knowledge base with hyperlinking[0] and backlinking[1] for discovery, and transclusion[2] for automating writing the same bland things. How I org in 2023 by Nick Anderson[3] goes over a great workflow for this. The advantage of this is that all of the words that you share are actually words that you wrote, instead of sharing words that Google suggested you share.
[0] https://en.wikipedia.org/wiki/Hyperlink [1] https://en.wikipedia.org/wiki/Backlink [2] https://en.wikipedia.org/wiki/Transclusion [3] https://cmdln.org/2023/03/25/how-i-org-in-2023/
Anyway, I like and use the feature daily at work. So do lots and lots of other people.
Like any other technological advancement, we're freeing ourselves from the burden of wasting mental energy manually doing those commonplace unoriginal tasks which can be easily automated, so there's more bandwidth left for human beings to focus specifically on the things which can't be
I love that they're almost never first to market but they find a way to distil value than others don't. It's the almond milk of technology.
https://www.apple.com/newsroom/2017/09/the-future-is-here-ip...
https://blog.research.google/2021/11/improved-on-device-ml-o...
Apple has just been more methodical to the rest of the ecosystem - essentially waiting to understand use-cases before fully embracing it across their ecosystem from Apple Silicon in Mac with neural and `device=mps`, CoreML, and now more-or-less full force with their ML studio now in Xcode.
Many apps didn't wait for this and Snapchat, as one example, has been successfully taking advantage of Apple Neural since the early early days.
There are many reasons Apple has been the world's most valuable company for over a decade. One certainly is that they have a prescient ability to predict where the market is going, even if they have to use their ever-building position to drag things that way. As demonstrated by the facts and references I provided above, not to mention the iPhone essentially re-defining what a smartphone is way back when Blackberry was the current market leader.
Do you have another source that goes into Apple's implementation?
This thread has been exhausting, I suspect due to the religious war that is Apple/Google. As I've said time and time again - if you're genuinely curious Google is your friend here and it takes seconds to find any number of apps, projects, etc using CoreML in 2017.
As far as I'm concerned, the war has always (since the mid-80s) been Apple versus everybody else. Nobody else makes clearly wrong claims like the comment at the start of this thread. What yanks my chain is that Apple is against user control, and we as developers should be above falling for their marketing that they're first or best when they very clearly aren't.
It's extremely unclear how any of these sources corroborate your claim, particularly the first link. It's why I asked for clarification.
In my top comment I said "Snapchat, as one example". Feel free to search around to find others that were more transparent on implementation details, such as this random weight loss app from 2017:
https://loseitblog.com/2017/11/10/on-device-image-recognitio...
They actually include the following sentence: "While we are still awaiting the official launch of Tensorflow Lite, Apple released CoreML with iOS 11."
And another:
https://dl.acm.org/doi/abs/10.1145/3132787.3132815
If you're genuinely curious there are plenty of references of apps using CoreML in 2017 (again, Google) but at this point I'm pretty confident you're just (lazily) trying to win an argument.
You're confusing when Google started designing its own mobile SOC with when Android devices (including Google's) first started using neural network accelerators, which happened months earlier on Android than on iPhones.
https://www.qualcomm.com/news/onq/2017/01/tensorflow-machine...
As for GP's claim, Android has indeed had what the iPhone is now getting for two years: https://blog.research.google/2021/10/grammar-correction-as-y...
Apple (famously) doesn't announce until new functionality/devices are actually available. When they announce real people in the real world get it within a week if not less.
A press release from Qualcomm that came a few months before Apple actually got it in people's hands only further demonstrates that Apple was working on this long before Qualcomm included it in the chipsets that lag on the market from Android device manufacturers.
> As for GP's claim, Android has indeed had what the iPhone is now getting for two years
You meant to say that Pixel had a variant at that time with Gboard. I'm not going to bother to do the research on Gboard on iOS but I suspect it was also available on iOS with Gboard at the same time or soon thereafter. If not that's on the Gboard team at Google for not supporting neural on iOS and Apple hardware that came far, far sooner.
On a somewhat-negative Apple take they are infamous for being, ummm, "inspired by" successful apps and add-ons in terms of what makes it to iOS. I wouldn't be surprised in the least if Gboard vs built-in iOS keyboard is another one of these cases.
This press release was for a new Tensorflow release for devices that had been shipping for many months before that. It was in people's hands long before Apple even announced anything similar.
> If not that's on the Gboard team at Google for not supporting neural on iOS and Apple hardware that came far, far sooner.
Once again, Apple's hardware came later than Qualcomm's hardware. Pixel phones come with a lot of features that could be implemented on iOS, but why should Google go through the work of porting it only to promote a competitor's inferior platform? For example, Google Maps with navigation shipped years earlier on Android. Grammar check with a transformer model is now on all Android devices in Gboard (for more than a year and a half) and still no iOS devices.
What's interesting about this "debate" is my original links very, very, very clearly show the reality: Google being excited about launching the Pixel 6 (in 2021) with Tensor silicon - a first for them.
All anyone on this thread has done since is refuse to acknowledge how obvious and clear that is while deflecting and throwing things at the wall.
What's clear to me is you're salty over Apple continuously being behind to the point that you misread articles in order to support your fanaticism and refuse to acknowledge it even after you've been shown the evidence.
Edit: I was referring to writing prose where you're making creative decisions rather than code which is closer to a technical document.
In code things are different. Traditional intellisense is usually just filtering a small set of possibilities. For example, auto completion of a reference to a variable.
This is different to code prediction where for example, a variable name will be suggested.
I can see value in code prediciton ala Copilot but personally don't use it.
With programming IDEs when I can I configure them to display suggestions below the text I type, but if there is no such setting I don't find inline autocomplete bothersome at all. (as long as it displays it's prediction in dark shade of grey or another color sufficiently different from the text I typed, also there has to be a special key to accept the suggestion, like tab, no "enter accept")
Take the word 'lie', often mistyped as 'loe'. This is because the o is next to the i. After accidentally typing 'loe', look at what iOS offers to replace it. low, lot, Joe. Soundalikes. This is because iOS autocorrect is skewered towards spoken vocabulary and not commonly mistyped words in english on a typical 1874 Sholes & Glidden qwerty layout. T9 worked because of this distinction. And iOS does not work as consistently because of this distinction.
My 'feeling' is that Apple were very much aware of this from the iOS outset, and not wanting to use T9, offered this poorer auto-correct, and are now riding over the hill shouting 'we've solved it'.
This may not be so for other languages.
Seems to work fine. Though I typically use Gboard because the swipe typing is much better.
I find the most restrictive thing about Apple's autocorrect and speech to text to be the limited vocabulary. Once you start using any industry terms it completely fails.
I reset my keyboard dictionary again on iOS 17, still exact same reproducible bug. It’s been at least 2 years.