853 karma · joined October 16, 2015
A few months ago I went down this same rabbit hole trying to determine if using Chinese has any inherent advantages due to the way more context is encoded into Chinese characters versus what you find in English.
The total token size may not take into account any benefits gained by Chinese characters potentially encoding more context.
I have been meaning to see if any Chinese native tokenizers have been created to account for this feature of the language.
Unfortunately, many Chinese and Japanese characters can have many different meanings depending on how they are used. As a result they do not break down as well to be tokenized as a language like English.
Maybe some kind of hybrid intermediate language would be most optimal.
The U.S. suffers from a geopolitical bubble which operates like a financial bubble. Inside the bubble, global nations rebuilding and trading without the U.S. reserve currency is perceived as war. When it reality it is just the natural course of countries and economies after WW2.
That question should be answered. Did human review of this target reveal it was a school? If so, did someone in the Govt approve the strike knowing this?
Was it too tempting of a target given who was likely in the same place at the same time (dropping off or picking up their kids from school) ? Did they use phone location data and since parents buy phones for their kids, it looked like a lot of IRGC and other state officials were at that location, when infact it was their children and teachers?
Even more disturbing, if they knew it was a school and still chose to strike, was the intent to wipe out the next generation of Iranian leadership?
Just so many unanswered questions with no good answers.
Of course it was a war crime.
Why would that be encouraging if the purpose of their chosen profession is to advance physics? This is sort of like mathematics being solved by AI with human researchers complaining that they are losing potential awards. It's an entirely self serving way to be.
That is not worth the investments being made.
Kind of an interesting rabbit hole with this one...
With other countries and open source, not to mention all the other companies, Open AI choosing to pause development wont change the course. Only OpenAI will lose.
Unless the secret is that no amount of compute or data can progress things any further because we have already reached the limit for what predicting the next token can do. Most of the advancements lately have been on the agentic side of things.
The massive amount of health, science and math problems being solved by AI has largely not materialized yet. Is this a sign the model capability has been over sold?
There are economic consequences to this as well because the data center build out has been a kind of circular financing. This has caused all kinds of macro economic impacts from increased prices to even the Treasury market itself having to compete with debt issued for the build out. No one knows what will happen once the data center build out stops, or if the yen carry trade continues unwinding, or if Treasury yields have to keep rising to compete with debt issued by the "Mag 7".
Once they stop spending in the current circular arrangement, then the profits that exist on paper disappear, and the bubble could pop.
So really for every reason, OpenAI cant really slow development. Unless there is a kind of Moores law for what predicting the next token can achieve.
At some point they will begin running out of human created data for model training and schools with strict AI policies will be the best source of non AI data going forward.
Thats kind of a symbiotic relationship that makes sense for students and AI companies.
Once AI output begins replacing human output and the models begin training on it, there may be increasingly few places left for data that isnt likely to lead to a model collapse.
AI as a core part of the OS that can just change or add anything you want. Linux is great for this because it has access to the source code for everything.
He said he didnt review the code line by line, just looked at the shape of it. Whatever that means.
And so the circuit courts coming to very different conclusions from the 9th circuit represents the inherent political corruption involved in the court system.
Circuit splits get resolved at the Supreme Court and I would make a prediction that it isnt a coin flip, probably more of a safe bet.
It is a very interesting concept and the company Q.Ant has delivered working products and software. Apparently it can greatly speed up certain training applications and future designs will solve some of the current problems.
The energy and performance advantages seem to be worth the R&D. If some form of optical memory to replace DRAM in the chain could be designed, that would solve several conversion costs between optical to digital.
Either way it is pretty cool to do math this way, basically just using light.
Im sure there are many more things that add up to focus more on throughout over latency as mentioned above.
It does mention UInt64 which is not a modern type and as far as I know is supported by every database.
It also compiles to SQL but it isnt clear where the advantage comes from other than using a different syntax to do things.
Then set up a pipeline between the human generated content from the colleges back into future AI models. Everyone gets an education and there is ongoing non AI training data to advance AI without any model collapse.
There are so many plumbing issues with the financial system happening at once that people really do not have a clear picture of what is happening in totality.
Start with Japan... Still the largest foreign creditor. However they are facing a situation where the yen carry trade that began after the plaza accords in the 1980s is unwinding. The recent Treasury intervention and the associated expansion of the repo market so they do not sell treasuries directly reveals a hidden structural weakness. This leads to the AI boom because cheap Japanese debt was recycled into larger yields in the U.S. And it's different this time because companies like Microsoft have what is rated as safer debt than the US Treasury itself. The yield on financing part of the AI boom has been much better than yields back home in Japan. So even Japanese institutional investors may start seeing domestic debt as more attractive as the carry trade unwinds and domestic Japanese debt starts paying more. This ties back into the recent treasury intervention and offering a way to unload U.S. debt without impacting the markets directly.
Which brings us to an ongoing structural change that will remove the largest foreign investor in U.S. debt. Which as we all know, is approaching $40 trillion with over $30 trillion of that being held as public debt. Meanwhile, the budget deficit is continuing to rise causing even more issuance of debt. After the 2008 crash, and over the next decade, the U.S. was able to issue debt basically for free and it issued a ton of it. But that debt matures and when it matures, it has to largely be reissued at current rates. There was no free lunch. With the Japanese largely pulling back and at best not selling the debt they have now to defend their currency or to chase domestic yields in Japan, that goes to the UK as the second largest foreign investor. And the UK clearly can't absorb what the US has to issue and refinance. China has been reducing its holdings because as the BRICS bank, etc, replace their need for dollars, they will begin using it in more creative ways.
Some speculation is that the U.S. intervention into the yen carry trade is being challenged by foreign actors, which did not happen during past interventions. Monitoring the daily spread, and if the yen continues climbing over 160 then this will be the first time the U.S. intervention failed to work. To prevent selling existing Treasuries to defend the currency, the Japanese investors will unload it in the repo market.
But this brings us to a new structural problem in response to the insane amount of debt being issued and refinanced and this is called the weighted average maturity. Recently, the U.S. Treasury implied that they are going to begin moving more issuance to shorter term debt because those markets are currently still much more liquid than the longer term markets that had issues revealed by the failure of Silicon Valley Bank. When SVB needed to cover deposits, they found the 20 and 30 year Treasury market was not as liquid as they had hoped. This same fact likely pushed the Treasury to have Japanese investors move towards the repo market instead.
But this brings a new problem, because there is only so much appetite for increasingly taking long term debt and reissuance and moving it towards the front of the weighted average. Currently around 6 years, and the goal is to move it between 2 and 4. Imagine the liquidity impact on moving large amounts of new debt issued and refinanced debt from the 30yr to more short term funding.
Next up, there is a need to have the Federal Reserve unwind its balance sheet even more quickly at the same time all of the above is happening. But it is the Feds balance sheet and forms of QE that have propped up the repo market itself under times of stress. So after Japan, the UK and China all have reasons to continue cutting back, and after the short term funding gets eaten up, the yields will have to rise dramatically to continue attracting investors for even the short term. The immediate response to this will first be to stop issuing the 20 year and reduce the 30 year. They may debate the 50 year but odds are if the 30 has a weak showing that there would be even less appetite for the 50.
But with the unwinding yen carry trade, we have to consider the impact on funding the AI boom going forward. Clearly this has been on the mind of nVidia recently, with the move to create a form of mortgage backed securities for GPU capacity. The largest tech companies have something like $2-3 trillion in shell companies set up specifically for the data center buildouts. The rest of the revenue seems to be going in a circular motion among several large companies. They know as soon as they stop funding the AI buildout, or are forced to by the unwinding of market forces, that the party will end. This is the very reason for the move from nVidia recently. And if mortgage backed securities don't make you feel better about it, they have also compared it to how airlines trade airplanes as assets. Neither is a comparison that they should want to make for historical reasons.
Now in the past, the U.S. allies would see the value in the financial alliance and would ride to the rescue of something like a 50 year bond to lock up these bubbles for a long time. After so many years of tariffs and other forms of coercion, even the recent yen internvention broke norms between the U.S. Treasury and the European Central Bank. Maybe in the past the Saudis, UAE, etc would have jumped in to invest, but clearly those days are over too... With massive implications for the petrodollar. Which itself is another major factor in the reserve currency system and is clearly under strain from the Iran War and other geopolitical factors.
The U.S. response to a debt crisis won't be to default, it will do what it always has done, and try to inflate itself out of the situation. It's just that they can't really take this path either. The more the Federal Reserve becomes the lender of last resort and expands its balance sheet to backstop moving all this to the short term repo market, the end result will be extreme inflation. The yields will begin rising on short term debt faster than we have been seeing on long term debt. Before long, it will be early 1980s style interest rates, and if they don't find a solution, this process will be self repeating... The $40 trillion, with roughly $1.2 trillion in deficits every year, and past reissuance, all into short term funding markets, where the only buyer capable of absorbing it all is the Federal Reserve, there is just no real way out of it.
Even all the structural problems with mortgage backed securities haven't been fixed since 2008. If the AI bubble pops, the energy prices keep increasing, the yen carry trade continues to unwind, any South East Asian conflict with North and South Korea, China/Japan/Taiwan, etc... Or with the Ukraine war ending poorly with Russia moving further into the baltic states...
Maybe hope that a guy who bankrupted several casinos can figure out to do when the largest casino in history runs out of money? I don't know.
EDIT: Sorry for the wall of text. Had a lot to say.