toth

LLM growth (or frontier ML model growth in general) is driven in part by the original Moore's Law, but right now the main driving factor is the growth of investment. I.e., how much companies are prepared to spend on training run. This has been growing at more than 10x per year.

I work in HFT, so feel free to dismiss what I say as biased...

There is this widespread perception that HFT traders are rent seekers, and bad for markets, and in particular for retail traders. Anyone well informed knows this is completely wrong:

  - HFT is not rent seeking, it is providing a valuable service (liquidity provision) that people are willing to pay for. Over the years, there are been multiple attempts to set up trading venues that exclude HFT traders and they all end up inviting them in because without them there's no liquidity and very little trading.

  - There are multiple studies that show that regulatory interventions aimed at reducing HFT, such as financial transaction taxes, increase volatility and spreads and raise effective trading costs for retail traders.

It is still possible that changing market structure or regulation to reduce the incentives for HFT (and in particular the speed race aspect of it) could be somehow beneficial for markets or society as a whole, but this is far from obvious.

As for having regulation steer what "brilliant minds" go into, you have to have a lot of faith in the foresight of regulators to argue for that. Could say a lot of similar things about different industries, such as the usual quip about the best minds of our generation working on maximizing ad click through rates.

I always wonder about what something in between popular science/math and academic journals would look like. Something oriented towards people that know a nontrivial amount but are not researchers. E.g., you can assume they know calculus and have a conceptual understanding of what things like manifolds and homotopy but are fuzzy on the details.

For a topic like in the article, you need a lot of words to give a very vague understanding if you are aiming for a general audience. Not clear it is worth it for the reader. For that more targeted audience it could be a lot shorter and give a little more detail.

Probably too small a market, but I would definitely enjoy that type of content a lot.

You probably are aware, but just to make it clear for others: topologically a torus has one hole (genus 1). The inner void is not considered a hole. Similarly, a 2-sphere (surface of solid sphere in 3d) has 0 holes (the inner void is not considered).

You are right, but I am pretty sure they just meant the inner tube ignoring the puncture for inflation.

Usually people just use donut/bagel for an illustration of a torus. Not sure why they used inner tube here - maybe to make it clear it is just the surface?

This type of instictive negative reaction to prediction markets is, unfortunately, common, but, I think, misguided.

Prediction markets are one of the (or just, the?) best ways of aggregating knowledge from multiple sources and producing the best predictions. Having good legible predictions of impactful events such as LK-99 replication is extremely useful for society - it would be an invaluable input for a savvy policy maker for instance.

What I think is silly is that vastly bigger amounts of money are put in betting markets for any mildly important sportsball game. Meanwhile, markets on LK99 replication, one of the most potentially important possibilities in the world right now have only on the order of hundreds of thousands of dollars in them.

And there is no scamming involved. If you are participating in a prediction market, either you have some reason you believe you know something the market does not or you should expect you are simply subsidizing those with better information. The latter is a perfectly reasonable thing to do - it's not easy for an average person to "pay off some lab", but if they provide liquidity to the prediction market they are giving an explicit subsidy for anyone that can answer the question.

Maybe not everyone, but if JPM doesn't then who does?

Very cool! The top answer has a 12,731 character regular expression that matches base 10 numbers that are divisible by 7. (There is an answer with only 105 characters, but that uses .NET "regular expressions" which have some pretty gnarly extensions and are definitely not regular languages).

The regex itself is generated from a relatively simple DFA based on mod 7 arithmetic on digits using a program (JFLAP apparently, not familiar).

Just from looking at the start of the regex you can see some cool things. For instance, TIL that all numbers of the form 1555....55554 and 85555....5554 are multiples of 7. Fun exercise to prove it.

You are right that this is state dependent, and I am certainly not a lawyer. I have no direct knowledge about WI. My personal experience concerns NJ/NY.

Still, what I understood at the time is that, if the employer is not paying you at all during non-compete period, they will have a hard time enforcing it anywhere.

I am also surprised by the claim that it could cost millions to litigate about something like this, how confident are you of that number?

(And about CA, my understanding was that noncompetes are simply to allowed there, so it wouldn't even apply)

Off topic, but I have to ask. What do you have against Kelvin?

That is very interesting. As you most likely know, Borges was famously a huge anglophile, he once said "I find English a far finer language than Spanish" in an interview. There's a video of his reasoning:

https://www.reddit.com/r/Spanish/comments/13cg3xv/jorge_luis...

I tend to agree with him. I am not a native English speaker and also find it more expressive than my mother tongue.

Not really. If you are not being paid, you can just take a new job during the non-compete period and if your previous employer sues you they will lose, and they know it. They could still sue you just to make a point and make you spend money on a legal defense, but it's unlikely you'd spend millions of dollars.

In my case, employer didn't want to pay. Hired some lawyers to send a letter saying that either they paid me or the non-compete was invalid and ended up getting paid full salary for the duration.

FYI, in all jurisdictions I know of, this is already the case. I.e., a non-compete period in which you are not paid to a level commensurate to what you got before is not enforceable. If employers try to enforce them in court, they will lose.

Source: had a non-compete clause in employment contract that had no mention of any pay during period. Employer initially did not want to pay me anything during period. Hired some lawyers and they paid 100% of my salary during that period. All the lawyers did was write some letters, never had to go to court.

This a letter anyone can sign. As someone pointed out Grimes is one of the signatories. You can sign it yourself.

Hinton, Bengio, Norvig and Russell are most definitely not getting prestige from signing it. The letter itself is getting prestige from them having signed it.

Roll to disbelief. That tweet is precisely about what I mentioned in my previous post that doesn't count: finetuning LAMA derived models. You are not going to contribute to the cutting edge of ML research doing something like that.

For training LAMA itself, Meta I believe said it cost them $5 million. That is actually not that much, but I believe that is just the cost of running the cluster for the the duration of the training run. I.e, doesn't include cost of cluster itself, salaries, data, etc.

Almost by definition, the research frontier work will always require big clusters. Even if in a few years you can train a GPT4 analogue in your basement, by that time OpenAI will be using their latest cluster to train 100 trillion model parameters.

State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months, huge amounts of electrical power, and expensive-to-create proprietary datasets. Not to mention large team of highly-in-demand experts with very expensive salaries.

Many ways to regulate that. For instance, require tracking of GPUs and that they must connect to centralized servers for certain workloads. Or just go ahead and nationalize and shutdown NVDA.

(And no, fine-tuning LAMA based models is not state of the art, and is not where the real progress is going to come from)

And even if all the regulation does is slow down progress, every extra year we get before recursively self improving AGI increases the chances of some critical advance in alignment and improves our chances a little bit.

I disagree. My read on him is that until very recently (i.e., possibly when GPT4 came out) he didn't take x-risks concerns seriously, or at least assumed we were still many decades away from the point where we need to worry about them.

But the abilities of the latest crop of LLMs changed his mind. And he very publicly admitted he had been wrong, which should be applauded, even if you think it took him far too long.

By quitting and saying it was because of his worries he sent a strong message. I agree it is unlikely he'll make any contributions to technical alignment, but just having such an eminent figure publicly take these issues seriously can have a strong impact.

I think you are wrong. The risks are real and, while I am sure OpenAI and others will position themselves to take advantage of regulations that emerge, I believe that the CEOs are doing this at least in part because they believe this.

If this was all about regulatory capture and marketing, why would Hinton, Bengio and all the other academics have signed the letter as well? Their only motivation is concern about the risks.

Worry about AI x-risk is slowly coming into the Overton window, but until very recently you could get ridiculed by saying publicly you took it seriously. Academics knew this and still came forward - all the people who think its nonsense should at least try to consider they are earnest and could be right.

As a EU citizen, I really wish we had competent leaders, particularly in Brussels. Have never seen any evidence that is the case, specially with regards to tech.

The last EU initiative at sweeping regulation in the tech space (GDPR) was disastrous. It imposed huge compliance costs for all entities, from large multinationals to small startups and business and even some individuals and nonprofits. For the multinationals that is totally fine with me, they can afford it, but for the other ones it is far from clear that the benefits outweigh the costs. They could have just made it not apply to small entities (either in revenue, number of users or some other metric) and it would have been arguably great. The way they did it they just gave a big advantage to large incumbents and put a strong handicap on EU based tech start ups (as well as other businesses to a lesser extent).

Years after the roll out, I still occasionally come across international websites that have opted to ban European visitors than figure out how to comply with GDPR.

Don't get me wrong, the GDPR was meant to address real on going abuses of personal data, and did some things right, but it could have easily had most of the benefits for a fraction of the cost if they did things right. Unfortunately, like all almost all regulators, Brussels tends to pay a lot more attention to hypothetical benefits of regulation than its predictable costs.

While I don't disagree with you, if you are going to say something like you should really at least give the right number. Or at very least include a disclaimer that 12 hours is not the right number.

2022 Meta revenue was 116 billion USD [1]. So the fine was 1.1% of yearly or revenue, or pretty close to 4 days of revenue.

In terms of yearly net income, it is 5.6% or 20 days of income. Don't think this is a trivial fine.

[1] https://www.statista.com/statistics/277229/facebooks-annual-...

This is really fun. Thank you!

You should really adjust for dividends, in which case it is still down, but only -7.5%. [1]

[1] https://dqydj.com/nikkei-return-calculator-dividend-reinvest... (I did Dec 1989 - Jan 2020. They don't have data latter than that)

This is definitely wrong. In multiple conversations with Uber drivers, they prefer the longer trips as it makes them more money. They are able to not take long trips if they prefer (although they don't know the exact destination before accepting a trip, they are told if it is long one), and choose to do so.

Leap year rules have to do with the actual length of the year being 365.24219 days - not with days not being 24 hours.

Not the OP, but it wanted to point out it has 10X less error on the holdout sample so it is not simply overfitting.

Congrats on the paper, very cool work. I bet the "Datasaurus dozen" is going to become the go to example for the dangers of relying on summary statistics, it is much more memorable than Anscombe's quartet.

Not sure why this is downvoted to gray, it is correct. In most countries, if you are a tax resident, your foreign income is taxed. What is unusual about US is that it also taxes foreign income for US citizens that are living abroad, almost no countries do this.

I see, that makes sense. I guess at that point you have something very much like std::map, except it doesn't keep itself sorted, you have to do it yourself.

I guess for cases like this where you initialize it once and never change it again it could even be a better choice (i.e., faster initialization).

I get that part, that's why you use std::find or std::lower_bound on the vector, but my point is that std::lower_bound will not give you the right answer unless your enum is defined with the names in alphabetical order.

Interesting write up. There was a claim I didn't understand though

> Let’s fix this. An easy way to get our std::vector implementation in-line with the rest of the containers is we can sort it and utilize an algorithm, std::lower_bound, in order to speed up our lookup times.

Unless I'm missing something, isn't this wrong? The point of the vector is that v[enum_value] gives you the name of enum_value as a string. Once you sort it this relationship no longer holds, unless it so happens that the vector was already sorted (which happens to be the case for "circle", "square", "triangle").