HNHacker News
TopNewBestAskShowJobs

EvgeniyZh

9,025 karma · joined April 27, 2016

https://evgeniizh.com/ https://t.me/j_links https://twitter.com/evgeniyzhe
submissionscomments
EvgeniyZh··on Has LLM killed traditional NLP?
250/s is few (4) ms per it
EvgeniyZh··on FrontierMath was funded by OpenAI
> Attempting to justify potential misconduct through semantic technicalities ("well, validation isn't technically training")

Validation is not training, period. I'll ask again: what is the possible goal of accessing the evaluation set if you don't plan to use it for anything except the final evaluation, which is what the test set is used for? Either they just asked for access without any intent to use the provided data in any way except for final evaluation, which can be done without access, or they did somehow utilize the provided data, whether by training on it (which they verbally promised not to), using it as a validation set, using it to create a similar training set, or something else.

> This directly contradicts established principles of scientific integrity where the spirit of agreements matters as much as their letter.

OpenAI is not doing science; they are doing business.

> This represents a stark logical reversal. The initial argument assumed benchmark manipulation would be meaningful enough to influence investors and industry perception. Now, when challenged, the same metrics are suddenly "meaningless." This is fundamentally inconsistent - either the metrics matter (in which case manipulation would be serious misconduct) or they don't (in which case there's no incentive to manipulate them).

The metrics matter to people, but this doesn't mean people can meaningfully predict the model's performance using them. If I were trying to describe each of your arguments as some demagogue technique (you're going to call it ad hominem or something, probably), then I'd say it's a false dichotomy: it can, in fact, be impossible to use metrics to predict performance precisely enough and for people to care about metrics simultaneously.

> The attempted simultaneous appeal to and dismissal of credentials

I'm not appealing to credentials. Based on what I wrote, you made a wrong guess about my credentials, and I pointed out your mistake.

> at this point, the argument OpenAI did something rests on unfalsifiable claims about the industry as a whole, claiming insider knowledge, while avoiding any verifiable evidence.

Your position, on the other hand, rests on the assumption that corporations behave ethically and with integrity beyond what is required by the law (and, specifically, their contracts with other entities).

EvgeniyZh··on FrontierMath was funded by OpenAI
> an unsubstantiated claim about widespread misconduct.

I can't prove it, but I heard it from multiple people in the industry. High contamination levels for existing benchmarks, though [1,2]. Whether to believe that it is just as good as we can do, not doing the best possible decontamination, or done on purpose is up to you.

> Yes, validation and test sets serve different purposes - that's precisely why reputable labs maintain strict separations between them.

The verbal agreement promised not to train on the evaluation set. Using it as a validation set would not violate this agreement. Clearly, OpenAI did not plan to use the provided evaluation as a testset, because then they wouldn't need access to it. Also, reporting validation numbers as performance metric is not unheard of.

> This reveals a fundamental misunderstanding of why math capabilities matter. They're not primarily about serving math users - they're a key metric for abstract reasoning and systematic problem-solving abilities.

How good of a proxy is it? There is some correlation, but can you say something quantitative? Do you think you can predict which models perform better on math benchmarks based on interaction with them? Especially for a benchmark you have no access to and can't solve by yourself? If the answer is no, the number is more or less meaningless by itself, which means it would be very hard to catch somebody giving you incorrect numbers.

> someone who has picked up some ML terminology but lacks fundamental understanding of how research evaluation, technical accountability, and institutional incentives actually work in the field

My credentials are in my profile, not that I think they should matter. However, I do have experience specifically in deep learning research and evaluation of LLMs.

[1] https://aclanthology.org/2024.naacl-long.482/ [2] https://arxiv.org/abs/2412.15194

EvgeniyZh··on FrontierMath was funded by OpenAI
Why wouldn't OpenAI cheat? It's an open secret in industry that benchmarks are trained on. Everybody does it, so you need to do that or else your similarly performing model will look worse on paper.

And even they respect the agreement, even using test set as a validation set can be a huge advantage. That's why validation set and test set are two different terms with precise meaning.

As for "knowing it's bad", most people won't be able to tell a model scoring 25% and 10% apart. People who are using these models to solve math problems are tiny share of users and even tinier share of revenues. What OpenAI needs is to convince investors that there is still progress in capabilities going at high pace, and gaming the benchmarks makes perfect sense in this context. 25% was surprising and appeared to surpass expectations, which is exactly what OpenAI needs.

EvgeniyZh··on Israel, Hamas reach ceasefire deal to end 15 months of war in Gaza
Even ignoring well-known Wikipedia bias in this issue, it is rarely a good source for "actual history"
EvgeniyZh··on The Illustrated Guide to a PhD
Even in US public universities have salary scales that are indexed. Moving to a different place while keeping tenure is possible too. Anyway I don't see how it's different from any other job, including the fact that having good relationships with colleagues is not a bad idea anyway.
EvgeniyZh··on The Illustrated Guide to a PhD
> that is essentially just contract work and not steady employment,

That's not true, the university pays you salary. Depending on details you may be able to increase your salary using grants, but you won't lose your salary if you don't. And when you hit tenure it is very hard to fire you

EvgeniyZh··on The Illustrated Guide to a PhD
> Hear a lot of bad things about the academic rat race, pressure to public even at masters/PhD level

Strongly depends on the advisor and your goals. If you want to stay in academia, some amount of publications is required. Your advisor, especially if he pays your salary, may also push you to publish. If both are not an issue, I guess you can even finish without publications.

> I could probably hack out some paper into journals but whether I could have any real impact "on demand" (versus say spontaneously coming up with something) is a big question mark, especially within the deadlines given in the program

Nobody comes up with good ideas on demand. As you progress in your academic career the rate of ideas (theoretically) grows. That's why you need the advisor: he can come up with ideas at rate sufficient for his students

EvgeniyZh··on Doctors Without Borders declares the war in Gaza as genocide
Exactly! Remember when Hamas was elected? In 2006, a year after Israel disengaged, removed all Israeli settlements, gave up control of Gaza-Egypt border.

Gaza elects Hamas, whose charter (at least then, 2017 version is toned down) is openly antisemitic, talks about jihad, killing Jews, protocols of elder Zion and Palestine from the river to the sea.

Giving terrorists more power as they actively talking about killing civilians is insane, however bad you feel for Palestinians. That's why the first step is the removal of Hamas, whatever the next steps are.

EvgeniyZh··on Doctors Without Borders declares the war in Gaza as genocide
They did [1], didn't they? Though they didn't claim that whole UK belongs to them...

[1] https://belfasttelegraph.co.uk/sunday-life/news/new-ira-plot...

EvgeniyZh··on Doctors Without Borders declares the war in Gaza as genocide
> I recommend the Amnesty International report

The report that says "current definition of genocide is too narrow to accuse Israel, so we need a different one"? That's exactly what I'm talking about:

> As outlined below, Amnesty International considers this an overly cramped interpretation of international jurisprudence and one that would effectively preclude a finding of genocide in the context of an armed conflict.

> The effects of Israel's actions are the same, no matter how you measure their intent.

The effect being lowest civilian to combatant death ratio in modern urban warfare? US in Iraq (battle of Mosul is commonly cited) had around 2:1.

EvgeniyZh··on Doctors Without Borders declares the war in Gaza as genocide
> There's plenty of proof of intent which has been addressed by actual international law experts

Do you mind citing specific proof you have in mind?

> The practical effects of Israel's action, regardless of intent, look like genocide.

What does it mean to "look like genocide regardless of intent" if the intent is the defining feature of the genocide?

EvgeniyZh··on Doctors Without Borders declares the war in Gaza as genocide
While all of these should be done (well not the one-state solution, but nobody really considering it anyway), how do you think it will stop the terrorist attacks?
EvgeniyZh··on Doctors Without Borders declares the war in Gaza as genocide
> MSF care less about the label

Why would they use a very strong label that they don't even have proofs for? All the genocide talk is only about labels. Genocide is something universally acknowledged to be bad, so calling something genocide is a good trick: who would argue for genocide?

> the international community to force Isreal into a ceasefire so they can do their job and save lives.

Well maybe saving hostage lives, that would also be a very strong argument for Israel to withdraw is something they should consider

EvgeniyZh··on Doctors Without Borders declares the war in Gaza as genocide
> almost all international bodies of law officially adjudicate the crime of genocide pursuant to the Convention on the Prevention and Punishment of the Crime of Genocide (CPPCG).

> This and other definitions are generally regarded by the majority of genocide scholars to have an "intent to destroy" as a requirement for any act to be labelled genocide

This post mentions intent once:

> While we don't have legal authority to establish intentionality

But anyway somehow

> What our medical teams have witnessed ... is consistent with the descriptions ... that genocide is taking place in Gaza

and then the title was changed from "Gaza death trap: MSF report exposes Israel’s campaign of total destruction" to "Doctors Without Borders declares the war in Gaza as genocide"

EvgeniyZh··on Doctors Without Borders declares the war in Gaza as genocide
Another day, another organization redefines genocide to fit their narrative, doesn't mention hostages held in Gaza and gets manually unflagged by admins. Business as usual.
EvgeniyZh··on The case against Google's claims of "quantum supremacy"
In 2019 you asserted [1] that attempts of creating distance-5 surface code will fail. Do you think you were wrong? If so, what was your mistake and why do you think you made it? If not, what's the problem with the Google's results? Have your estimations of feasibility of quantum computers changed in light of this publication?

[1] https://arxiv.org/abs/1908.02499

EvgeniyZh··on The Google Willow Thing
That's not true, there are enough interesting problems for NISQ. Quantum chemistry for example.
EvgeniyZh··on The Google Willow Thing
It's hard to say. For example the Google's paper talks about some rare (once an hour) strong errors. Are they fundamental or have some easy fix? We don't know.

One obvious problem is cooling. We can't cool million qubits in a single fridge, so we will need to split them between fridges and communicate. Also, the wiring is really complicated already and hard to scale (one reason IBM has heavy hex is to have more space).

Another problem is connectivity. For transmon qubits connectivity is fixed. Applying gate to two far qubits requires a lot of swaps, which is expensive. This is less of a problem for ions or cold atoms because they can couple any two qubits; but they likely wouldn't be able to for large amount of qubits.

Another thing is classical control, because the classical data needs to be processed at high speed. Some companies develop specialized hardware for that.

None of these is necessarily fundamental, but these problems need to be solved, in addition to usual scaling (it is hard to manufacture these devices and it becomes harder with each qubit).

EvgeniyZh··on Willow, Our Quantum Chip
Main players IMHO are IBM and Quantinuum, the latter employing different platform (ions). Neither could perform the same experiment I think, but have their own advantages. QuEra also looks good but are not as mature yet imho.
EvgeniyZh··on Willow, Our Quantum Chip
He told when the preprint was published

https://scottaaronson.blog/?p=8310

EvgeniyZh··on Understanding Google's Quantum Error Correction Breakthrough
I think it would be very __impactful__, but it is not really useful for humanity, rather opposite.
EvgeniyZh··on Understanding Google's Quantum Error Correction Breakthrough
I wrote a long-ish comment about what you can expect of QC just yesterday

https://news.ycombinator.com/item?id=42212878

EvgeniyZh··on Understanding Google's Quantum Error Correction Breakthrough
It's just a standard example of a code that works classically but not quantumly to demonstrate the differences between the two. More or less any introductory talk on quantum error correction would mention it.
EvgeniyZh··on IBM Quantum delivers on 2022 100x100 performance challenge
What is that reason? Quantum mechanics is a well-established theory with multiple experimental confirmations, but also quantum computers already work.
EvgeniyZh··on IBM Quantum delivers on 2022 100x100 performance challenge
All large-scale quantum computers are private; they technically do not need public support. I'd argue that having moonshot projects is a net positive for society, and I'm really happy that Google, for example, invests a non-trivial amount of money in them.
EvgeniyZh··on IBM Quantum delivers on 2022 100x100 performance challenge
I'm condensed matter theory PhD student and my research is close but not directly related to quantum computing.

There are some usecases for quantum computers and we are pretty sure that QCs are able to do stuff classical ones can't, but it is somewhat limited. In fact there is xprize to think of some applications [1].

You can separate usecases by whether they require fault tolerance. Fault tolerance is still very far away, and even single fault tolerant qubit (error rate of say 10^(-14)) would take a lot time from now, leave alone thousands/millions of them. We only had working error correction (i.e., error corrected qubit having lower error rate than physical one) this August [2] and it's still not better if you take into account the error correction overhead. Challenges include scaling manufacturing, control, cooling, etc.

Without fault tolerance you're into what's called NISQ (noisy intermediate scale quantum) where you just deal with errors instead of correcting them. As for now I guess we are in thousands of total gates we can apply before noise kills us. Maybe 10000 in the very very best case. Moreover the currently leading approach, superconducting qubits (Google, IBM), have fixed connectivity and applying gate on two disconnected qubits have additional overhead. Alternative approaches (Quantinuum's ion, QuEra cold atoms) don't have these limitations but have others.

The killer application for NISQ, in my opinion, are quantum simulations, i.e. simulations of quantum systems with quantum computers. Our ability to simulate quantum systems are fairly limited by the exponential growth of the state space, and smarter algorithms have their limitations, and you always simplify the model significantly. There may be some insights in simulating more complicated models. One obvious use for that is chemistry, but classical chemical algorithms are very well developed and classical computers are huge compared to quantum ones. Also chemical simulations require relatively deep circuits (N^3 scaling I believe), which is the main limitation of NISQ. Simulations for physical models (say, Hubbard model) are probably easier, but using this to justify billions spent would be hard. These applications in my opinion are pretty close (a number of years) to the moment you have real advantage (i.e., solve a problem in this domain nobody can solve classically), but this is optimistic opinion.

There are also applications for which the advantage may be there but we are not sure. Optimization is one example, though I think optimization on classical data will never close the vast resource gap (we have trillions of transistors in classical clusters vs hunders of qubits), even if there is advance. Optimization on quantum data probably is better on quantum computers, but again, not much pressing problems there.

For the fully fault tolerant the best known example is factorization. It is definitely better complexity then any algorithm we have classically, good enough to expect realistically sized QC one day will beat any classical computer. That said, while this has clear consequences from security standpoint, breaking some cryptography schemes it not exactly the most exciting thing IMHO, and people will switch to post quantum crypto way before (there was a recent work that could break existing PQC, but there was a bug in it [3]). There is Grover's algorithm for search but I doubt practical advantages there.

All in all it may sound a bit pessimistic, but I think if the technology arrives people will figure out how to use it.

[1] https://www.xprize.org/prizes/qc-apps [2] https://arxiv.org/abs/2408.13687 [3] https://eprint.iacr.org/2024/555

EvgeniyZh··on ICC issues warrants for Netanyahu, Gallant, and Hamas officials
I guess my question is towards admins to decide which stories to unflag and not to users. I'm sure many stories from both sides could got to the top if users wouldn't be able to flag them.
EvgeniyZh··on ICC issues warrants for Netanyahu, Gallant, and Hamas officials
I don't think I've seen any pro-Israeli post in top since the beginning of the war. Definitely anything I submitted was flagged to death almost immediately, even if it was hacker-ish (say, the analysis of the Hamas statistics). You can say of course that users decide what they want, but for political stories at least I don't think it is straightforward
EvgeniyZh··on If AI seems smarter, it's thanks to smarter human trainers
I'm currently pursuing PhD in theoretical condensed matter physics. I tried submitting questions to Humanity Last Exam [1], and it was not too hard to think of a problem that none of top llms (Claude, gpt, Gemini + both o1 models) got right. What was surprising for me is how small my bag of tricks was. I could think of 5-6 questions in my direct area of expertise with simple numerical answer that were hard for llms, and another maybe 5 that they were able to solve. But basically that was all my expertise. Of course there is stuff that can't be checked with simple numerical answer (quite a lot in my case), and there are probably additional questions that would require more effort from me to give a correct answer. But all in all, I suddenly felt I'm a one-trick pony, and that's given that my PhD is relatively diverse.

[1] https://agi.safe.ai/submit

← PreviousPage 4 of 14Next →