Comment by ganafagol | Hacker News Reader

Comment by ganafagol | Hacker News Reader

Parent Full thread

And it's also a quite nonsensical use of ML technology, to be honest. Neural nets are good for problem domains with fuzzy definitions of right and wrong as you find them in the physical world out there. Is this a tiger or a rock? Is this food or poison? Should I walk around this pond or swim through it? That's what the human brain is good at handling and neural nets are trying to approximate that.

Programming languages, on the other hand, are domains that are entirely man-made. We have created the rules, they are known in its entirety (in contrast to knowing all objects over there in the shade below the tree), and they are optimized to be so precise that they can be followed mechanically by a comparatively simple machine. Sure there are a few "undefined" corners but that's not because we don't understand them but because they are deliberately left open. Even they are precisely defined. So, with two precise definitions of semantics for two programming languages, it should be possible to build a translation based on those rules. Inherently, this is not an "empirical" endeavour, it's a logical/mathematical well-defined one.

Using neural nets for this is using the wrong hammer for the job. It's like using them for deriving the implementation of a sorting algorithm or to find the first 100k digits of pi. It just does not make sense.

And I'm not talking about "idiomatic" translations. That's a less well-defined domain an maybe that would be more suitable, but the post explicitly excludes this kind of thing. In principle, without providing examples of idiomatic style to the learning machinery, this can't ever work.

> "undefined" corners but that's not because we don't understand them but because they are deliberately left open. Even they are precisely defined.

Not sure what language you are referring to, because this statement is very vague, but it's worth pointing out that undefined behavior in the C/C++ sense is absolutely not precisely defined. If a program executes undefined behavior in either of these languages, the entire program execution is meaningless, including the time "before" undefined behavior occurred. The standard leaves room for the implementation to do absolutely anything before or after that point. In practice, what actually happens is the crazy complex interactions of a slightly broken abstraction that will depend on memory layout, optimizations, libraries, execution history, the phase of the moon, etc.

All that said, this isn't hard because undefined behavior. This is hard because languages are crazily complex and simply doing the semantically equivalent thing amounts to an emulation of one language in another, including all the implicit conversions, move constructors, copy constructors, overflow behavior, multiple inheritance, virtual dispatch, the whole mess. If you just do a full-fidelity source-to-source translation that amounts to emulation, you end up with an unreadable, gross mess.

Reminds me of that time I exported my MATLAB code as C.

> it's worth pointing out that undefined behavior in the C/C++ sense is absolutely not precisely defined.

I think OP means "it is precisely defined by the spec which actions result in undefined behaviour"

Yes. Thank you.

So why does it work better than the commercially available tools at the moment as claimed in the article?

Because the commercially available tools aren't that good and because they kept working on it until it was better.

If they kept working on it I think they would run into an asymptote. Maybe they could get closer and closer to 90% accuracy on a task with real hardware, 92% boiling the oceans, and 93% with a Dyson sphere, 93.5% if you can harness a quasar. At that point it probably passes a whiteboard interview and the people who have to fix the bugs can console themselves that the last programmer had neither a brain nor a soul.

That system has an approximate, not an exact model of the domain it works on and that is why it has an asymptote. Turning a graph-structured program into a vector is like mapping the curved surface of the Earth onto a flat map -- except instead of it being a 3-dimensional space it is more like a 1000-dimensional space. Information is destroyed in that process and forever lost so there will always be important characteristics of the problem that it will never "get".

If the message recipient is a person they will meet you halfway and might even accept bullshit if it is presented with complete confidence and lack of shame. The computer will interpret exactly what you said and reveal that you're a dog. (e.g. mute animal)

This article is a classic example of Facebook PR doing a wonderful job of selling the research and linked paper [1] claiming too much in the introduction. Please, please talk to actual researchers before you buy such claims.

If you go through the paper - you have to check the evaluation section to see how they measured their success. They used some programs from GeeksforGeeks to evaluate their approach. Problems on GeeksforGeeks do not represent the vast majority of programming tasks encountered in daily life. This is very much in contrast to the overarching claims presented in the introduction of the paper.

Second issue with the evaluation: they use BLEU scores to judge how good their translations are. BLEU makes sense for natural language translations (even that is widely debated in the NLP community these days). For a program there is no concept of an almost correct program (based on how things look similar), it is either correct or not. Eg. if I am asked to write a program to add two numbers and I write `x - y`, I am not almost correct, I am completely wrong. And in some ways that is what their model does, it optimizes for BLEU scores.

Third, the correctness of the programs are tested based on 10 random inputs. Are 10 random inputs enough to cover the entire input space that can be accepted by a program?

It is indeed a great advance in the application of ML technology, but it is nowhere close to the broader claims. One can even debate, ROI on time spent in gathering and curating data and then checking the correctness of translation from such system vs the ROI on writing rules for a rule based system since all programming languages are easily expressible that way.

[1]: https://arxiv.org/pdf/2006.03511.pdf

Reply on news.ycombinator.com