Chris Lattner Joins Google Brain | Hacker News Reader

Chris Lattner Joins Google Brain

techcrunch.com

277 pointsjanober102 comments

I find it strange that probably the most successful and influential compiler and PL developer in a generation doesn't seem to be interested in compilers or PL anymore.

There is an obvious overlap between programming languages and machine learning at Google: TensorFlow.

For example, see "A Computational Model for TensorFlow" https://research.google.com/pubs/pub46196.html

There are tons of interesting things going on in TensorFlow from a programming language perspective. It has optimization at the low level, like CUDA and SIMD back ends via the Eigen library [1] (which is pretty crazy C++ metaprogramming in its own right).

But it also has optimization at the high level / cluster level, e.g. deciding which nodes to put computations on, to minimize data movement across various networks, etc.

It also has multiple front ends. Python is the main one, but IIRC people were developing others (maybe not at Google).

I worked adjacent to the TensorFlow team, and A LOT of people had BOTH ML and PL skills [2]. It's not an either-or thing. It's best when you have the same person with both sets of expertise.

I think a big problem with large parts of the academic PL community is they're not exposed enough to real applications. Difficulty isn't proportional to real-world benefit. It's certainly difficult to invent type systems to statically detect minor problems, but that doesn't mean it's important for creating and maintaining software. (Sorry, had to rant about that.)

Machine learning is a domain rife with programming language problems, but of course it takes a long time to develop that expertise. I'm sure Lattner would be a good person to synthesize knowledge in the different domains.

[1] http://eigen.tuxfamily.org/index.php?title=Main_Page

[2] edit: I should really say ML and distributed computing skills. But most people with distributed computing skills know a decent amount about programming languages; they overlap in MapReduce-type big data frameworks too.

Jeff Dean got his PhD under Craig Chambers doing PL work; Sanjay Ghemawat was also a compiler guy, they went on to mapreduce and I think, at least Jeff Dean, is heavily involved in Tensorflow.

Google hires a lot of PL PhDs to do work that isn't very related to PL (we joke about PL PhDs working on ads a lot). One could guess that they just make good developers because of their experience, but the specific knowledge might not be much in demand.

Right and Craig Chambers later joined Google. He led/developed Flume [1] (a Java framework on top of MapReduce) and the unfortunately named Cloud Dataflow [2] (which aims at unifying streaming and batch computation)

I say unfortunate because pretty much all the big data frameworks could be called "cloud dataflow", including TensorFlow itself.

People using Google Cloud might recognize the name of VP Urs Hoezle, who worked on the OO language Self in the 90's with a lot of the same people [3].

And AFAIK that's where v8 came from... Urs hired his former colleague Lars Bak and told him to write a fast JavaScript interpreter for Chrome.

[1] https://research.google.com/pubs/pub35650.html [2] https://research.google.com/pubs/pub43864.html [3] http://dl.acm.org/citation.cfm?id=619798

Wasn't V8 developed by Bak & co in a separate company, which was bought by Google? (Didn't find any source, but based on what I thought I read in the news at that time ...)

v8 wasn't developed by a separate company, but Bak did bring at least one of his colleagues along to Google (I'm pretty sure just as new employees to Google). And he had another company that worked on a SmallTalk VM called OOVM, targeted at embedded devices, right before he joined Google:

https://astares.blogspot.com/2012/04/inside-oovm.html

However it apparently wasn't acquired by Google:

http://www.businesswire.com/news/home/20040727005088/en/Esme...

Other discussion:

http://forum.world.st/OOVM-details-td4561492.html

> There is an obvious overlap between programming languages and machine learning at Google

This doesn't explain his very brief stint at Tesla as head of AutoPilot.

Read his resume, that will explain it very well.

The question was why a PL developer has seemingly turned away from the domain in which he's prominent, to work on machine learning, where he has no prominence to my knowledge. His resume does not explain it very well, or at all.

> which is pretty crazy C++ metaprogramming in its own right

Expression templates aren't that crazy, in the grand scheme of C++ meta programming.

But they are the ur-crazy.

Not even close. At the very least, you have to consider APL.

Makes me wonder if Google is working on a programming language for machine learning...

That's what TensorFlow is. It happens to use the Python interpreter as the front end, and for metaprogramming, but the language has its own semantics (see the paper above).

You could probably an invent your own syntax for it (and I'm sure someone has), but that's a small part of the picture.

Interesting idea, indeed. Add to this programming language:

1.True multithreading.

2.Native support of spread computation to heterogeneous computation devices, across CPU/GPU/Whatever-Chip-You-Call

3.Auto vectorization.

4.PyTorch style autodiff, and should be able to JIT hotspot as the program runs.

5.Native distributed support with built-in primitives.

6.Visualization as standard library, even language primitives.

This actually blows my mind. I know that probabilistic languages exist, but something very interesting could be done here.

It may have to do with the fact that he just wanted some new challenges to sink his teeth into. As an old school hacker myself, I can relate. Even really cool technologies start to feel mundane and tedious after you've done them for long enough.

The real challenge with senior positions is adjusting to new environment/dynamics, not to mention the expectations that come from "legend" status. I'm not a legend, but I certainly had a much easier time in the earlier stages in career just focusing on the tech direct. When you're the hot new hire, a certain group at the new company expects a lot form you, while others who have been there longer think it's just hype. (I'm assuming something similar happened at Tesla).

Wish him all the luck on the new adventure! And being an AI geek myself: NO AI cannot solve everything. Not even close. (at least for now)

Sentient machine conversations are straight up funny to me. Still, I can't help get over the cool/geek factor of working on Machine Learning/Deep learning or exploring new algorithms that could significantly improve the field.

If you want to understand his motivations, he gave a wide ranging interview to the ATP (Accidental Tech Podcast) guys at the time he jumped ship to Tesla.

Basically it seemed to boil down to him now being a people manager as much as he is an engineer - he really enjoys helping teams of people succeed at complex technical problems. This doesn't surprise me at all, given how the nature of work often changes as one climbs an org tree.

http://atp.fm/205-chris-lattner-interview-transcript/

As a machine learning researcher I have to say that you should join us too! AI is the future!

PL is an area with big challenges, but AI has even bigger challenges and the impact in the society is bigger, much bigger, too. I would change the question and ask why anyone shouldn't be interested in AI?

Because not every problem can be solved by AI.

I'm more interested in IA (intelligence augmentation) actually, which seems to overlap with PL nicely.

Not now, but if you can really create AI, it shouldn't be indistinguishable of human Intelligence.

My point is that it is unlikely to make big steps now to advance in PL, but it is very likely you can do it in AI. The last 5 years have been crazy and the people working on it now is much more than few years ago.

Note: most probably you wont be getting human-like AI anytime soon

> Not now, but if you can really create AI, it shouldn't be indistinguishable of human Intelligence.

If we really create AI that is "indistinguishable of human Intelligence", then all questions about human jobs or research become moot, no?

If someone was to create AI that is human-equivalent (aka strong AI) then that would be their final project as a human being. There would be no need for human professional or academic effort after that point.

>'If someone was to create AI that is human-equivalent (aka strong AI)'

Creating AI with human level intelligence isn't going to be so great, most people are complete idiots. Creating AI with above human intelligence is a different story.

A well-known "paradox" states that tasks that require savant-level intelligence from humans (like playing chess) are actually doable with computers, but tasks that any idiot can do (say, pouring yourself a cup of tea) are crazy hard.

Well... They said the same in the industrial era. There are a lot of things to consider, for example:

- AI might not like to work, as we don't like to work

- You suppose that if you create AI then the AI can create a more intelligence AI and this to the infinity. We don't know if the intelligence is unlimited or limited. We might have reach the top intelligence. (note: the more intelligent the brain the slower it is, and a bigger brain does not relate to intelligence)

Sounds a lot like saying people should get into management so that they can get other people to do stuff for them. People can do anything people can do, so you should get into people.

And it's totally true! A good manager can make anything happen that their team can accomplish, and their impact can be huge. But some people want to actually work on the stuff, not work on the stuff that works on the stuff.

Actually, compilers are a great example of this dynamic - when you work becomes more and more impactful, but more and more removed from actual problems people have. Say, you write a script to solve a specific problem someone has. But that is not very impactful - one person is happy and that's all. So you generalize your program to a whole class of problems. But that's not the end. Next, you write a framework for writing programs similar to the one you've developed on the previous step. And of course, all languages being shit, you'll want to write a compiler for a language to write your framework in. So you end up with a thing that is very impactful but also pretty useless on its own.

Nice example. Really, lots of technology is like this. If you enjoy arithmetic, you do it. If not, you use a calculator. If you enjoy digging, grab a shovel, otherwise use a backhoe. Nothing wrong with picking the layer you enjoy, and a backhoe isn't always the best answer anyway.

PL seems easier to get into and make useful things with, like a static analyzer that enforces some rule at a big co. From my perspective it seems like your need something PhD level to do useful things with it and get paid better than a standard FANG software engineer.

I'm probably wrong although, it's probably an exposure thing.

Well as another machine learning researcher, I think you can make a major impact working on problems that aren't in AI.

Now if you are a PL person interested in AI we would love to have you and your expertise!

Oh and if you want to use AI to solve PL problems like program synthesis and program completion, that's cool too.

Haha I was going to say, solving synthesis via PL+AI is way more exciting than either technique alone.

-- A former PL researcher making AI way more useful via vanilla PL techniques

Welcome to the party!

He is still on Swift core team so why you think is he is not interested in Compilers/PL? A person of his/her caliber should be able to work on multiple things with very high level of performance.

Google has its own GPGPU compiler: https://research.google.com/pubs/pub45226.html

It's actually all open-source in clang: https://www.youtube.com/watch?v=KHa-OSrZPGo. Try it out!

That's a great talk. Thanks a lot!

Well he already said in a couple of interviews something like "I write C++ so that you don't have to".

Also a few other Swift team members used to be part of ANSI C++ comitee.

Don't you get bored doing the same thing for more than a decade?

I'd expect Google is looking to extend XLA: https://www.tensorflow.org/performance/xla/

Well maybe he realized Swift is a terrible programming language he should do something else.

(Yes I know, he's still involved in Swift. Its a joke. Though Swift really is a terrible programming language).

AI/ML is the most exciting fields in CS now. It brings challenges on so many levels: algorithms/software implementation/hardware(client/server/network communication)/resource placement(heterogeneous computation ), and the list goes on. I would go as far as to say it has the potential to ascend as the new purpose of computation itself. Hardly blame someone jumps the ship to goldmine this new wild west, however established that person in his own field may have become.

I'm finding that some traditional developers nowadays are more and more interested in AI/ML, rather than just plain coding. Maybe they're able to tell which way the wind is blowing a little better than most of us? After all, one of the applications of ML will be to learn how to write software. It could very well put traditional developers out of a job.

PL / compilers has a lot to do with logic and formal reasoning, so I wouldn't be surprised if he's interested in applying that to other domains. Merging DL with more traditional reasoning frameworks is an interesting question I know a few people have tried attacking.

Of course, that's just speculation.

After having been working on compiler and PLs for more than a decade, you would feel bored and want new challenges. Besides, Chris did not leave PL, he is still involved in Swift, LLVM etc.

Maybe he'll create a higher AI language for TF. That'd be really really cool.

Quite possible that his last contract's non-compete still makes that hard...

Super interesting developments..

Also, re:

>AI can't democratize itself (yet?) so I'll help make it more accessible to everyone!

This needs to happen more on the software side. You can buy a world-class quad-gpu machine for about half the cost of a vehicle. You can build a world-class single-gpu machine (what I have) for a fifth of that. It's literally amazing how accessible the hardware is, compared to almost any other science which requires 7 figure funding to get world-class results.

The software, the learning, the math, all need to become more and more accessible so more people can pick up and train an algorithm.

I lost access to a TrueCrypt volume from years ago, and using AWS GPU spot instances, I was brute forcing ~1000000 permutations every 40 seconds last night for $7/hr. The hardware is incredibly accessible.

Were you able to crack it?

Yes. Shame on past toomuchtodo for using too few characters, but also thankful I didn't need to spend hundreds of dollars on compute time.

How many characters was your password?

Well, if he was trying 1M combinations every 40 seconds, for $7 per hour, and he didn't need to use hundreds of dollars per hour of commute time, let's say 10 hours = $70. That's 900M combinations per hour, so 9B combinations in 10 hours.

If he was trying combinations using upper-case characters, lower-case characters, numbers, and let's say 20 symbols, that's 82 possible combinations for each one. We'd expect him to find the password after exhausting half of the search set, so we want log base 82 of 18B. That suggests 5 characters.

If he let's say just used lower-case characters and numbers, that's log base 36 of 18B, which suggests 7 characters.

I mean...you can't make math simpler. like...math is math. you can create resources that make it easier to teach math (and for the record high quality textbooks on AI/ML ABOUND). I think you want software and libraries so you can build things without understanding the math. To some degree this already possible (see my english major linkedin friends playing withs sci-kit etc etc). Fundamentally though, I think you need at least some basic understanding of the mathematics to interpret results/create new ideas. Arguing otherwise is like saying you can code without knowing what a queue is...I mean sure you can probably create something that compiles and runs but your horizons will be more narrow than someone with a full computer science education and grounding. just like trying to do ml without at least some understanding of the math is like a blind man trying to open a door.

case-in-point: I am on hn right now because my research is going badly because I am using another researcher's code and I don't fully understand the correspondence between the paper and the code. I am getting terrible results on some multi-class classification (i.e. approximately random). I am not sure if I am using some wrong parameters or because the signal I am looking for doesn't exist in the data. Not knowing at least some mathematics would make this a LOT harder

Re: "I don't fully understand the correspondence between the paper and the code"

Have you tried contacting the paper's author(s)? Preferably the PhD student that did the actual work?

I used to do that when I couldn't make sense of a paper. Not all authors will respond, some might brush you off with a generic remark to read the appendix or whatever. But it's worth a try. In one case, the author frankly told me that the code didn't work as well as it sounded in the paper -- the algorithm was so instable that it wasn't very useful. But they couldn't have written that in the paper, or it might not have been accepted.

Yeah I am collaborating directly with the primary author...but I think you might be right I would have gotten more helpful advice for the graduate student who spent like a year writing this.

Good advice-I'll keep it in mind for future troubles. Thank you!

mean...you can't make math simpler. like...math is math.

Notation makes a huge difference. LinAlg is a good example of this. And a PL is notation.

Simpler notation might make it easier to implement algorithms without fully understanding them. Not sure how much good is going to come from that.

But if you fully understand the problem, notation is not an issue.

I disagree. Try using Roman numerals for example. Or doing linear programming or any sort of optimisation without using matrix notation.

mmm I do see your point. Notation can frame how we think. I hate to be that guy though...but I am a bit triggered by calling linear algebra LinAlg. It is like calling California Cali! Or Analysis -> Anal.

Excellent job mathing, mathperson. May the math be with you!

Just wondering - does a normal person stand a chance to join Google Brain, or do you need to make node.js, swift, own self-driving car or another crazy project to get in?

Not sure about in general, but it seems like their residency program [0] gives a good shot at landing on the team afterwards and is accessible to "normal" people passionate about ML

[0] https://research.google.com/teams/brain/residency/

Another way is to do 20% and then transfer in which is what I did. That seems doable (I don't have PhD for example).

What do you mean by 20%?

Start on another team at Google (preferably using ML) and do 20% time with Brain.

>…he created the Land compiler and LLVM.

I think you mean Clang and LLVM

http://nondot.org/sabre/

"I lead and am the original author of the LLVM Compiler Infrastructure, an open source umbrella project that includes all sorts of toolchain related technology: compilers, debuggers, JIT systems, optimizers, static analysis systems, etc. I started both LLVM and Clang and am still the individual with the most commits."

The parent is likely talking about the "Land" typo.

It was probably a clang->land auto correct on a phone. It used to be I only had to worry about my fingers hitting the wrong keys, now I have to worry about my computer not believing that what I wanted to type was what I actually typed.

My mistake. I skimmed over the word "Land" without really noticing it.

Add voice dictation and things get even weirder. A lot of times it's magical with some crazy word thrown in.

Agreed on the voice thing. I've found some really odd sound alikes in my comments when I've used it (trust me it sucks to type a long response on a phone keyboard :-) A common one is substituting 'by' and 'buy' at different times.

It's always fun to try to dictate "wreck a nice beach." It's such a classic example of sound-alikes gone wrong that it seems engines go out of their way to interpret it as "recognize speech" unless you take extreme measures.

That would be a weird autocorrect, since clang is an actual word.

Is Google's work culture more like Apple or Tesla? Curious if this is likely to address whatever about Telsa wasn't keeping him happy. Or maybe it was more about the specific work he was tasked to do?

It's like neither, but probably pretty similar to the academic UIUC culture where LLVM got started. Having both worked at Google and followed Chris Lattner's work on LLVM since before he was at Apple, I suspect he'll be happier there than at Tesla.

I've worked at both Google and Apple. They're very similar, but in general with big tech companies there's more variation between teams than there is between companies on average.

From what I've heard from friends working in those 3 companies, Tesla is more like Apple, with Google being clearly different from both.

I was kind of worried he would not find anything... :D

Seriously, I wonder if stars like Lattner even need to submit a CV or their mail simply floods with offers.

I am guessing he doesn't have to write code on a whiteboard.

Damn, I always thought interviews of guys like him went along these lines: https://www.youtube.com/watch?v=XUhsNIeIgpY

I think they don't even have to explicitly show attitude like in the video you shared. Their past accomplishments in managing complex systems and projects can more than speak for themselves. :)

This looked like a pretty idiotic advise for the most part.

But how will they know if he can invert a binary tree?

Well, he still has one: http://nondot.org/sabre/Resume.html

When he tweeted about leaving Tesla, he got about sixteen billion replies from people asking him to come work for them.

I have zero experience with this, but I imagine working for three major competitors in such a short time can lead to some complicated situations regarding industrial secrets.

Compilers (specially open source) may be a stretch, but Tesla AI and Google AI? Can a contract clause protect one company when you're working for the other on the exact same field?

Again, I have no idea what I'm talking about, just curious.

Non-compete clauses are illegal in California. That's one of the big reasons for the tech boom there.

So, how does that work with, non-California companies, like Microsoft? Former MS employee joins FB/Google. Now can MS sue them?

What about the years of "moratorium" in cases like Scott Forstall?

Worrisome for Apple that its AI efforts weren't interesting enough to keep Lattner, or win him back.

I would think that the whole Siri fiasco would keep anybody in AI/ML the hell away from Apple.

Care to elaborate on the Siri fiasco? I'm out of the loop.

If by fiasco you mean leading the way the industry would take and being on par with current competitors (better at some things, worse at others)

Siri on iOS 11 is on par with Google Assistant, give or take. It's not world shattering, but i would not call it a fiasco.

Swift on Android confirmed.

Interesting as a company change (IMHO), from privacy-focused Apple to Google... :D

Privacy focused? Oh, you mean how they pivoted when their Ad business failed.

Interesting description of Apple's business!

Didn't their ad business fail because they refused to share/collect as much user data, and hence advertisers were less willing to spend on their platform?

Not to my recollection. They exited the Ad business because it was a financial failure. Do you really think Apple would have exited a business if they were on track to make Google like revenue from it? No way - the shareholders would have raided the building.

My comment was ironic...the "IMHO" was there for avoiding Apple fanboy.

Reply on news.ycombinator.com