Business questions engineers should ask when interviewing at ML/AI companies

Anybody recognizes how similar these questions are to what an investor would ask? This is what I'm always telling developers. You are an investor. You invest your life, health, best hours of 5+ days a week, ambition. Your investment is not as important to the company as a few million bucks would be, but to you it should be more valuable than a few million bucks. It's all you have. And you want to give it to someone in exchange for money, you better make sure that someone is worth your investment. So read what investors are asking a prospect company, and ask the same questions. Only accept if you would give them a million bucks.

After working for a few companies who (tried to) make ML products, I think one of the most important questions to ask is "who owns the data you are building models on?". It is way harder for a company to build good models off data they don't have complete access to and full knowledge about. The worst (and unfortunately common) scenario for companies trying to do AI is that data scientists don't have full access to all priors necessary to build good models, and the owners of the data don't really know much about it either. Spells death of company

The fastest way to make progress on these business questions is often to build hacky MVPs that look like they're doing something smart, but behind the scenes are powered by humans or dead-simple algorithms, and get them in front of customers ASAP.

I recently joined a seed-stage startup solving a business problem via audio analysis in the manner I described above. I'm not spending much time doing ML yet, but I'm banking on my belief that we're solving a valuable problem (customers want to buy our hacky MVP) and that ML can and will be needed to scale our solution. By deeply understanding the customer as a first step, I think the ML systems we build will be business critical and enduring. Time will tell

Or, you can't figure out how AI is helping solve the problem they claim to solve, and you realize they are just throwing every buzzword out there in hopes you'll be impressed.

https://www.youtube.com/watch?v=kzT3yfe2o-I

> Google got great because of PageRank, but it stayed great due to network effects.

I think this overstates network effects and under-appreciates branding and simply reinvesting significant capital into continued R&D.

"Why does anyone need this? Like all advice, this sounds deceptively simple. But make sure you get a very compelling answer here."

Right now I'm at the Web Summit, in Lisbon. I saw a few ML/AI startups here, but I was surprised there weren't a lot more. There was some imbalance. There were many startups trying to create new online social networks for niche groups -- business professionals, women, entrepreneurs, sales professionals, etc. I do think there is some innovation that can happen in social networks, but generally I regard it as an over-crowded space. For the most part, the benefits offered by the Internet seemed to have been mostly absorbed by the 1994-2008 cycle, and what's left is fairly minor compared to what happened previously.

One of the few areas where I still see the possibility of significant traction (the creation of large companies) is with information gathering and a combination of ML and NLP. We've already seen a wave of startups which were no more than pretty GUIs over existing technology. At the risk of being unfair, BigML.com is just a nice interface over existing ML tools, and API.ai is just a nice interface over some NLP tools. AmenityAnalytics goes a bit further combining their web scraping and NLP scripts with a nice interface that customers can use to filter the incoming data. But there is still a wide space for companies to go much further in this field.

All the same, I agree this is a good question: "Why does anyone need this?" You should really ask it whenever you are joining any startup.

Typical of SV to focus solely on the business viability of a firm to the detriment of everything else. I’d like to work somewhere knowing that I have a clean conscience. Why not add:

Where do you get your data from?

If your data is sourced from users of your product, do you tell them what you’re collecting?

Some other questions I would want the answers to:

1) Is a Non-Compete Agreement required and what are the terms (time and geographic coverage; compensation during the non-compete time frame)?

2) Is the company going to assert ownership over IP created before employment began?

3) Is the company going to assert ownership over IP created that is unrelated to current responsibilities (or IP in a area unrelated to the company's current business), especially IP created on the employee's own time, not using any company resources?

The best use of these questions is to get an idea whether your equity might be worth anything. Ask how many customers they have, how fast they're growing, how long does it take to onboard customers, etc. What do they see as an exit (is it a 10M, 100M, 500M a unicorn?). When do they see it happening?

If, let's say, you have 0.01% of a Series A that exits at a billion in five years. You'd expect to be diluted by half and there is around a 20% chance of exit that high so roughly ((0.01%billion)/2).20 /5 or about 2k a year in 'expected value'(or why, numerically at least, it's likely not worth it to take a paycut to work at a startup unless you're seeing substantial equity).

Did any of you succeed in asking such questions to companies (ML/AL or not) and ended up receiving reasonable responses to your questions?

My experiences have never been great when asking such loaded questions. To be clear, I interviewed extensively at small startups and I'm only taking those interviews into consideration for this comment.

The HR and technical interviewers dedicate a large amount of time for questioning you, and reserve the last few minutes to answer your questions. I lost count of how many times the interviewers would just blabber something meaningless in a rush, rather than answer a questions with patience and honesty. Of course, I made my decisions on whether or not to join the team based on such experiences. However, I cannot think of any remarkable instances where the interviewers answered such questions without getting impatient.

Any chats with directors/PMs/C-level executives after receiving an offer were also not very informative. I walked out several times going in and coming back from such chats with no questions properly answered because: (a) they are still figuring out, or (b) they cannot discuss certain details because you haven't said yes to them.

This is really solid advice when looking at it backwards too--if you, as a founder, cannot produce reasonable and compelling answers to these questions, you don't know what you're doing, where you're going, or how to actually get there.

Great read.

Not sure why ML/AI is in the title. Seems applicable to any startup.

These questions should be asked about every startup, not only ML/AI.

I am in academia, but some of my students have returned somewhat disgruntled from their internships in companies claiming to do machine learning, while in practice their systems were primarily rule-based and the work consisted of writing rules [1].

I now recommend students to find out whether companies are just name dropping ML or doing serious ML work. Typically, the best way is to ask former employees/interns. But I question directed at this also would not hurt.

[1] There is nothing wrong with that, but it sets the wrong expectations.

So, if I'm interviewing with a startup in AI, and asked all these questions, how many good answers I should expect? Is it ok for a startup to have a defensible business to solve a problem 10x better, knows how to make money in a big market, but have no experience in marketing and haven't talked to many potentially users?

As a wanna be startup founder, I found my ideas have bad answers to at least 2-3 of these questions.

>> 2. How was this problem being solved before the AI came around? Was the pre-AI “manual” solution good enough? Common answer: “we’re replacing humans.” That isn’t enough. Often having a human is desirable (bedside manner, dexterity, perfection a requirement). Often a human is affordable due to margin structure. You’re looking to get a sense that the product provided is something that was never possible before, 10X better, or just-as-good but 10X cheaper. Not 20% cheaper. 10X.

That "10X" sounds like a bit of a heuristic, but is it a good one? Surely, what you're looking for is some assurance that the rewards from the use of machine learning justify its cost and that the profit the company can make out of it is higher than the profit they would be (or were) making without it?

It doesn't even matter even the profit is really much higher. In business, as in war (and, er, board games) any edge you can get is enough to push ahead. If a company can make even 1% more of what it did before, thanks to some new technology or whatever other trick, then that trick is worth pursuing in earnest.

Should the engineers ask the same question when a high profile VC, like YC have already invested on the company? The questions look more like what a VC should (have) asked before funding.

Many ML/AI companies especially if funded and therefore highly visible may get acquired as the market is very hot. Isn't that good for the engineers?

You can't really ask the hard questions during the interview phase. Before you get a job offer, they have all the leverage, and people don't like being asked hard questions when they're the ones interviewing.

Best thing to do is try to get a contact inside the company, then when you get the offer, call back and start asking those questions.

Even better still - don't agree to an interview unless you know in advance that the job will move your career in a good direction, then don't even bother asking questions. It's not like they have to tell you the truth anyway.

The gotcha here is that most desirable AI startups SV wants to fund now (understandably) follow a pattern more closely described here: http://www.bradfordcross.com/blog/2017/6/13/vertical-ai-star...

Generally, their advantage is in the way they use data to solve a vertical specific problem. They more "bolt on" AI. AI isn't really the focus of most of these companies. This is just what investors want to hear about.

The article here isn't calling them out specifically, but AI infra startups are what are being discussed here as the "hammer looking for a nail" startup.

YC funded a few of them including Skymind (my company) and deepgram (mainly audio). There are a lot more of these startups such as vicarious that mainly publish papers but raise tons of funding hoping for a deepmind style exit (they didn't really have a business model).

With that context out of the way, I'll also maybe just state some learnings from the other side of the table.

Skymind has typically made money trying to reduce dependencies on cloud providers by decoupling the AI infra from google cloud and co from the cloud provider itself.

A horizontal play like this is by its nature very hard. We maintain the whole stack including our own framework. We also started in 2014 and have a decent foothold in enterprise. I wouldn't recommend trying to do this in 2017.

That being said, one other semi similar horizontal AI startup that is being started are the chip companies. Those are significantly harder to run than even what we're doing.

The most common "failed" type of startup that I think of when we think "horizontal" plays are Machine Learning as a service, which is hugely a loss leader for the various cloud providers (some companies are building on top of these providers though).

This is where you see Metamind, Alchemy API, Scaled Inference (founded in 2015), even Nervana before they sold were trying to a "nervana asic cloud" among others.

Maybe a lot of investors from SV view this as a waste of time dissecting, but I would personally love to see a bit more content on acknowledging some of these trends in the market.

The allure of "AI as a service" and the horizontal dev tools infra play is that you can try to build the next AWS similar to what the container and database companies are trying to do. Execution is definitely key for this to work though. Research also can't be the primary focus.

I won't comment on what will or won't work there

So. Much. B.S.

There's only one question they need to ask:

"How much are you going to pay me?"

Edit: The reason anyone would wax poetically about everything else is because he or she realizes that they are not being paid enough. It is like day trading - the moment it goes against a bad trader he or she calls it "investment"

Edit: Thanks for the down votes! It makes me feel warm and fuzzy inside to know that it is still possible to swindle people!

If startup: When do you need the next financing round?

While I appreciate the nature of these questions, many of them exclude network-driven consumer businesses. You know, those like Tinder, Facebook, Twitter, Medium, Yelp, FourSquare... If you're spending time with a consumer company like that, I'd focus less on LTV and more on organic growth rate.

> "Reverse engineering from the technology to the market almost never works."

Completely disagree.

If the technology is creating some value for someone, and

if you have only one smart business dude that knows how to approach the right people and validate markets,

reverse engineering from technology to market _does_ work.

Source: Did that myself.

Question 6. How often do people talk about ARPUs with potential hires? I am sure people will tell you that there are 150 million potential users but not their revenue models. It is easy to under shoot or overshoot ARPU.

Shouldn't the interviewing engineer know the answers to these business questions beforehand during their research of the company? (particularly if the company is upfront about their ML/AI-usage)

Is it different from https://en.wikipedia.org/wiki/Wisdom_of_the_crowd

Yes, quite good actually.

Unfortunately, sometimes the hype is so strong, the cart comes before the horse and gets acquired for quite a lot - which is why these companies exist.

This advice applies to all kinds of fledgling businesses, not just ML/AI.

Good questions, and they apply to many more companies than just ML/AI

This is silly. You’re the one being evaluated here. Can you code, or are you just a “sophisticated” smartass?

The trick is to find out all these answers WITHOUT asking the interviewer.

And here I was thinking these would be questions about the moral / ethical / social implications of the ways in which many companies use these technologies..