Netflix VP of IT on the Future of Infrastructure
amplifypartners.com
amplifypartners.com
No, it's the business continuity way of thinking. Outsourcing commodities -- such as servers, virtual or otherwise -- is one thing.
Outsourcing your core operational tools, software, and all your data is another matter entirely. Preferring SaaS at a company large enough to afford on-premise solutions is just nonsensical, and I expect it'll either blow up in his face, or just create a never-ending tax on end users who are constantly dealing with a mishmash of vendors, accounts, disappearing services, broken software, and instability.
At scale, stability and continuity is worth more than the opex/capex costs of internal IT.
I didn't get that he was talking about moving to cloud/SaaS vendors as a whole, but more the IaaS/PaaS space - the "Hey, this app runs RoR, can I just run it from a Heroku dyno rather than a VMWare box in a cage" type of move. Granted, there's still a cost associated with that, and a need to plan the move, but it's less of an issue. They are making heavy use of SAML SSO it appears, so the account/vendor thing probably wouldn't be as much of an issue.
You might get better response times knowing it's a local problem vs let's say an office in NYC hosting your ticketing system. Regardless, I'm sure they have put a lot of thought into this set up and to me sounds amazing. No ties to a physical location = win.
Yes and no. First off, I don't think Netflix can afford to build the kind of infrastructure they're using -- certainly not without changing focus of their engineering resources. They famously do a lot of work to move data closer to the end users globally -- they're not "just" a US company. Essentially, they'd have to have an operation that would be "qualitatively" similar to AWS in order to be able to do what they do (and at a smaller scale overall, I think that would end up being quite expensive).
I'd argue Netflix is one of the few companies I can think of where this "all cloud all the time" idea for infrastructure might actually make sense. I agree it's a big risk though -- and probably not good advice for most companies.
It would also appear that Netflix is planning on actually selling a product (video rental) and make money off that, rather than have that merely as a vehicle to drive other, sometimes tangential, innovation. That remains to be seen, of course.
They do that to themselves so that when it actually happens, they are prepared and the end user (almost) never sees it.
Its a cost benefit analysis issue.
Netflix took a very early stance that they wanted a large portion of IT ops pushed out of the company.
One still needs to provide the means to supporting the actual tools which are required to the core business continuity -- but the industry has matured a great deal in the last ten years to the point where you have multi-billion dollar enterprises where their whole core business is that singular department that Netflix has chosen to push out of their management purview via outsourcing that cost....
This is not a bad business decision - the AWSs DOs etc have commoditized all the portions of that previously in-house IT department in a way which everyone can benefit.
The risk is in the last mile.
The most needed disruption in the tech-world today is in the carriers in their current incarnation. THEY MUST DIE.
Anyway care to expand on some of the less Googleable acronyms?
- MDM/MAM
- NAC
- EDW (synonymous with ETL?)
NAC = network access control
EDW = enterprise data warehouse (archiving old information while preserving access)
The last "innovation" I'm aware of in this area, is skolelinux/edulinux work with packaging samba/ldap/kerberos/lts in a easy(ier) to manage package for Debian:
https://wiki.debian.org/DebianEdu/Documentation/Wheezy/Archi...
the problem is having tools that communicate with each others and an easy setup.
yeah, SAML kinda sucks to use too.. and works like kerberos anyway. OpenID, Hawk, etc - also in fact work exactly the same.
As mentioned up-thread MS AD does a great job of enabling in-house CA and management -- and it's mostly that I want. I want to use certs for auth most places, and I want it easy! Openssh have shown that public key auth doesn't have to be hard -- but also doesn't have a very compelling story around managing access. The new cert-system might be an improvement -- but it absolutely needs some infrastructure around it to be easy to deploy (and verify).
if it changes, it warns you.. but in most cases you're going to know why it changed or just accept the change anyway (which is a problem when you admin 10000 servers of course as the warning might be a real issue)
central trust/revocation is still an issue everywhere to this day, i think. both technically ("my client trusts this, but do i?") and from the useability pov.
Well, yes and no. Do you mean the new cert stuff that's in standard openssh? Which has stuff like:
The marker is optional, but if it is present then it must be one of “@cert-authority”, to indicate that the
line contains a certification authority (CA) key, or “@revoked”, to indicate that the key contained on the line
is revoked and must not ever be accepted. Only one marker should be used on a key line.
Or did you mean the x509 patch?
I do indeed like kerberos - but I still want a straight forward and reasonably robust framework built on certificates.
Central auth, auto certificate deployment, encryption by default. The nice bits are Windows only unfortunately.
Even if I was willing to introduce a w2k8 server -- it's hardly trivial to integrate across infrastructure. Eg: set up client auth for ssh in such a way that online verification of certs against a list of cancelled certs works -- and that there are no other ways to authenticate to ssh servers.
I absolutely agree that AD is one of the best things MS ever rolled out -- it's unfortunate they a) broke (or bent) some standards when doing it, and b) just like .net and sql server are great platforms, they're not for me (any more) -- I'd much rather play in an open environment. Mostly so I'm not dependent on a single entity for continuation of services and development.
I know RedHat have their directory server, and Samba4 has basically copied some of the architecture from AD (roll up LDAP, cifs, kerberos all in an integrated set of services) -- and that's great. I'd still like to see a single open design that actually works (and that last bit means it needs to be tested across heterogeneous environments).
I don't think such a system would actually be too hard to implement these days, we have a lot of great components that just need to be fit together and "blessed" with some rigorous packaging and documentation. Perhaps the "best" way would be to wrap kerberos principal key exchange in a public key transport of some sort (but at that point you'd really only be using kerberos for backwards comparability, you'd have moved the trust and authentication implicitly to your CA infrastructure (possibly with a low lifetime of service tickets) -- which could be good or bad depending on your point of view).
Basically what I want, is to have a way to throw a (most likely private) CA-cert on a new box, and then have that box request a cert via on-line csr to a gateway -- that gateway should then be able to forward the csr to the CA (which for high security setups should be air-gapped, for most settings might be a daemon running on the same box). Then once machine certs are set up, probably use service-certs for services (if this sounds a lot like kerberos, that's not an accident) -- or just assume one service per (virtual) machine.
For users we'd need something similar, and we'd need a working online check for validity that defaults to disallow, probably with some caching for local login on laptops/workstations to be able to to some* authentication even when offline (obviously configurable, depending on use-case).
After the years of attack on kerberos (among others) I think many of the risks are well understood -- the challenge is just to build something that is simple enough, but yet works. Dictate a single format for certs, possibly a very limited set of algorithms (but history seems to indicate that some sort of visioning is needed, maybe explicit "valid sets" rather than open negotiation?).
Anyway, sorry for the post, probably should've been a blog post :-)
I think it's amazing the decisions that get made with explosive growth/hiring that end-up on roadmaps that read similarly to organizations that have been around much longer.
There's no criticism here. I think Netflix is an amazing company and it is the this sort of strategic vision (and the openess of both it and the organization overall) that reminds me that we're all on this rocky ship together and it's amazing that any of it works sometimes.
[1] http://www.slideshare.net/mdkail/it-ops-2014-technology-road...
Generally in smaller software companies I hear R+D and consumer-facing applications referred to as "engineering" with external-facing infrastructure (like the production datacenter) referred to as "operations," with "IT" being reserved for this internal backoffice kind of stuff.
In other places, especially larger corporations, I've often heard everything having to do with a computer lumped in as "IT."
> We are implementing “certificate-based authentication” instead of the standard username/password auth against Active Directory.
I wish we were all doing this. How long is it going to take to get a usable certificate-based client/user authentication mechanism on the web?
edit: Also see e12e's comment.
Netflix talks generally can be fascinating and inspiring. However, when considering IT it's also important to consider the charter and challenges of Netflix IT.
i.e. it's no more valid or invalid that the talks of how IT is delivered in so-called build vs. broker models in other companies in other industries http://dilbert.com/strips/comic/2013-07-05/
Requiring VPN everywhere, how is that cloudy?
Finally, using stuff like AWS is nice, but unless they have a specific contract (which they may since they advertise them a lot), its a LOT more expensive when you start having a lot of processing (ie big companies like netflix)
Keep in mind I'm an idiot, and I have idiot friends.
This makes me think of http://meldium.com