Hacker News Reader

I recommend Rancher for the small business that wants to roll their own Kubernetes for free. Rancher 2 is a bit underdocumented right now, but we had great success with Rancher 1.

Your bare metal hosts can be provisioned as Rancher hosts, which are dumb slaves managed by a Rancher server.

From the Rancher server, you can trivially express "I want this Docker container deployed across n Rancher hosts" or "I want 1 instance of this docker container across all Rancher hosts with the tag "has public IP".

You can group your Rancher hosts into Sandbox and Prod Rancher environments. You can easily install Rancher's load balancer service on them, or mount network storage, or register secrets upon them (like private Docker registry keys).

It also gives you health checks, host monitoring, and zero downtime redeploys. Super easy to use from the UI or CLI. Easy to install, too.

I've also been testing out Rancher recently and I got way further in my testing of it than I did in my test deploy of a kubernetes cluster using the install docs from Kubernetes. I got closer with kubespray, but had to put it down to work on some other things before I got anything useful set up. I hope to be getting back to my Rancher test this week.

I spent a lot of time setting up a private docker registry, before I realized that I probably could have used one from AWS.

I ran into an issue with using custom registries that really stuck me for a day or two, I put details of how to fix it here: https://github.com/rancher/rancher/issues/13676

I really like the look of rancher but the thought of debugging some issues with it scares the pants off me.

Since they don't have a pricing page (have to ask for a quote). In what price range are they in?

Rancher is free and open source. Their Docker images are publicly available. I assume Rancher Labs sell support, but we did not buy support, nor did we need any.

Thanks for the info.

Rancher 1 is superb. After testing Rancher 2 for a couple of weeks I feel sad that they spent time on developing this instead of working on enhancing Rancher 1. I can't seem to find what I can do with Rancher 2 admin panel that I cannot do with what is shipped with Kubernetes. It actually feels more limited and buggy. After those couple of weeks of studying documenations and blog posts I gave up. I was not able to setup any app. Rancher 1 works like a charm. I would have used it for my new projects, but since they are not likely going to develop it I might just go with stock Kubernetes.

I currently use kubeadm for our small business cluster. 1 master and 2 nodes. We haven't put a ton into it but it seems to be running pretty well so far. Have done an upgrade from 1.7 to 1.9 without much/any downtime. Is there much more of a learning curve with Rancher?

Rancher 1 is extremely easy to learn. Generally you do everything from the UI (I only use the CLI to view combined logs). High-availability is easy too; they provide a load-balancer (haproxy) with good UI integration.

Rancher 2 has a nicer UI and is more tightly related to Kubernetes. But there are moreorless zero docs, so be careful.

With either version: it's trivial to grab the master and node Docker images, and deploy them to your local machine to have a play.

If you're going the kube route, I would go with GKE with pre-emptible nodes. You can have a single micro base node ($25/month), and then an auto-scaling, pre-emptible pool for a discount of 80%/month vs spot prices on AWS/GCP. There's not a lot for you to do once you set it up. With pre-emptible the nodes will on average be unavailable 5-10% of the month, but typically not at the same time.

There are cheaper baseload alternatives appearing, but it'll probably be 12 months until they're stable.

https://labs.ovh.com/kubernetes-k8s

https://github.com/hetznercloud/hcloud-cloud-controller-mana...

This. GKE is rediculous how easy it is to spin up a cluster. 60 seconds max. Why fight setting up K8S?

Running K8S on other environments isn't a difficult job, but you do need a certain level of experience running systems. You need to set up EtcD, in prod you have 3 or 5 of them - there are a couple of topologies for running EtcD and peer discovery (you will need to RTFM). This is all your K8S state, so you need to set up backups for these.

Each node you are going to include in your cluster needs to have docker and kubelet running. You will need to start kubelet with your init system (systemd, etc).

Then all you need is to bootstrap a control plane. The control plane has 3 components. The API server needs to communicate to etcd, everything else just talks to the API server. The API server is stateless and can (should) have multiple instances running. The other two parts are the Controller and Scheduler. Also stateless. The tool Bootkube[1] can generate all the configuration and perform the bootstrapping for you.

To answer your dot points, all but 4 are easy enough with using the kubernetes docs and some systems/networking knowledge. Point 4, assuming you mean starting new machines based on load, will require you to look at which systems are supported by the autoscaling system and potentially add an integration for your environment. There is the horizontal pod autoscaler that will run more of your service when needed and the cluster autoscaler[2] which will start more machines when it can't run as more instances of your service on the current number of machines.

Edit: There are also tools like Patroni, Stolon, and postgres operators that will assist with your DB (postgres in this case) management. Scaling, HA, backups, etc.

1 - https://github.com/kubernetes-incubator/bootkube

2 - https://github.com/kubernetes/autoscaler/blob/master/cluster...

just as a sidenote... you don't actually need to know anything anymore to get a working kubernetes install on your own hardware.

all required services will get automatically deployed into kubernetes after the initiation with kubeadm.

that knowledge would still be useful if you're plannign on going into production with a kubernetes setup.

- It's certainly possible to run Kubernetes on bare metal. We do it. I recommend kubespray to set things up, there are a bunch of other ways.

The two big things you get on the cloud that you need plugins for on bare metal are networking and storage. Networking plugins are basically fire&forget these days. Storage is definitely not at that stage yet. You can use a plugin that provides redundancy & failover like Rook/cephfs or Heketi/gluster or you can use local storage and handle redundancy/failover similar to how you'd do it without Kubernetes.

- Kubernetes is like any other abstraction. It complicates things and adds another layer to manage and understand, but it lets you manage disparate server apps similarly. So it really only makes sense if you have a good number of disparate server apps to manage.

- If you're looking for 5 nines or something like that, as a small operator you're more likely to screw things up through a PBKAC or misconfiguration than you are via a hardware failure. So you probably want two clusters with manual failover between them. That way you can switch to one cluster while you operate on the other. The second cluster could be a GKE cluster scaled down, but that's still a bunch of complexity and some expense. You're talking about zonal failures, so you'd need two clusters to protect against that anyways, so...

- you probably need several layers of backups. An etcd backup, ark, and your persistent storage backups. Ark handles persistent storage backups on the cloud, but if you're on bare metal, you need to do that yourself the old fashioned way.

It's coming to DO! https://www.digitalocean.com/products/kubernetes/

They said sometime in June I thought, and we're nearing the end without any communication. Wonder if they’ve hit a snag.

Hey there! I'm the product manager working on the DigitalOcean Kubernetes product and early access program. We have updated our announcement blog post with some new information (at the bottom). We will be opening up early access phase 1 on July 10th. https://blog.digitalocean.com/introducing-digitalocean-kuber...

Never got an email about it, so, my point still stands. I don't look back to blog posts expecting updates.

>what if the whole zone fails and all DB containers are shut off?

I agree that the container clustering world tends to focus on stateless scenarios, conveniently ignoring the "hard stuff". However, that doesn't mean it can't be done. k8s can be told to maintain persistent volumes that can be claimed by containers. This mechanism can be (has been) used to implement "real" DB HA. e.g. see this talk : https://www.youtube.com/watch?v=Zn1vd7sQ_bc

I haven't used it, but Rancher can be used to deploy Kubernetes on your own infrastructure

https://rancher.com/what-is-rancher/what-rancher-adds-to-kub...

DigitalOcean is also working on a managed kubernetes service, which should be less expensive than AWS,GCP,etc

> DigitalOcean is also working on a managed kubernetes service, which should be less expensive than AWS,GCP,etc

What makes you say this? A 4 GB RAM VPS on DigitalOcean seems to have a very similar price to the corresponding GCP VPS. The load balancer offerings show similar prices too.

Getting all of the mentioned requires a real cloud provider which offers more than just servers. Thus you need to choose among the cloud providers. Digitalocean is another provider offering a managed k8s. GCE gives you 600 usd credit - this is valuable if you want to learn new tech.

I actually switched all our servers from hetzner to aws because for our needs it was cheaper and I could spend my time on things I care about. For me K8s only handles non persisting data. All data other is stored outside (Managed DB (like RDS) or S3) of it. You only need to backup the masters. If you use something like GCE that is managed for you & free.

> K8S is only supported on expensive hosts (AWS, GCE, Azure), is it possible to spin it up on hetzner/ovh or another provider that is cheaper than AWS, possibly even on my own hardware?

Of course. There are literally dozens of distributions and repackaged versions now[0]. The advantage of GKE et al is that they keep them up to date -- still a bit harder than it looks.

> Be cheap

It very much depends on what "cheap" means and what your needs are. GKE with a cluster autoscaler is fairly inexpensive for what you get. But if your time is free and you like to tinker, self-managing on a VPS might be for you.

Be prepared to invest a fair amount of time. Kubernetes is simple in each part, but there are lots of parts to learn.

> Automatic failover (This should cover majority of downtime imho)

This is more of an app concern. Kubernetes itself won't really do this for you. Something like Istio will probably also be necessary to help with traffic management.

> Backups

Very much up to the distribution / hosting environment.

> Automatic scalability (CPU over 95% for last 2 minutes or smth like that? add new containers pls)

Horizontal Pod Autoscaler; a Vertical Pod Autoscaler is under development. Other autoscalers are being developed for more specialised uses (I've worked on one) and I expect there will be some concept of pluggable autoscaling soon.

> Load balancing

Numerous solutions in this area. Typically left to the IaaS. Istio can do this.

> How are K8S backups handled? ... several containers spawned of the DB

Regarding DB backups: I would again use a hosted service if at all possible. Several containers can share a volume in a Pod or across Pods. But long-running stateful services is not yet quite the Kubernetes sweet spot.

> What restarts K8S orchestrator if it fails?

Again, numerous solutions. I work for Pivotal, our solution in PKS is to run everything with BOSH. Something like GKE uses Google's own monitoring and recovery systems.

I should add here that I don't think GKE is necessarily the "best" of the public hosted options. Just the only one I've used in anger so far. I actually prefer to do development work on minikube whenever I can. If you go down that route, shop around!

[0] https://kubernetes.io/partners/

It depends on how you define small business. You can have a SaaS app with a lot of users and revenue, and a small team and a couple servers and maybe not benefit from k8s.

Or you can have a startup with a small team and small number of users, but the legal and market requirements may dictate things like availability requirements for which k8s can add a lot of value that would normally cost you hiring multiple ops folks and writing a lot of boilerplate code.

For the latter kind of business, k8s is probably a good investment that is worth the added costs of running on a more expensive cloud provider. You’ll probably have to stick to them for security/compliance reasons anyway.

For the former type of company... k8s is probably only worth chasing if you are already seeing scale, or if you see becoming more like the latter. In that case, I would first test my app on something like GKE to see what it means from that perspective. Then I would look at deploying on my preferred hosting provider.

If you can’t get your app to run well/how you like on GKE, it’s probably not worth planning a cluster deployment just to find out the same.

1) If you're willing to do some custom setup, it's quite easy to spin up Kubernetes on bare metal or an OVH VPS using kubespray or kubeadm.

3)IMHO even for small setups because you buy into a <pseudo standard> ecosystem. Backups can be handled for example by stash[0] instead of having to (partially) write your own backup solution.

4)I haven't tried this on OVH, but as it's a openstack based environment certain items are well integrated e.g. persistent volume handling [1]

[0]https://github.com/appscode/stash [1]https://kubernetes.io/docs/concepts/storage/storage-classes/...

I'm a big fan of the Google App Engine standard runtime. It's not too cheap if you have lots of traffic, but a lot can be squeezed out of the free tier, and it's very opinionated, but, for proof of concept, it's great.

My wife's company website ran on it for years.

Shameless plug: https://www.juusohaavisto.com/microservices.html

To answer your question briefly: Kubernetes is hard to stay up-to-date, but it aims to improve later on. Kubernetes was largely created to solve problems scaling physical hardware -- I would not setup Kubernetes unless you run into a problem where you are growing out of dedicated host offerings thus need to update your apps across multiple such devices (easier options exist, look at LXC clustering). You need to define "reliable". If it is just your app or hardware crashing, you can probably get away with two dedicated computers in different colocations.

Kontena Pharos (https://github.com/kontena/pharos-cluster) aims to make installation of Kubernetes as easy as possible. Works on any cloud or on-premises, also ARM64 supported.

It handles HA-setup, etcd member replacements, hardened configuration for enhanced security (follows NIST SP800-190 recommendations). You can easily extend the installation with addons of your own or from any Kubernetes ecosystem project

(Disclaimer: I'm one of the contributors)

FWIW I've found MAAS and Juju the easiest way to get a Kubernetes cluster up and running on bare metal - I've not tried every other method though so YMMV :)

Kubespray is a winner for me.

Yes it is.

I do it with kubeadm and metallb (for my public ip) I'm not sure how good ip failover will work on hetzner/ovh, but on your own hardware in your own network it is not a problem.

(K8s does not need backups, only your etcd and configs should be backupped but it is unlikely that you loose your whole control plane in one step.)

You could run Kubernetes on a Raspberry Pi cluster using the Hypriot distribution.

Docker Swarm - you have 80% of the power of kubernetes and 10x simpler.