ELK stack deployment with Ansible
blog.comperiosearch.com
blog.comperiosearch.com
I get the feeling that "full Configuration/Deployment" tools like Ansible (Puppet, Chef and Salt) are getting eclipsed by tools like Docker.
For me Ansible was just a "half-way" station. 1. Manual software installs 2. Script assisted software installs 3. (some) server Configuration/Deployment with Ansible 4. Docker (Rkt, appc) 5. ??
Is this a fair assume that tools like Ansible are on the way back out? Or should you keep it to configure the Docker hosts?
I work with this every day, Ansible, ELK, other systems. I have no need for docker. We have a large vSphere environment where I can deploy VMs through API on demand and I can scale designs with ease.
You might need Docker containers, but only you can say if you need Docker for your deployment.
Docker, assuming you don't basterdize it, gives you a largely immutable image that will actually be the same between your environments. That's about as close as guaranteeing it'll work as you'll get.
Does your Ansible setup provide that? I suspect not.
Docker makes it pretty easy for you to run multiple apps with conflicting libraries on the same host. Does Ansible help you do that? Nope. You could make it do it but it hardly makes it any easier.
I'm not saying Docker is some sort of panacea. Actually I think it's been a buggy mess. But I do believe it or something like it is part of the future we want.
Wrapping an app in docker and throwing it at compute running a distributed scheduler just simply feels like a better solution than writing an ansible script that places it somewhere and installs all the libaries then forces you to figure out how you deal with failure.
Obviously Docker is just a small part of a bigger picture. You could quite happily use Ansible to help yourself deliver them dockers.
My Chef and Packer one does. The deltas between a Vagrant image and an AWS image are exactly the ones I know to be there and are there because the environments are different. Those minimal differences exist because these environments are fundamentally different and I need to be mindful of them when operating at scale--something that Docker will occlude but never, by its nature, solve any better.
The point was, start with docker, get that for mostly free. You can feel free to further manage the underlying hosts with chef, puppet, or ansible, but docker will make the systems interface to your app quite a bit neater and you'll still chef to configure the app.
Your CM now does less, which is a good thing.
Throw in mesos or kubernetes because your CM isn't really amazingly helpful at deciding where things should run and your CM will now do even less.
That doesn't mean your CM will definitely go in the bin, but if all it's doing is changing a couple of system settings during an image build, it arguably could be bash, though I personally would rather it wasn't.
First, Docker seems to be still quite a young technology, and you will probably need some sort of orchestration on top of it.
Depending on what would you need to solve, ansible scripts might be order of magnitude simpler.
In some instances you won't be able to replace your ansible (or chef) scripts \w docke no matter how hard you try (i.e, we have iOS build farm on OS X servers, and we are attempting to build some windows selenium grid nodes, e.t.c)
On the other side of this spectrum, there still might be many thins where you are better of using Heroku, or some other battle-tested paas.
The way i see this is that Docker helps me manage /usr - i can make a Docker image with all the binaries, libraries, and static data files that my application needs, then send that somewhere to be executed.
But Ansible etc help me manage /etc - i can create configuration on a machine that is appropriate to the environment (should this use the real or test payment gateway?), the purpose i'm using the application for (is this nginx container serving static files, or reverse-proxying an app server?), the current state of the rest of my infrastructure (what's the IP of the metrics server today?), etc.
Configuration is the stuff which needs to be different on different machines. It doesn't make sense to bake configuration into a Docker image, and Docker doesn't give you useful tools for handling configuration any other way.
Ansible etc can also be used to manage /usr, of course. But at the moment it looks like this is not the best way to do it.
Now, if someone could come up with something to help me manage /var, that would be great!
Configuration Management may help to set non default permissions under /var/ paths, or to check for default permissions status, or prepare business logic automated actions over that variable data.
Right. And 'Docker for /var' would be something that made all that easier. It would make it really easy to set up live, secure, restorable, backups of essential data; to rotate, compress, and delete logs; to extract all important logs to a separate host, and aggregate and index them, etc. At the moment, a configuration management tool can help you set this up, but you still have to write the configuration management script by hand.
We need to increase a sysctl value on our systems with role X. As fast as possible. How does docker help there?
We need to run a set of given commands for a security audit of our systems. We need to execute some actions conditionally by role on the infrastructure. etc... No Docker help there.
We need to know the credentials accepted by container "foo" on a given past day of the year. Do the report.
Pretty sure lot of (experienced) people has seen problematic and ugly deployments.
We've seen worse and better things, done with put_a_name_here technology. No mater what name you put in the sentence. Which programming language, which stack, which integration... no mater... at all.
Don't let you go by "magic" tools. There is no magic in Docker, just money behind so you are forced to listen about it, think it's better, etc. That's it.
It's good at what it is. And one of that things, is money. It's better solution than others available, because it has been widespread, even between those avoiding it, because it's supported on major providers, etc, etc, etc. Industry. People that does not even know, neither never did use directly, the technologies below Docker, is talking about it, with big ego.
It does not solve all the needs of a devops, startup or company. Neither the new $tool to appear on viral ways tomorrow, will do. Be sure.
Have a nice day.
For example, ssh keys: Kubernetes has a concept of "secret volumes" that are used to distribute secrets. i.e. don't store mutable (and secret) state in the container.
What we're excited about is using it as a building block in an internal PaaS so that individual application teams have a Heroku-like interface for deployment. Hosts are cattle (and managed by a traditional CM tool), containers are immutable and do not even have shells installed, etc. We don't have mature tools for orchestration, scheduling, networking, service discovery, etc. yet but this approach is what all the hype is about and much of the software popping up to fill that niche, i.e. Mesosphere and Kubernetes, is really interesting.
Machines having roles, engineer SSH keys being on machines, containers being logged into, etc. - in attacking these things you are attacking a strawman.
Docker is pushing CM out to the individual applications, but the tooling just isn't there yet for it. I suspect an area of major growth and change is going to be around managing container builds and portfolios. The way we are doing it now is not the way we will be doing it in a year or two. We are in the pre-puppet/chef days.
And that doesn't even touch on orchestration, which something like Ansible or Salt can help out with. Ultimately docker doesn't even really attempt to address any CM problems, and in a lot of ways it makes the challenges even harder.
The concerns you raised are exactly the areas we struggled in when we moved to Docker. Our newer apps tend to be written with containerization in mind, though we also wanted and are porting older/legacy apps to take advantage of containers.
1. I somewhat liked 12factor's env-vars-as-config for its simplicity and ease of use but there are limits to its flat key-value string-only structure, not to mention the debatable security concerns that may arise when storing secrets in them.
2. Some apps, esp legacy ones, need files for their config, especially those that are costly/harder to rewrite/reintegrate. Config files benefit from being easily versioned, can be templated (Ansible's jinja ftw!), not to mention the richness of syntax/structure in can give (xml/yml/json/bin/etc).
3. We sometimes do a hybrid approach of doing a "source envfile.sh && exec binscript" as entrypoint of our containers, with envfile living in a mounted volume, managed by a CM tool.
4. There are lots of innovation here in specialized tools such as confd/etcd/consul/skydns but completely agree that versioning is a real concern. I like the heroku/deis-style approach of creating a new release for each change in the config. Bonus points to easy rollback on a previously working config (whatever system you're using, be it env files or config files).
5. As with projects/whole-stacks of multiple containers/hosts, I like the rancher/maestro-ng/docker-compose style of managing your system stack. Next wishlist for this would be a stack with support for pinned versioned docker images + smart rollback/rollfwd support for each config change anywhere in the stack.
Ansible filled this gap for us: we have a git repo per project, various playbooks for deployment/lifecycle-management of docker containers, with multiple environments supported (dev/qa/prod using same playbook), support for multiple maintainers and secrets management via ansible-vault. This is workable for us on a few projects, with small scale/set of hosts/containers.
Containers make sense for applications or processing but if you have to tie a database to a host you lose a lot of the container benefits IMHO.
Config management tools will be around for a while, if nothing else you need to configure those docker hosts.
I think the big win if we get there will be using Docker to provide images and portability, chef (or puppet) for configuration, and something like Karamel to orchestrate and provide concise cluster definitions with GitHub as the artifact server.
The linked project uses CM tools to get ELK running in a vagrant image. That alone is not terribly interesting - because I probably want to run it on my production cluster - not some standalone image.
I can edit the configuration - but this introduces dependencies on my particular environment (is it AWS, GCE, VMWare, bare metal, etc.)?
What I find compelling about Kubernetes (which to my mind is much more interesting than Docker alone) - is that it attempts to abstract at a much higher level. In theory (yet to be proved in battle...) I can deploy a k8 configuration on any number of environments, and the "description" ( a bunch of k8 yaml files), is the same.
Is Docker being hyped beyond belief? Yes, but I do think there is something significant about containerization.
Does anyone have any ideas here? Has anyone ventured down this path?