Getting started with Salt: a Puppet/Chef alternative built in Python on ZeroMQ | Hacker News Reader

Getting started with Salt: a Puppet/Chef alternative built in Python on ZeroMQ

linuxjournal.com

169 pointsjonasvp72 comments

There is also Ansible [1] which is written in Python and connects to servers using SSH (combined with sudo if needed). This means you don't need to install anything on the servers you're managing, since most distributions have Python >= 2.4 included. For more details see the requirements page [2].

[1] http://ansible.cc/

[2] http://ansible.cc/docs/gettingstarted.html#requirements

And my own tool, this time in Perl, which is thoroughly documented here:

http://www.steve.org.uk/Software/slaughter/guide/

There is also Fabric [1] to allow management of remote servers using Python scripts.

[1] http://docs.fabfile.org/en/1.5/

Fabric is a deployment tool, which makes it useful in many cases where something like Salt would be overkill, but is definitely not in the same category.

Also cuisine[0], a more fully featured configuration management tools built on top of Fabric's lower level functions.

[0]: https://github.com/sebastien/cuisine

And it can also use zeromq ;)

I like its syntax too, its simple to understand and model[1].

[1]https://github.com/ansible/ansible/tree/devel/examples/playb...

Can someone who has used both chime in? I've used ansible a bit and like it, but I've never heard of Salt before.

I'm hoping for this too. Especially somebody who has used both exhaustively. At first glance the architecture of both seems to be pretty sensible, clean, minimal and good in all the ways that puppet isn't. I half heartedly picked salt based upon the following:

* More commits in github. * More pull requests in github. * Main website is ranked higher by alexa.

However, after using salt for a short while, I ran into a couple of problems (no mercurial state, no ability to specify versions when using a pip state), which ansible seemed to be able to handle.

So, I don't know... salt seems to be more popular. A bit. That's all I can tell.

Salt does have support for Mercurial through the hg state: http://docs.saltstack.org/en/latest/ref/states/all/salt.stat...

It's also pretty easy to extend and the team who maintain Salt are really friendly and accepting of patches/pull requests.

Ansible is moving at a pretty good clip too.

We do strongly encourage squash commits so I wouldn't go by counts alone, but pick what tool you like to use most and that best fits your needs, and your style, and what you want to do with it.

Well, that's the problem...both look good to me. Similar style, technologies and philosophy behind both.

I'd really appreciate a deeper comparison, though - especially if you feel that there are certain areas/circumstances where ansible may shine over salt.

Sure, you can see some of my comments here.

http://news.ycombinator.com/item?id=5244740

Main thing on Ansible is the focus on multi-tier and process orchestration, without having to rely on coding up event paths between nodes -- admittedly it's a niche problem, but to me, deployment is as important as config management, and often harder. That being said you can write deployment in anything if you try hard enough, but I built what fit my brain and the particular needs of a few past companies deploying multi-tier web apps. While it is easy to have ad-hoc in the product (Puppet also does this with mCollective), it's a bit harder to be able to both model deployment and declarative state, and then make that into content you can hand over to a security auditor who doesn't know the language and he can say, yep, that's exactly what we want.

We're using salt and while it's incredible buggy still it's the most impressive system we've used so far. It's faster than anything else because it does not go through the process of signing in through ssh and creating a unix session.

That might sound like a boring detail but it opens up many possibilities by doing things like asking all machines for their system time and then doing something with it.

To speed up ssh login time, try adding the following to your ~/.ssh/config:

  Host *
    ControlMaster auto
    ControlPath ~/.ssh/auth/%r@%h:%p
    ControlPersist yes

For me this changes connection times from 1.7 seconds to 0.189 seconds when connecting to a host over the internet.

What does this do?

Maintains a persistent ssh connection to each host once the first is created. Any subsequent ssh commands will be sent over the existing channel, skipping the time-consuming handshake process.

Interesting tip. Thanks!

Note that you might need to create the auth subfolder using `mkdir ~/.ssh/auth`, otherwise it might throw an error on connection: `muxserver_listen bind(): No such file or directory`

I love Ansible. After having used puppet and chef at the company I work for, everyone was thrilled about how easy and boiler-plate free Ansible is.

Chef is great for bigger architectures, but if your goal is to provision a limited set of servers and document it's configuration in a readable format, Ansible is truly fantastic.

Thanks! BTW, I'm not sure limited applies. There are several setups with thousands of nodes and users doing some pretty complex multi-tier orchestration magic (N-tier, load balancers, monitoring outage windows, Jenkins, all integrated, etc). And you've got ansible-pull for mega-large scale if you want.

The thread starter meant, AFAICT, that Ansible scales down nicely. If you have 3-4 boxes, maybe 0MQ is overkill, and even running a a Chef client constantly is overkill.

Exactly. Having to install chef and all it's dependencies to install nginx and a few config files in the end feels really wasteful. With ansible, there are zero server-side dependencies.

What's misleading in these requirements is that the "Python 2.6+" spec implies support for Python 3, which the Ansible authors have actually said they won't support for ideological reasons. It would be more accurate to write "2.6/2.7".

Not ideological reasons but more so pragmatism. But sure 2.X series.

And 2.4+ nodes can be managed.

Once Python 3 is the thing everywhere I'm sure we'll have compliance. Ditto with, say, OpenStack which is also holding off. You kind of treat them as two different languages to a degree, if you still want to support things like RHEL 5 (2.4), you have to.

There's also cdist which is python and shell-script based - does not even need python on the target machine, only ssh and a posix shell.

check it out: http://www.nico.schottelius.org/software/cdist/

disclaimer: I work with the author, but not actually on the cdist codebase (not enough to brag about, anyway)

Ansible has various connection modes between the client and the server. SSH is the default, but the stock release also including a "fireball" transport that uses a ZeroMQ daemon to dispatch incoming command requests.

Salt starts off with negative marks because they decided to type the letters A-E-S in their codebase: https://github.com/saltstack/salt/blob/develop/salt/crypt.py...

Specifically, they appear to be doing AES-CBC with HMAC-SHA256. There's nothing obviously wrong with it (randomized IVs from os.urandom, both authentication and encryption and with distinct keys...) but I would hope the standard for cryptography is higher than "nothing obviously wrong with it". Specifically, I would hope that the standard for cryptography is somewhere along the lines of using a peer-reviewed cryptosystem that has stood the test of time such as TLS, OpenPGP or even NaCl (given the name, particularly the latter would be pretty good). I'm not saying TLS doesn't have flaws; I'm saying appropriately configured TLS is better than rolling your own crypto.

I'm not entirely sure why they did that, but my best guess is that ZeroMQ pretty much doesn't support TLS.

> I'm not entirely sure why they did that, but my best guess is that ZeroMQ pretty much doesn't support TLS.

Yup, ZeroMQ doesn't support TLS.

You know it's weird, I've been noodling with ZeroMQ for about a year now and I just never connected those mental dots myself. Bugger.

It's worth noting that 0MQ was developed as a library for HFT/algorithmic trading applications. In those environments, TLS isn't necessarily a requirement because it's a closed network. If you need TLS so your trading app can talk to the ticker-plant securely, you have bigger problems than you think.

Then it should not be used anywhere else.

Since it is, it really should have TLS support.

http://docs.saltstack.org/en/latest/topics/index.html

""" Salt takes advantage of a number of technologies and techniques. The networking layer is built with the excellent ZeroMQ networking library, so the Salt daemon includes a viable and transparent AMQ broker. Salt uses public keys for authentication with the master daemon, then uses faster AES encryption for payload communication; authentication and encryption are integral to Salt. Salt takes advantage of communication via msgpack, enabling fast and light network traffic. """

That explains the feature set, but it doesn't quite explain why it doesn't use TLS; TLS also starts with public key cryptosystems and then generates signing and symmetric encryption keys, oftentimes using precisely the same base algorithms (AES-256 and HMAC-SHA256).

(Also, msgpack seems completely orthogonal to this issue.)

It's not really as easy as "use X library". As another commenter mentioned - using stunnel might be better than "rolling your own" based on well tested primitives, but "just using openssl" isn't necessarily a good solution, see eg this recent discussion:

http://lists.randombit.net/pipermail/cryptography/2012-Octob...

Yes, ZeroMQ doesn't have TLS support... you should use stunnel with it if you need an encrypted tunnel.

Seems like you're essentially saying that you don't trust it because it's not a peer-reviewed crypto system using TLS. Can you see that it's flawed in its specific implementation?

This is a real community project. If you can see a way to improve something, just open a pull request on github.

> I would hope the standard for cryptography is higher than "nothing obviously wrong with it"

Can you please show me where that was the design decision that selected AES? Or are you just making an assumption that the code was not written by professional cryptographers and extensively peer reviewed?

AES-CBC is still a perfectly valid safe choice for implementations today. The most successful attacks against it, are ironically, due to OpenSSL's poor implementation. (http://eprint.iacr.org/2010/594)

The point was that they wrote their own crypto on top of (the low level AES). That is a bad idea, especially since TLS has similar properties to their system.

I don't really consider salt as an alternative to puppet/chef.

Although it can do many of the same things as both of them, there are a few important differences. 1) By default, it encourages the use of YAML definitions which makes the codebase more readable/manageable over time. 2) It incorporates a first-class remote execution environment that replaces the need for parallel ssh or ssh almost entirely. 3) It leverages zmq as a data bus, and can easily transfer files or live data over the bus in multiple different topologies. 4) It uses python instead of ruby, which to many sysadmins & devops engineers used to bash scripting is generally more readable than ruby code.

Ansible shares a lot of similar characteristics, and it's great seeing both projects taking off. Definitely room for more than one system in this area, and everyone benefits from the healthy competition. It's not terribly complicated to have different systems manage different parts of your stack anyhow. You can easily use chef/salt/ansible all at once and use each project to manage different layers of your stack.

Just don't use capistrano... :-)

> It uses python instead of ruby, which to many sysadmins & devops engineers used to bash scripting is generally more readable than ruby code.

I disagree. If anything, Ruby is closer to Bash (in a good way, mind you) whereas Python's various syntax choices (white-space sensitivity, parens for function calls, some things only available as imported modules, etc.) makes it more different.

Consider bash:

    if [ `which foo` == "/usr/bin/foo" ]; then
      do_stuff
    fi

compared to Ruby:

    if `which foo`.strip == "/usr/bin/foo"
      do_stuff
    end

compared to Python:

    import os
    if os.popen("which foo").read().strip() == "/usr/bin/foo":
      do_stuff()

Also, I would be careful about making generalizations without the data to back it up. "Many sysadmins & devops prefer..." is what Wikipedia editors would call weasel words.

I have only anecdotal evidence myself; I personally used to be a Python guy but now prefer Ruby; and some time ago my company hired a sysadmin/devops guy who came from Python, but grew to prefer Ruby once he started to learn it while working for us.

Considering that both Puppet and Chef are written in (almost entirely) Ruby, I would say that the sysadmin/devops field is currently leaning more strongly toward Ruby than Python. Although I'm sure bash dominates both.

Don't get me wrong, in terms of productivity and so forth, Python and Ruby are entirely on an equal footing.

I generally agree with your response. My only slight irritation is with how you wrote out the Python code. It looks a bit...uglier? and more verbose than it needs to be, especially compared to the others. This looks better to me:

  from commands import getoutput
  if getoutput('which bash') == '/bin/bash':
      do_stuff()

Although if your goal is to determine if e.g. bash is located in /bin, then I would do:

bash:

  if [ -e '/bin/bash' ];
  then
      do_stuff
  fi

(I don't know Ruby.)

Python:

  import os.path
  if os.path.exists('/bin/bash'):
      do_stuff()

I know I'm just splitting hairs. :)

The point was definitely not the logic of the examples! Rather to show three core similarities/differences:

* Ruby has bashisms like `foo`; Python requires a function call.

* Ruby has a nesting syntax similar to bash with if/end; Python is whitespace-sensitive.

* Ruby has method calls without parentheses, similar to bash commands, whereas Python requires them.

> Although if your goal is to determine if e.g. bash is located in /bin,

Actually, that was not the goal at all. The goal (which, again, is irrelevant to my point) was to determine if the PATH-resolved executable binary "foo" was in a specific location. Rest assured that I would not use "which" in a real-life app. :-)

(Thanks for commands.getoutput(), my Python is rusty.)

Not really the kind of comment HN generally approves, but I felt compelled to share...

> Just don't use capistrano... :-)

At one of my previous start-ups, we used to call it, "crapistrano". That is all!

I've been using Salt for about 10 months. It's pretty simple and is exceptionally fast when compared to many alternatives. Also, the code is absolutely clean, easy to read and well commented. We actually use Salt's code when we need an example of how to do python right... especially when you want to make diffs easy.

We've started using Salt for configuration management and it's everything we'd hoped for. It took some time to get our state tree figured out but now it works like a charm.

I've kicked salt's tyres recently and really like it. I think me favourite aspects are being able to use jinja and the fact that everything is all in the same place for the definitions. (As opposed to spread around various json/ruby files in chef for example.)

Salt is using its own homebrew transport encryption protocol instead of TLS (which, among other things, serializes everything as JSON o_O): https://salt.readthedocs.org/en/latest/topics/specs/salt_aut...

Why are they doing this? Good question!

ZeroMQ does not provide transport level encryption. By performing the encryption at the message level, it lets Salt publish the messages to everyone that is subscribed, but then only the minions that can decrypt the data actually run the code.

And what's the threat model for this protocol, exactly? How many cryptographers have actually studied it? Encrypted transport protocols are notoriously hard to get correct... just look at how many times TLS, SSH, and IPsec have been compromised.

Their abstract mentions little about the algorithms they're actually using. For example, they mention "RSA authentication" but not what authenticated encryption mode they're using (looking at the code, it's encrypt-then-HMAC. At least their MAC comparison function looks constant time-ish)

I don't think the people who created this have any business designing an encrypted transport protocol.

Please do explain how you expect pub/sub to work with TLS when the pub/sub part of the protocol doesn't speak TLS, only "data" ... and data is flowing only one way.

I highly recommend Salt stack. It's a great piece of software: straightforward and gets the job done. After using Puppet, Chef and cfengine, I'm glad there's salt :)

Didn't Salt build upon Func? And didn't the author of Func go on to write Ansible? My experience between Salt, Rundeck, Ansible, and Fabric was that Salt/Rundeck were too heavyweight and that between Ansible and Fabric, Ansible mopped the floor with Fabric. Just my $.02.

Yeah, I was one of the Func authors. That minion/overlord terminology, globbing, delegation, minion-to-minion, etc, are ideas clearly from there. I think mCollective was the first to take our Func-on-AMQP ideas and implement them, actually, though Salt was probably the first to try to stick a CM layer on the top of a Func-based-system, like I was planning but hadn't got around to ("Remote Rocket Surgery" was the placeholder codename for that on top of Func).

As far as Func->Ansible goes, Seth Vidal (another one of the original authors) also does a lot of commits and wrote our yum module -- and is now using it for Fedora Infrastructure.

I wouldn't really recommend anyone look at Func anymore -- it's fading just a little bit (taken over by Steve Salesvan, also ex Red Hat), but it's still in use at places like Tumblr, which is a pretty large setup. Still, it was fun to do. We sort of did multiprocessing before multiprocessing.py :)

Adoption is going pretty well at the moment and it has been fun to watch. Several setups in the couple thousand of node ranges, lots around Big Data applications.

So far I have only used salt at home in a lab environment, however it is promising. I especially love that I can dictate my directory structure for configuration by my own standards.

Salt is a promising tool for configuration management and remote execution.

But I believe it's still in it's early stages and need to go way much much more. My colleague decided upon using salt instead of chef or puppet. I'm ok with this decision but had too made some tweaks and send bug fixes or features upstream while testing environment.

I believe there may be many more to come.

Still, it's easy to use, configure and also it's still really nice on the eyes with being python. It's trivial to make fixes to the code base.

I'm currently a reasonably happy puppet user, but I do like the look of Salt. Puppet's configuration DSL always struck me as a little over-wrought (and slightly too inconsistent to remember easily). I would have preferred YAML/XML/ini files.

I complained about cfengine's syntax in a blog post a while ago[1]; someone mentioned that a patch for YAML-based cfengine manifests had been proposed. I haven't seen anything since, but then ... well I don't follow the cfengine news too closely.

It largely seems to be that a lot of these systems take their directives as nestable key-value pairs, so I think YAML is a good fit.

Amusingly, the first link someone posted on my blog post was to Salt. I didn't dig too deep at the time as it appeared to be more at the REF end of the spectrum to me.

[1] http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign...

If you want YAML, check out ansible ( http://ansible.cc/ )

ansible has very readable concise YAML syntax[1]. Plus, no bootstrapping required (its optional, haven't tried it though) and works off ssh by default (also has a 0mq mode).

[1] https://github.com/ansible/ansible/tree/devel/examples/playb...

Met the guys behind Salt recently, and they seem pretty nice and host the local Python meetup. I'd like to try to use it some time, but the docs seem a bit on the impregnable side; while it seems they have decent coverage, there isn't a lot to really glue it all together and from just a quick skim, it seems like it'd be a pain to start from those. They should really invest in some decent "Getting Started"/tutorial blocks and attempt to explain the concepts narratively. I noticed perhaps a bit of this, but as I said, on a quick skim, it seemed quite insufficient.

Also there is cdist (http://www.nico.schottelius.org/software/cdist/) not so popular but interesting.

It seems that cdist recipes must be written as shell scripts.

That is absolutely right - and thus your sysadmins do not need to learn a new language / DSL. cdist is mostly about being simple

Just when I narrowed my options down to ansible and chef…

This seems to be more in line with chef/puppet, with a dedicated daemon on the client.

It's more similar to ansible than puppet. Both are python based and use zeromq. All four are declarative and AFAIK, all four have a dedicated daemon on the client.

I would rather not get into "X is better than Y", but this is not quite accurate.

I'm not saying that's a bad thing, but I see a distilled Puppet syntax there, where Ansible (http://ansible.cc) is more about modeling both resource states and ordered processes across machines.

Ansible actually uses SSH by default but there's a 0mq accelerator if you want to use it. We don't just execute individual SSH commands, modules are remote resources that model idempotent state -- but don't stay resident. The 0mq accelerator came later. There's also a pull mode, and a local mode in Ansible. Using SSH is nice because you don't have the additional PKI, and all that comes with having to maintain that. (No needs of accurate NTP and DNS either, and working with Kerberized SSH and using your OS credentials is possible). Need to do non-root things? You can do things (if you have permissions) entirely as non-root.

Furthermore, Ansible does not have a dedicated daemon on the client. It's daemonless, and consumes no resources when not running. When running with 0mq, the daemon is ephemeral, stopping after a set period of time, to not consume resources.

It also requires no bootstrapping, so if you have blank cloud images with nothing on them, or brownfield systems in the field, you can just start managing them immediately. cloud-init and so on are great for this, all cloud tools can inject keys. For this reason (no agents), it's starting to look very appealing on embedded devices and network hardware as well.

While implemented in Python you can write modules in any language. Finally, Ansible's core focus is around making multi-tier orchestration easy (somewhat of a niche case), so that leads it to look a lot different at the language/resource level. But if you have just one tier, you can use it too.

It's frequently used to deal with rolling updates involving N-tier applications with monitoring servers and load balancers all in play, though you can definitely use Ansible just for classical config management.

I think part of the reason Salt may look similar to Ansible is Salt is clearly inspired by Func, especially the minion/overlord terminology, globbing, minion-to-minion, etc, and I was one of the original authors of Func at Red Hat. We were toying around with a message bus Func at the time and decided not to do it.

So, yeah, lots of tools in this space.

In Ruby config land, there's also Sprinkle, Rexify, etc! So many options!

>In Ruby config land, there's also Sprinkle, Rexify, etc! So many options!

The Rexify (or more precisely Rex) that I'm aware of is written in Perl - http://rexify.org/ | https://github.com/krimdomu/Rex

Ansible's ZeroMQ is an optional extra feature - out-of-the-box it works fine without a daemon on the client, as it just uses SSH.

It's the same for salt. Neither one has to use ZeroMQ, neither one has to use a daemon.

What remote transports does it have that are daemonless?

Is there a search functionality, similar to what Chef::Search provides ?

Why?

Reply on news.ycombinator.com