Fabric Python with Cleaner API and Parallel Deployment | Hacker News Reader

Fabric Python with Cleaner API and Parallel Deployment

tav.espians.com

183 pointstav32 comments

Fabric maintainer here -- Tav's been quite active on our IRC channel while developing much of this, and it's great to have more people excited about improving the tool!

As tswicegood says in another comment, some of these changes (@task, parallelism, etc) aren't unique to Tav's work, so when it comes time to merge we'll draw on the various implementations to see what feels best.

The features unique to his fork will be evaluated like any other contribution and merged in if they fit well and are generally useful (which many seem to be).

The plan right now is post-1.0 (1.0 is scheduled for Pycon or earlier) to round up a lot of major contributions and get them into core, so people like tswicegood don't have to work off of their own forks quite as much :)

Another cool trick: Integrating Cloudkick tags with Fabric. Using the Cloudkick API - you can express things like "run certain_func on everything tagged 'loadbalancer'".

https://support.cloudkick.com/Integrating_Fabric_with_Cloudk...

We dog food this for all of our production deployments. For example, when we ship a new version of cloudkick.com, we just run "fab ship", which looks for all servers tagged "cloudkick.com" and runs the appropriate deployment script on each of them.

Tagging can be done through the CK UI, automatically with the agent, or with things like puppet or chef. This allows you to fully automate deployment and other tasks around groups of servers.

I figured out something like that using the Python Linode API last week: http://asktherelic.com/2011/02/17/dynamic-fabric-commands-fo...

While I like a lot of the stuff you have incorporated into your fabric, it seems to me that you are trying to take credit for things that have existed as branches to the main codebase of fabric for quite a while. Just off the top of my head: task decorators, parallel execution, better host management and command-line task listing all exist in other people's patches. If you have redone the code, good for you, but at least mention that the ideas and prototype implementations aren't yours. (I don't think a claim to ignorance of such is really that valid either, since the Redmine for the project has tickets for most of the things you've done, and has links to the branches containing them).

As to getting stuff into fabric: a very valid complaint (and one I have myself, since like a lot of people I run my own hacked version of fabric) is that fabric development moves very very slowly. There are several different feature branches in several people's code-bases that could have been released 6 or more months ago. Perhaps if you start pulling other patches that have a lot of functionality complementary to yours, you could have a product good enough to push development on the mainline too (ala gcc/egcs of old...)

At this point I'm pretty sure I've learned my lesson about release timing (I've also signed on with a job which should give me more time to work on Fabric, since they use it daily). Definitely deserve all the flak I get for being slow, though.

Let me get this right. Due to convergent needs, dozens of Fabric users post various ideas to the issue tracker and mailing lists. And over the years, some people hack together personal branches solving some of these issues and, despite the incredible amount of time that's passed, they fail to get their work into the mainline branch.

Eventually, one hacker comes along — who, having been a Fabric user for a long time, has independently experienced similar needs and finding no decent solution, decides to take on the task of implementing the various features himself — making sure that it's clean, backwards compatible and fully documented. And your response is to accuse him of trying to take credit for other people's work?!

I can understand your frustrations with the pace of Fabric development, but accusing another hacker of taking credit for other people's work simply beggars belief. Even more incredible is that so many people decided to upvote you! For the record, I'm not taking credit for anyone else's work. I came up with the various ideas myself and you can look at the code to see that it's not based on anyone else's work. Thank you.

As a side note, your blog is beautiful. Did you design it yourself?

Thank you for the compliment — I was going to write an article a few weeks ago, but ended up spending the evening redesigning the blog instead: http://tav.espians.com/new-site-design-for-2011.html

Procrastination has its benefits I guess, heh. If you fancy them, the css/templates are in https://github.com/tav/blog and the site is run using yatiblog — https://github.com/tav/ampify/blob/master/src/pyutil/yatiblo... — the source is all public domain, so do with it as you please.

I'm also migrating my blog right now, so don't mind if I do! Thank you!

Dude, I'm just saying it is pretty dubious that you show up at the community, poke around the tickets, ask lots of questions, probably notice the roadmap on Redmine where lots of the existing solutions are tagged for inclusion, and after all that interaction claim you singlehandedly fixed everything wrong with the project. I'm sorry if you are somehow upset by my doubt, but you have to admit the whole thing seems just a bit "too good to be true".

As for the difference between independent implementation and code-stealing: I have not and will not accuse you of stealing code. If it came off that way, I apologize.

I will however continue to assume your rooting around other people's work gave you ideas, gave you paths for completion of tasks, and gave you enough working information on reasonable solutions that you can not accurately say you did it by yourself.

Where are you giving the various "defsystem"-implementations credit? I bet 90% of fabric's features where already implemented in any of those 20 to 30 years ago.

Well I've never even heard of defsystem, let alone rooted around in it's implementation.

Nice strawman, but the subtle difference here is in one case the code was not looked at and talked about and in the other it was.

Just a quick note: these features are definitely experimental. The @task decorator has been in stable forks of Fabric for over a year at this point (this is the 3rd, if not 4th independent implementation of it). Likewise, parallel tasks have also been relatively stable for quite a long time.

Good to hear, hopefully the task decorator will make it in soon because I think it's a much superior API.

The little hack on Fabric I use currently takes the opposite approach -- I use an @internal decorator that hides certain functions from the task list but keeps the default Fabric behavior around for anything else. It's just a few lines of codes that mess with a private list Fabric uses to track internal functions:

    from fabric.main import _internals

    def internal(fn):
        _internals.append(fn)
        return fn
    _internals.append(internal)

That's an interesting idea. I like the idea of turning things off too. In some cases, that's more useful than explicitly defining a @task. Care to open a ticket on code.fabfile.org with it? Maybe we can work that in as an option in 1.1 (when @task is set to hit).

Sure, I'll try to file one later so it can at least be considered.

What I was trying to say in my earlier comment was that I would actually prefer the explicit @task decorator be part of the official Fabric API because it's more clear and, well, explicit. I prefer my @internal decorator as a hack in the meantime because I imagine it's less surprising to someone who might stumble upon my fabfile without any explanation from me.

Yea, definitely add a link to that in http://code.fabfile.org/issues/show/76 please! Might be nice to give people even more control. I think @task is "enough" but if we can have a richer API while not overburdening things, could be a plus.

Did some filter-branching and rebasing to get tav's work on top of current master: https://github.com/traviscline/fabric/tree/tav

This looks nice, but I have to be honest...I'm not going to switch from the version of fabric I use to this. Why? Even though I do have to deploy to more than 30 hosts at once, the thought of the time that would need to be spent verifying that there are no bugs in this would make it impractical to switch. I have to imagine there are others like me who feel the same way. It's a shame that the people this would most benefit are probably the least likely to switch.

I don't believe Tav posted this as an actual fork, but simply highlighting his branches/patches, in the same way that many other Fabric users have code waiting to be merged into core.

When the time comes to merge/incorporate them, we will definitely be wanting peoples' opinions on whether they feel the changes would fit into core Fabric :)

I admit to substantial ignorance here, but can anyone who's using fabric, or for whom these patches to fabric are an important improvement, speak to any kind of pro/con comparison they did with func (https://fedorahosted.org/func/)?

I've been using func for parallel deployment to 15 - 20 servers for about a year now, but I have to admit I didn't really do any research into other options at the time. Its written by the guys who wrote yum and cobbler, which are already core to my setup, so it seemed natural to just stay in their ecosystem. Thing is I never hear about anyone else using it, so I'm curious if anyone did more diligence than I and had specific reasons of choosing one or the other.

I had not heard of func until this thread. I chose fabric because it was the only Python-based option I was aware of.

The @task decorator is a real good idea. Many good changes in 1.0.

The only thing i miss is a virtualenv context manager (with virtualenv(blupp):...).

I've just started playing with Fabric, and I like your take on its functionality better. If they're resistant to merge you, why not fork into something new?

You definitely have something very useful there, from a casual read.

Whilst forking is always a possibility, I'd rather contribute back to the project and help improve Fabric collectively. The community is made up of really nice guys and it'd be great if we could figure out a way of getting the various ideas to work together.

Thanks for checking out the docs and your encouraging words!

Jeff mentioned on twitter that he's keen on merging some of the stuff post 1.0.

Yes. Those of us who have been in the Fabric community working with one another are aware of this. From my conversations on IRC with @tav, I thought he was as well.

It's always encouraging to see new people get involved. It's quite another to see someone relatively new to the community try to go around that community by announcing "new features" for a project they've barely started work on.

Sorry if that comes off a little bitter, but seeing this as the top post on HN today rubbed me the wrong way.

Plus one for the task decorator.

Very nice work all around.

Cool ideas. We're just now moving from a 1 machine deployment to a 4 (and probably more) machine deployment at work and I can already see the value in this.

There is definitely some cool stuff in there. I particularly like the @task decorator, the directory support, the context and of course the parallel tasks.

Nice work tav.

I cant see how adding such a wast amount of features makes the API cleaner. You made the library more complex than what it already was. Fixing Fabric should in my opinion be done, not in a backwards compatible way, but by forking it and taking away all the crap you don't want, then build a cleaner API. Adding more features to something which already is bad doesn't make it better.

Beautiful work. I've always felt that Fabric strikes a effective balance between simplicity and flexibility and love how you're extending it without breaking its spirit.

Reply on news.ycombinator.com