On Go’s Web Application Ecosystem
thechangelog.com
thechangelog.com
I still feel like for the majority of cases, this type of speed difference is not going to matter. The difference between the Go built in web framework at 190,687[2] (1.3 ms) json responses per second and Flask, a popular Python framework at 18,945[2] (3.1 ms) is trivial when you look at the bigger picture.
If you are receiving enough traffic for that to make a fundamental difference, you'll likely have enough potential or realized revenue to pay for a few more [Flask, Rails, "slow" framework] servers, or have enough developer time to warrant building it in Go.
The upside to using a "slow" framework is massive. Try writing an app like Ticketee[3] in Go, even with Gorilla et al and you'll realize the productivity gains you're missing out on.
For a small API, backend services, command line apps Go makes sense. But the big selling point there isn't speed, it's likely deployability.
I love Go. It's good for teams, deploys well, is enjoyable to program in, does things quickly. But if a client comes to me wanting a web application with forms, view logic, payments, 3rd party integrations – I'll take my Rails, thanks.
> Given all this, frameworks (and ecosystems) like Flask, Sinatra, Django and Rails still have an advantage for when you want to get something done today: they’re very useful toolboxes, handle a lot of the boilerplate and have a lot of examples and packages to leverage.
The author clearly recognizes this opinion – but there seems to be an idea that you can just "use Go" in place of existing web frameworks. That's not the case (yet!). I think this is largely driven by the myth of performance requirement.
[1]: Things like Node.js probably fit into this argument too
I guess it'll be interesting to watch how the frameworks mature and start to approach the Rails/Django or other micro-frameworks...
I disagree, and think that's a huge difference. The latency may not be, but requests per second? That could be a significant amount of difference if you start pulling decent traffic: you need ten Flask servers for every one Go server. That's a lot of ops complexity you can save yourself right there.
Flask, mind you, has a much more mature ecosystem (WTForms, SQLAlchemy, et. al) and development is likely to be faster on that front, but if you're looking at building something scalable and don't mind a bit of groundwork now, there is an advantage to using Go. It just may not be an advantage for every team (and that's fine: horses for courses).
If you're doing that kind of sustained request volume then you'd like have web server redundancy for availability reasons , right? Reverse proxying from nginx to a Go app vs. a flask/uwsgi app becomes a pretty small problem.
I'm definitely not arguing that Go isn't more "scalable". By all of our measures, it is. I'm talking about the huge sacrifices you make to productivity to use it as a developer for specifically modern web development.
I guess I don't picture a company saying, we're going to use GO through-and-through for our dev platform. I picture it more as a "well, we now have a custom messaging service that needs to scale better" or something, much like when people choose to drop into C or C++. It also seems that there's more dev-friendly languages to build the web layer in.... I also don't see many web-dev types thinking in GO very well.
So, why GO at the web-layer?
• I prefer to find my errors at compile time. Consider the time to find and fix a missing method in a go compile (one 'make' and all the info is still in the developer's brain cache) versus eventually having a user drive the website into that method and having a mystery occur on a duck typed language. (hope you have good stack trace logging on the back end)
• I prefer a language that doesn't repeatedly break all my code. When I write and deploy a site, it should stay written. (Looking at you PHP. Granted, this was over a period of 15+ years, but all the more reason I don't want to go back and look at that code.)
• Efficiency is nice if you are ever going to hit a performance bottleneck. To elaborate: you get to defer the "OMG I need to map this load over more than one machine!" crisis for a while, and if you end up in a "I need N machines!", then making N smaller is good for your well being.
• Builds are fast enough that I don't care. I generally have a makefile for a website anyway to get everything organized and encoded, one more rule to build the go is no burden.
The weakest spot I'm finding is the html/template end. My strong typing evaporates at that boundary. I'd rather have the template get precompiled into go code and then compiled into functions with nicely defined interfaces on its parameters; faster to run and catch all those typos in field names. The current go mindset of a "go only", non-extendable, build command forecloses experimentation in this direction. (Unless you are a barbarian from the olden days, like me, and wrap everything in a Makefile.)
When I make changes, refreshing the browser results in a re-compile which takes less than a second (assuming I have modified several files) on my 2012 Macbook pro.
With Go, unit testing is also blazing fast. My entire test suite (around 30+ tests) takes around 0.2 seconds.
Just for comparision, I used Play 2 (Scala) for a short while before that. Re-compilation would take in the region of 10 seconds. Startup costs are high in the Java/.NET world :(
So, yes, I look at web server of an application as a pretty distinct part of the back-end, separated from many other complexities in the system. I also view it as the beginner (can't think of a better term [1] ) part of the back-end-system where there's a lot of devs who can conceptualize this layer who can't the other stuff. It's these devs whether I question they'll conceptualize GO in general.
[1] EDIT: I just thought of a better way to describe "beginner" with more clarity: non-computer-science-types who are competent at working in the web layer.
Deployment is far simpler with no libraries/gems/packages - single binary
Memory usage is an order of magnitude smaller than languages like Ruby or Java.
It's pretty fast (much faster than Ruby for example)
String manipulation isn't too painful - all the tools are there for manipulating unicode text and producing html.
Why not?
This is to accompany blackfriday (Markdown) and text/template (templating).
Markdown permits HTML, and this allows some scope for nasty stuff to get in, or for bugs that may exist in blackfriday to be exploited leading to HTML that could be the source of a XSS attack.
We're currently running our user generated content through this: https://github.com/microcosm-cc/cleanse and more specifically this: https://github.com/microcosm-cc/cleanse/blob/master/src/main...
Which is a set of rules for OWASPs HTML sanitizer: https://code.google.com/p/owasp-java-html-sanitizer/
This works extremely well, except for the fact that it means in our Go code we're calling out to a process and asking for a Java process to be kicked up for each request.
If no-one beats us to it we'll be porting that to Go... but for us it remains a "When we need to" from a scaling or performance perspective.
Still... we'd love to see group work on a whitelist based sanitiser that we can contribute to rather than us go and write our own at some indeterminate point in the future.
http://godoc.org/code.google.com/p/go.net/html
Parse the HTML, walk the result, write that which is acceptable.
I have to restrict by tag, url scheme, and url server name in various contexts.
ParseFragment throws an error on bad input, but actually I just want that stripped and to carry on processing things. If a user has put in a mostly usable piece of HTML and then got something wrong as an error rather than bad intent then permissiveness in how we handle that should rule.
And then I wondered about the wisdom of creating a potentially large security library on a not quite nailed down API.
Ultimately, given that this is a security thing, I figured it's best to go with the proven many-eyeballed solution that was had widespread acceptance.
Feel free to use the package we've provided, the bit of go code you need for it is:
import (
"os/exec"
)
func SanitiseHTML(html string) (string, error) {
cleanse := exec.Command("java", "-jar", "/usr/sbin/cleanse.jar", "--permissive")
writer, err := cleanse.StdinPipe()
if err != nil {
return "", err
}
_, err = writer.Write([]byte(html))
if err != nil {
return "", err
}
err = writer.Close()
if err != nil {
return "", err
}
buff, err := cleanse.Output()
if err != nil {
return "", err
}
cleanse.Start()
return string(buff), nil
}https://github.com/microcosm-cc/bluemonday
Feel free to use it and help out.
Initial work was all done by Matt Jibson as part of his Google Reader Clone: https://github.com/mjibson/goread
We've chosen Go for two really significant reasons: a) to reduce attack surface area and b) to enable us to distribute binaries to customers who want an enterprise/firewall install inside their own network.
Speaking about surface area, we've reduced the amount of framework code in stack (rails, gems, etc are all off the table) to zero, and we've reduced the app paths to a few tens of lines of code, in most cases.
Whenever one trades off a framework "to make things more secure" one has to be sceptical, there's a massive scope to introduce SQL injection, XSS and other nasties. In Go, in our case this is fairly limited. So take that reason with a pinch of salt if you aren't building a VERY well defined app with VERY small and skilled team. I don't expect this to scale well.
The second part is also a niche value, whilst it will have some benefits for us running our hosted version of Harrow, it'll have more profound benefits when we can sell binary installations to customers.
Finally, learning how to build applications in an API centric manner, without the crutch that is Rails has been interesting, and has made us all better programmers, we have a more interesting architecture now, and are making better use of our tools (we're leaning heavily on PostgreSQL, triggers, stored procs, LISTEN/NOTIFY etc) and we have a faster system.
On the flip side of things, timing attacks are now really something we have to be mindful of. For instance the login endpoint returns "404 Not Found" regardless of whether it's an incorrect password, or the account doesn't exist. Except, interestingly we're using bcrypt (we're switching to pdbfk2 before the public launch) we need to look up the user before we can hash and verify the password. Meaning we respond in < 1ms when the user is not found and 4ms when the user is found, but the password is incorrect. (and 4ms when the password is valid). This also applies to a lesser extent in responding with "404 Not Found" when trying to snoop on someone else's project/logs/build/etc. The same way they Github does. But it's less critical there as there isn't the artificially long password hash comparison step.
All told the decision to build in Go has made the development of the project slower, but has in my mind at least lead to a better quality of software, with things that don't typically show up in Rails/et al projects (foreign key constraints, triggers in the DB).
Of course all these tools are sharp weapons if handled incorrectly, and testing triggers/functions in the database can be a nightmare from a unit/functional testing point of view.
We've avoided technology proliferation (the entire stack is Go, and PostgreSQL, with worker nodes which rely on LXC). The entire application uses three packages not taken from the Go standard library. (db driver and bcrypt library)
Ohh, and our test suite compiles, and runs (all of it.) in less than three seconds. That hasn't hurt much, either.
Interesting approach to the stack, I'm considering the same for some side projects. I like the exercise in a fully API driven approach. A major advantage is the pure separation of client and server means refactoring/changing languages becomes more straightforward you can also isolate data/logic from presentation more cleanly.
I've seen lots of system utility stuff, but I don't think I've ever actually seen anything other than how-to-use-this-framework web apps?
Or at least, the language resists the sort of ORMs that the web world has become accustomed to, with tons of convention-over-configuration and such. A more static one could probably be done.
I find myself wishing the "go" toolchain would give me a defined way of running certain code before the compiler, so I could generate Go code with the go source tools. From there you could easily compile yourself up some ORM code by examining a database at runtime. It's yet another privilege it claims for itself but gives you no way to hook into.
There is Revel, but it seems to be getting a bad rap ("non-idiomatic" label used as a stick).
I don't expect everyone to agree with me surrounding Revel, but I certainly think it's lack of modularity is a negative.
https://groups.google.com/forum/#!searchin/golang-nuts/marti...
The Revel comment was more about a criticism I've read several times now.
I liked and bookmarked the article. :)
Martini is still young (less than a week) but the core is pretty fleshed out.
No boosting here (I don't even have that many twitter followers). I just put together a framework that doesn't step on your toes and people ate it up :)
Having Java/Scala background I used to have static checking and performance for free and I cannot feel about Go the same as someone who switched from Python, PHP, Ruby stacks. I code mainly in Scala, and even if I like Go's simplicity, compilation speed and the absense of Java burden, I really miss features like default immutability, pattern matching, generics, reach collection toolset, powerful typesystem. The language looks just not expressive for me to really enjoy.
Maybe I am corrupted with FP, cannot value simplicity or I should focus more on getting things done not on the process of writing code. Just wonder, if anyone else have similar impressions / feelings?
1. Write a backend webservice in Go's net/http + Gorilla
2. Use something like angular JS to interact with those webservices. (From the server's perspective this means static HTML/CSS/JS assets that can again be served by Go, from memory if desired).
My first couple Go web applications did things the html/template way and were structured on the way I used to go JSF applications- but since I switched to the above approach my projects have been better performing, faster to develop , better scaling and easier to maintain. As we all know, we seldom get all those things in one technology choice without trade-offs.
I've even considered Github pages for the HTML. But probably not for production :)
One of the goals of Martini is to rally developers to create some more reusable web modules in the form of handlers and middleware. Martini has gotten a great response because it doesn't infringe it's api on your modules/middlewares.
I personally think that this is the way to move forward with web development in Go. Less coupling, pull in the dependencies you need, get stuff done.
view := Views{
DIV("myclass",
H1("Example HTML structure"),