NilAway: Practical nil panic detection for Go
uber.com
uber.com
I found it a little funny that their big "win" for the nilness checker was some code logging nil panics thousands of time a day. Literally an example where their checker wasn't needed because it was being logged at runtime.
It's a good idea but they need some examples where their product beats running "grep panic".
The advantage of NilAway is not just detecting nil panic crashes after the fact (as you note, we should always be able to detect those eventually, once they happen!), but detecting them early enough that they don't make it to users. If the tool had been online when that panic was first introduced, it would have been fixed before ever showing up in the logs (Presumably, at least! The tool is not currently blocking, and developers can mistake a real warning for a false positive, which also exist due to a number of reasons both fundamental and just related to features still being added)
But, on the big picture, this is the same general argument as: "Why do you want a statically typed language if a dynamically typed one will also inform you of the mismatch at runtime and crash?" "Well, because you want to know about the issue before it crashes."
Beyond not making it all the way to prod, there is also a big benefit of detecting issues early on the development lifecycle, simply in terms of the effort required to address them: 'while typing the code' beats 'while compiling and testing locally' beats 'at code review time' beats 'during the deployment flow or in staging' beats 'after the fact, from logs/alerts in production', which itself beats 'after the fact, from user complains after a major outage'. NilAway currently works on the code review stage for most internal users, but it is also fast enough to run during local builds (currently that requires all pre-existing warnings in the code to either be resolved or marked for suppression, though, which is why this mode is less common).
Nilability of return values should be part of functions public interface. It shouldn't come as a surprise under certain circumstances of using the code. The problem of global inference is that it targets both the producer and the consumer of the interface at the same time, without a mediating interface definition deciding who is correct. If a producer starts returning nil and a consumer five levels downstream the call-stack happens to be using it, both the producer and caller is called out, even if that was documented public api before, just never executed. Or vice versa.
For anyone who had the great pleasure of deciphering error messages from C++ templates, you know what I'm talking about.
I understand the compromises they had to take due to language constraints and I'm sure this will be plenty useful anyway. Just sad to see that a language, often called modern and safe, having these idiosyncrasies and need such workarounds.
Hi! I use global type inference and I love it.
$ nilaway ./...
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x100c16a58]
If you don't squint, then I don't think so.
Otherwise, since pointers are frequently used to represent optional parameters, generics + sum types would get the job done; for that use case, it's one of two steps to solve the problem. I don't foresee Go adding sum types, though.
It's basically mandating Rust's Default trait or the C++ default (no argument) constructor. In some places you can live with a Default but you wish there wasn't one. Default Gender = Male is... not great, but we can live with it, some natural languages work like this, and there are problems but they're not insurmountable. Default Date of Birth is... 1 January 1970 ? 1 January 1900? 0AD ? Also not a good idea but if you insist.
But in other places there just is no sane Default. So you're forced to create a dummy state, recapitulating the NULL problem but for a brand new type. Default file descriptor? No. OK, here's a "file descriptor" that's in a permanent error state, is that OK? All of my code will need to special case this, what a disaster.
> All of my code will need to special case this, what a disaster.
No, your code should handle the error state first and treat the value as invalid up until that point, e.g.
foo, err := getVal()
if err != nil {
return
}
// foo can only be used now
They made trade-offs and are conservative about refining the language; that cuts both ways but works well for a lot of people.
The Go team does seem to care about improving it and for many that use it, it keeps getting better. Perhaps it doesn't happen at the pace people want but they always have other options.
Perl was also a successful language with significant adoption. At least back then, we didn’t know any better.
In twenty years the industry will look back on golang as an avoidable mistake that hampered software development from maturing into an actual engineering discipline, for the false economy of making novice programmers quickly productive. I’m willing to put money on that belief, given sufficiently agreed upon definitions.
Lisp, Haskell, OCaml all likely tickle your PL purity needs, but they remain niche languages in the grand scheme of things. Does that make them bad?
I think Go will be the new Java (hopefully without the boilerplate/bloat). It's good enough to do the job in a lot of cases and plenty of problems will be solved with it in a satisfactory manner.
Language wars are only fun to engage with for sport, but it's silly to get upset about them. Most languages have value in different contexts and I believe the real value in this dialog is recognizing when and where a language works and to accept one's preferred choice may not always be "the one".
Modern PLT and metaprogramming and more advanced type systems enable the creation of even more complex abstractions and concepts, which are even harder to understand or reason about, let alone maintain. This is the antithesis of whatever software engineering represents. Engineering is almost entirely about process. Wielding maximally expressive code is all science. You don't need to be a computer scientist to be a software engineer.
> Much of the language design was motivated by learnings within Google.
And the main problem Google had at the time was a large pool of bright but green CS graduates who needed to be kept busy without breaking anything important until the mcompany needed to tap into that pool for a bigger initiative.
> What other "modern" language that has more of a focus on software engineering, putting readability and maintainability and stability at the forefront?
This presupposes that golang was designed for readability, maintainability, and stability, and I assert it was not.
We are literally responding to a linked post highlighting how golang engineers are still spending resources trying to avoid runtime nil panics. This was widely known and recognized as a mistake. It was avoidable. And here we are. This is far from the only counterexample to golang being designed for reliability, it’s just the easiest one to hit you over the head with.
Having worked on multiple large, production code bases in go, they are not particularly reliable nor readable. They are somewhat more brittle than other languages I’ve worked with as a rule. The lack of any real ability to actually abstract common components of problems means that details of problems end up needing to be visible to every layer of a solution up and down the stack. I rarely see a PR that doesn’t touch dozens of functions even for small fixes.
Ignoring individual examples, the literal one thing we actually have data on in software engineering is that fewer lines of codes correlates with fewer bugs and that fewer lines of code are easier to read and reason about.
And go makes absolutely indefensible decisions around things like error handling, tuple returns as second class citizens, and limited abstraction ability that inarguably lead to integer multiples more code to solve problems than ought to be necessary. Even if you generally like the model of programming that go presents, even if you think this is the overall right level of abstraction, these flagrant mistakes are in direct contradiction of the few learnings we actually have hard data for in this industry.
Speaking of data, I would love to see convincing data that golang programs are measurably more reliable than their counterparts in other languages.
Instead of ever just actually acknowledging these things as flaws, we are told that Rob Pike designed the language so it must be correct. And we are told that writing three lines of identical error handling around every one line of code is just Being Explicit and that looping over anything not an array or map is Too Much Abstraction and that the plus sign for anything but numbers is Very Confusing but an `add` function is somehow not, as if these are unassailable truths about software engineering.
Instead of actually solving problems around reliability, we’re back to running a dozen linters on every save/commit. And this can’t be part of the language, because Go Doesn’t Have Warnings. Except it does, they’re just provided by a bunch of independent maybe-maintained tools.
> enable the creation of even more complex abstractions and concepts
We’re already working on top of ten thousand and eight layers of abstraction hidden by HTTP and DNS and TLS and IP networking over Ethernet frames processed on machines running garbage-collected runtimes that live-translate code into actual code for a processor that translates that code to actual code it understands, managed by a kernel that convincingly pretends to be able to run thousands of programs at once and pretends to each program that it has access to exabytes of memory, but yeah the ten thousand and ninth layer of abstraction is a problem.
Or maybe the real problem is that the average programmer is terrible at writing good abstractions so we spend eons fighting fires as a result of our collective inability to actually engineer anything. And then we argue that actually it’s abstraction that’s wrong and consign ourselves to never learning how to write good ones. The next day we find a library that cleanly solves some problem we’re dealing with and conveniently forget that Abstractions Are Bad because that’s only something we believe when it’s convenient.
Yes, this is a rant. I am tired of the constant gaslighting from the golang community. It certainly didn’t start with “generics are complicated and unnecessary and the language doesn’t need them”. I don’t know why I’m surprised it hasn’t stopped since them.
- The language has barely changed since inception
- most if not all behavior is localized and explicit meaning changes can be made in isolation nearly anywhere with confidence without understanding the whole
- localized behavior means readable in isolation. There is no metaprogramming macro, no implicit conversion or typing, the context squarely resolves to the bounds containing the glyphs displayed by your text editor of choice.
The goal was not to boil the ocean, the goal was to be better for most purposes than C/C++, Java, Python. Clearly the language has seen some success there.
Yes, abstractions can be useful. Yes, the average engineer should probably be barred from creating abstractions. Go discourages abstractions and reaps some benefits just by doing so.
Go feels like a massive step in the right direction. It doesn't have to be perfect or even remotely perfect. It can still be good or even great. Let's not throw the baby out with the bath water.
I think for the most part I'm in agreement with you, philosophically, but I don't get the hyperfocus on this issue. Most languages you consider better I consider worse, let's leave it at that.
I agree. Go has it's warts, but given the choice between using net/http in go, tomcat in java, or cpprestsdk in c++, I'll pick Go any day.
In practice: - The toolchain is self contained, meaning install instructions don't start with "ensure you remove all traces of possibly conflicting toolchains" - it entirely removes a class of discussion of "opinion" on style. Tabs or spaces? Import ordering? Alignment? Doesn't matter, use go fmt. It's built into the toolchain, everyone has it. Might it be slightly more optimal to do X? Sure, but there's no discussion here.
- it hits that sweet spot between python and C - compilation is wicked fast, little to no app startup time, and runtime is closer to C than it is to python.
- interfaces are great and allow for extensions of library types.
- it's readable, not overly terse. Compared to rust, e.g. [0], anyone who has any programming experience can probably figure out most of the syntax.
We've got a few internal services and things in Go,vanr we use them for onboarding. Most of my team have had PR's merged with bugfixes on their first day of work, even with no previous go experience. It lets us care about business logic from the get go.
[0] https://github.com/getsentry/symbolicator/blob/master/crates...
And here is my opinion:
I think in 20 years, Go will still be a mainstream language. As will C and Python. As will Javascript, god help us all.
And while all these languages will still be very much workhorses of the industry, we will have the next-next-next iteration of "Languages that incorporate all that we have learned about programming language design over the last N decades". And they will still be in the same low-single-percentage-points of overall code produced as their predecessors, waiting for their turn to vanish into obscurity when the next-next-next-next iteration of that principle comes along.
And here is why:
Simple tools don't prevent good engineering, and complex tools don't ensure it. There are arcs that were built in ancient Rome, that are still standing TODAY. There are buildings built 10 years ago that are already crumbling.
And yet the mainstream consensus is that C and JavaScript are terrible languages with deep design flaws. These weren’t as obvious pr avoidable at the time, but they’re realities we live with because they’re entrenched.
My assertion is that in twenty years, we’ll still be stuck with go but the honeymoon will be over and its proponents will finally be able to honestly accept and discuss its design flaws. Further, we’ll for the most part collectively accept that—unlike C and JavaScript—the worst of these flaws were own goals that could have and should have been avoided at the time. I further assert that there will never be another mainstream statically-typed language that makes the mistake of nil.
For that matter I think we’ll be stuck with Rust too. But I think the consensus will be that its flaws were a result of its programming model being somewhat novel and that it was a necessary step towards even better things, rather than a complete misstep.
Oh, they all have flaws. But whether these make them "terrible" is a matter of opinion. Because they are certainly all very much usable, useful and up to the tasks they were designed for, or they would have vanished a long time ago.
> and its proponents will finally be able to honestly accept and discuss its design flaws
We are already doing that.
But that doesn't mean we have to share anyones opinion on what is or isn't a terrible language, or their opinions about what languages we should use.
And yes, that is all these are: opinions. The only factually terrible languages are the ones noone uses, and not even all languages that vanished into obscurity are there because people thought them to be "terrible".
Go does a lot of things very well, is very convenient to use, solves a lot of very real problems, and has a lot to offer that is important and useful to us. That's why we use it. Opinions about which languages are supposedly "terrible" and which are not, is not enough to change that.
An new language has to demonstrate to us that its demands on our time are worth it. It doesn't matter if it implements the newest findings about how PLs should be designed according to someones opinion, it doesn't matter if its the bees knees and the shiniest new thing, it doesn't matter if it follows paradigm A and includes feature B...the only thing that matters is: "Are the advantages this new thing confers so important to us, that we have a net benefit from investing the time and effort to switch?"
If the answer to that question is "No", then coders won't switch, because they have no reason to do so. And to be very clear about something: The only person who can answer if that switch is worth for any given programmer, is that programmer.
We have a 1MM line c++ codebase at work, a rust third party dependency, and a go service that's about as big as the rust dependency. Building the Rust app takes almost as long as the c++ app. Meanwhile, our go service is cloned, built, tested and deployed in under 5 minutes.
It's not "a bit faster", it's "orders of magnitude faster". We use a third party rust service that compile occasionally, a clean build of about 500 lines of code plus external crates (serde included) is about 10 minutes. Our go service is closer to 5 seconds. An incremental build on the rust service is about 30s-1m, in go it's about 5 seconds. It's the difference between waiting for it to start, and going and doing something else while you compile, on every change or iteration.
> what ergonomics does Go give you that Rust doesn't
- Compilation times. See above.
- How do I make an async http request in rust and go? in go it's go http.Post(...)
In rust you need to decide which async framework you want to use as your application runtime, and deal with the issues that brings later down the line.
- In general, go's standard library is leaps and bounds ahead of rust's (this is an extension of the async/http point)
- For a very long time, the most popular crates required being on nightly compilers, which is a non-starter for lots of people. I understand this is better now, but this went on for _years_.
- Cross compilation just works in go (until you get to cgo, but that's such a PITA and the FFI is so slow that most libraries end up being in go anyway), in rust you're cross compiling with LLVM, with all the headaches that brings with it.
- Go is much more readable. Reading rust is like having to run a decompressor on the code - everything is _so_ terse, it's like we're sending programs over SMS.
- Go's tooling is "better" than rust's. gofmt vs rustfmt, go build vs cargo build, go test vs cargo test
For anything other than soft real time or performance _critical_ workloads, I'd pick go over rust. I think today I'd still pick C++ over rust for perf critical work, but I don't think that will be the case in 18-24 months honestly.
(Sometimes your domain type really does have a suitable natural default value, and you just make that the zero value.)
enum Gender {
Unspecified,
Male,
Female,
Other,
}
impl Default for Gender {
default() -> Self {
Self::Unspecified
}
}
enum Gender {
Male,
Female,
Other,
}
type Gender int
const (
Unspecified Gender = iota
Male
Female
Other
)
It’s nil pointers all over again, but for your non-pointer types too! Default zero values are yet another own goal that ought to have been thrown away at the design stage.
That's... not what I'm looking for out of my type system. I'm mostly looking for autocomplete and possibly better perf because the compiler has size information. I really hate having to be a type astronaut when I work in scala.
So, I mean, valid point. And I do cede that point. But it's kind of like telling me that my car doesn't have a bowling alley.
I don't really use them that much, so they're superfluous for the most part. Sort of like a car having a bowling alley. I mean, I'll take them if it doesn't complicate the language or impact compile time, but if they're doing to do either of those, I'd rather just leave them.
Adding default branches into the couple of switch statements and a couple spots for custom json parsing that return errors for values outside the set doesn't seem like a bad tradeoff.
I also imagine I'd use a bowling alley in my car ~2 times per year, tops. So that seems like a better analogy to me.
edit: I guess I should bring up that I don't use go's switch statement much either, and when I do, 99% of the time I'm using the naked switch as more of an analog to lisp's cond clause.
Re: custom UnmarshalJson implementation - you still have to remember, and do it for every serialized format (e.g. sql).
A default case in a switch only solves, well, switching. If a rogue value comes in, it will go out somewhere. E.g. json to sql or whatever things are moving.
I mean, yeah, but eventually you have to do something with it. And the only useful thing you can really do with an enum is switch on it...
The point the others are making is you are using a less safe type of programming equivalent to driving without ABS and TC.
From your point of view, "TC and ABS is useful so I use it unlike enums".
From their point of view though, you are the person not using ABS and TC insisting that they offer nothing useful.
Can you explain how owning a car without ABS and not driving it is less safe?
Edit: Wait, or am I not using breaks? I think the analogy changed slightly during this whole process.
I don't know that it is, I'm speaking based on the implication in this thread that no-enum and no-ABS people are overconfident to a fault.
I do believe that those who claim they don't need enums, static typing, etc are probably overconfident and have a strong desire or need to feel more control.
I'm not sure though, at least for GC'd languages, how enums sacrifice control.
var g Gender // ok so far.
if err := json.Unmarshal("99", &g); err != nil { panic(err) }
// no error and g is Gender(99)!
Go is simple and gets you going fast... but later makes you stop and go back too much.
First we define the type, hiding the pointer/non-existent value:
type Optional[Value any] struct {
value Value
exists bool
}
func (o Optional[Value]) Get() (Value, bool) {
return o.value, o.exists
}
if value, ok := optional.Get(); ok {
// value is valid
}
// value is invalid
Here's a full implementation I wrote a while back: https://gist.github.com/MawrBF2/0a60da26f66b82ee87b98b03336e...
Maybe we’ll get a Golang 3 with sum types…
What answer do you expect?
Partly this is out of memory of the good/bad old newsgroup days where this kind of thing somehow worked ok, until it didn't, but it definitely doesn't work on the sort of forum that HN is. We'd like a better outcome than scorched earth for this place.
I am toying around with a similar project, with the same goal, and it is DIFFICULT.
I'll definitely get to learn from their implementation.
Hopefully with time, when exploring union types and perhaps a limited form of generalized subtyping (currently it's only interface types) we'll be able to deal with nil for good.
Nil is useful, as long as correctly reined in.
A good way to rein in behaviour is with types. If you need Nil in your domain, great! Give it type 'Nil'.
The untyped nil type is just not a first-class citizen nowadays.
But with type sets, we could probably have ways to track nillables at the type system level through type assertions.
And where nillables are required such as map values it would be feasible to create some from non nillables then ( interface{T | nil})
But that's way ahead still.
Tested via vim and looks good!
We definitely have gotten some useful reports there already since the blog post!
We are aware of a number of sources of false positives and actively trying to drive them down (prioritizing the patterns that are common in our codebase, but very much interested in making the tool useful to others too!).
Some sources of false positives are fundamental (any non-trivial type system will forbid some programs which are otherwise safe in ways that can't be proven statically), others need complex in-development features for the tool to understand (e.g. contracts, such as "foo(...) returns nil iff its third argument is nil"), and some are just a matter of adding a library model or similar small change and we just haven't run into it ourselves.
- You're confident that a flagged value is actually non-Nil?
- A value was Nil but you prefer it that way?
I have some code that eventually core dumps and honestly I don't know what I'm doing wrong, and neither do any golang tools I've tried :(
maaaaaybe there's something that'll check that your code never closes a channel or always blocks after a specific order of events happens...
Yes. Per rsc (https://research.swtch.com/gorace)
> In the current Go implementations, though, there are two ways to break through these safety mechanisms. The first and more direct way is to use package unsafe, specifically unsafe.Pointer. The second, less direct way is to use a data race in a multithreaded program.
That races undermine memory safety in go has been used in CTFs: https://github.com/netanel01/ctf-writeups/blob/master/google...
These are not idle fancies, there are lots of ways to unwittingly get data races in go: https://www.uber.com/blog/data-race-patterns-in-go.
In C and C++ SC/DRF (Sequentially Consistent if Data Race Free) turns into "All data races are Undefined Behaviour, game over, you lose". In Go SC/DRF turns into "All data races on complex types are Undefined Behaviour, game over, you lose". If you race e.g. a simple integer counter, it's damaged and you ought not to use it because it might be anything now, but Go isn't reduced to Undefined Behaviour immediately for this seemingly trivial mishap (whereas C and C++ are)
The reasons C compilers “tend to go out of their way to make weird things happen” is they optimise extremely aggressively, and optimisations are predicated upon the code being valid (not having UBs).
Go barely optimises at all, and does not have that many UBs which could send the optimiser in a frenzy.
Implicitly shared memory is literally the default behaviour of the langage, and you have to be careful to keep that controlled or contained.
Pretty much as in every other shared memory concurrency langage.
The quip about sharing memory by communicating is cute but it’s just that, the langage does not encourage let alone enforce it.
In fact it went out of its way to remove some opportunities e.g. because `go` is a statement there is no handle which could communicate the termination and result of routines.
But I think that focusing on nils is a wrong analysis. The problem is the default zero-values dogma, and that is not going to change anytime soon.
Sometimes you also need a legitimate empty string or 0 integer, but the language cannot distinguish it from the absence of value.
In my codebase, I was able to improve the readability of those cases a lot by using mo.Option, but that has a readability cost and does not offer the same guarantees than a compiler would. The positive side is that I get a panic and clear stack trace whenever I try to read an absent value, which is better than nothing, but still not as good as having those cases at compile time.
No amount of lint checkers (however smart) will workaround the fact that the language cannot currently express those constraints. And I don't see it evolving past it's current dogmas unfortunately, unless someone forks it or create something like typescript for go.
The Go team is very careful to avoid breaking changes (cue all the usual Well Actually comments regarding breaking changes that affected exactly zero code bases) and rightfully so. Their reputation as a stable foundation to build large projects upon has been key to the success and growth of the language and its ecosystem.
I have about a million and one other issues I'd like to see resolved first that don't involve breaking changes. It's a known pain point, the core maintainers acknowledge it, but suggestions to fundamentally derail the entire project are ludicrous.
Focusing on nils is fine. NilAway is fine. It's a perfectly reasonable approach and adds a lot of value. This solves a real problem in real code bases today. There is no universe wherein forking to create a new language creates remotely equivalent value.
For example we could have a new non-nilable pointer type (that would not have any default value), or an optional monad natively in the language (or any other thing in-between, there are many possibilities). That would allow the compiler to statically report about missing checks, without breaking backward compatibility.
But we all know that it's not going to happen soon because while not breaking any existing code, it goes against the "everything has a zero-value" dogma. That was the meaning of my message.
https://github.com/golang/go/issues/57644
https://github.com/golang/go/issues/19412
I interpret your comments as propagating FUD in bad faith.
Even if sum types were introduced, it would not help with nil values because - as you said - backward compatibility won't be broken.
If I had the luxury of spare time to contribute, I would probably spend it switching away to another richer language instead, because it would be cheaper, solve more of my problems and with a higher degree of certainty. And that's not even mentioning the attitude and toxicity of the community compared to most of other languages facing critics and ideas.
> backward compatibility won't be broken.
I'm not sure what your point is anymore. You are clearly divested. Your assessment is totally unfounded. What are you trying to accomplish here?
You can disagree with me and criticize my points, but I do not feel like it's done in a good faith or is leading to anything constructive.
So I'm going to stop the discussion here and let the readers judge by themselves what I meant, and what to conclude about it.
Insane that Go had decades of programming mistakes to learn from but it chose this path.
Anyway, at least Uber is out there putting out solid bandaids. Their equivalent for Java is definitely a must-have for any project.
For java projects I think NullAway has gotten so good that it really takes the steam out of the Kotlin proponents. Hopefully NilAway will get there too.
Yup, every time I write some Go I feel like it's been made in a vaccum, ignoring decades of programming language. null/nil is a solved problem by languages with sum types like haskell and rust, or with quasi-sums like zig. It always feels like a regression when switching from rust to go.
Kudos to Uber for the tool, it looks amazing!
Go is just obstinately living in the 90s. I guess that's not really a surprise. It's pretty much C but with great tooling.
The problem with Go (for this specific issue, there are lots of problems with Go) is that they have wedded themselves extremely strongly to zero-defaulting, it's absolutely ubiquitous and considered a virtue.
But without null you can't 0-init a pointer, so it's incompatible with null-safety.
I think C# pretty much left the idea of everything having a default behind when they decided to fix nulls. Though obviously the better alternative is to have opt-in defaulting instead.
† Strictly speaking, objects can be partially initialized and partially uninitialized in Rust, but this is harmless as the borrow checker statically ensures that uninitialized fields of objects are never accessed.
True, and because of this, the language can be learned over a weekend or during onboarding, new hires can rapidly digest codebases and be productive for the company, code is straightforward and easy to read, libraries can be quickly forked and adapted to suit project needs, and working in large teams on the same project is a lot easier than in many other languages, the compiler is blazing fast, and it's concurrency model is probably the most convenient I have ever seen.
Or to put this in less words: Go trades "being-modern" for amazing productivity.
> It always feels like a regression when switching from rust to go.
It really does, and that's what I love about Go. Don't get me wrong I like Rust. I like what it tries to do. But I also love the simplicity, and sheer productiveness of Go. If I have to deal with the odd nil-based error here and there, I consider that a small price to pay.
And judging by the absolute success Go has (measured by contributions to Github), many many many many many developers agree with me on this.
Yeah, I truly hate this field
Sure, it could be done. Lots of things could be done to Go. The people who invented it are among the most brilliant computer scientists alive. It's a pretty sure bet that they know about, and in great detail, every single thing people complain Go doesn't have.
So every thing that is "missing" from Go isn't in it for a reason.
"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." -- Antoine de Saint-Exupéry, Airman's Odyssey
> You have to use an interface. That’s much more complex than using a sum type would be.
More complex how and by what metric?
But to understand the specification and how it can be used to do a programming, you need to have at least a cursory understanding of turing machines and related theory, which isn't necessary to learn Java or python.
Under your definition, the conceptually simplest language is something like SUBLEQ, (the specification is only a single line!) but in this case, being able to implement the language, and learning the language aren't the same thing. Learning the language generally means, like, useful for given purposes.
Removing footguns (nulls are a footguns, race-able concurrent APIs are a footgun) can make it easier to learn even though this may introduce new features (in this case sum types) to solve the problem.
Had those brilliant computer scientists not been employed at Google, it would have been another Oberon or Limbo.
But it's not a guarantee for success either. Google+ anyone?
The only ones that are actually taking off, have the baking of major corps, even if it is only giving money into the project.
“Optimising your notation to not confuse people in the first 10 minutes of seeing it but to hinder readability ever after is a really bad mistake.” — David MacIver
Go is a simple language that anyone can pick up in a weekend, but productivity plateaus once you’re doing anything that requires hard constraints or complex systems (the same is true for JS, Python, and other scripting languages).
This is a false dichotomy. One does not imply the other.
Go is also not a simple language. It is deceptively difficult with _many_ footguns that could have easily been avoided had it not ignored decades of basic programming language design.
Many things also aren't straightforward or intuitive. For instance, this great list of issues for beginners: http://golang50shad.es/
No it isn't, and yes it does. By definition, the more features I add to something, the more complex it becomes. So yes, Go achieves it's simplicity precisely by leaving out features.
> this great list of issues
I just picked three examples at random:
"Sending to an Unbuffered Channel Returns As Soon As the Target Receiver Is Ready"
"Send and receive operations on a nil channel block forver."
"Many languages have increment and decrement operators. Unlike other languages, Go doesn't support the prefix version of the operations."
All of these are behavior and operators that are documented in the language spec. So how is any of these a "footgun"?
I literally can not imagine a worse behavior than my program blocking forever. Of all of the things my program can do, short of giving remote code execution, blocking is literally the worst one I can think of.
The behavior of nil-channels always blocking is on purpose, and tremendously useful in functions where we receive from multiple channels via `select`. It allows the function to easily manipulate it's own receive behavior simply by setting the local reference to the channel to `nil`.
Since `selects` can also have `default`, the resulting functions don't have to block either.
More complex for whom? Not having generics made the compiler simple, but having to copy and paste and maintain identical implementations of a function (or use interface) adds more complexity for users.
Similarly, adding a better default HTTP client arguably makes Go more complex, but the "simple" approach results in lots of complexity and frustration for users.
> All of these are behavior and operators that are documented in the language spec. So how is any of these a "footgun"?
Perhaps I could have been clearer. I didn't mean that the entire list was of footguns, just that there are lots of confusing and unintuitive things beginners need to learn.
Some actual footguns off the top of my head:
- using Defer in a loop
- having to redeclare variables in a loop
- having to manually close the body of a http response even if you don't need it
Yes, and go opts to include features that unnecessarily increase complexity in this manner, such as nil values.
> All of these are behavior and operators that are documented in the language spec. So how is any of these a "footgun"?
By this logic, no language with a spec can have footguns. C and C++, notorious for their footguns, both specify their behavior in the spec, so do they not have any footguns?
You know what's a footgun? Uncaught exceptions popping up in places far away from where they were created at which point you have very little context to deal with it robustly. Use after frees. FactoryFactoryFactories.
You're right, I meant to link that in reference to how Go can be difficult to learn despite how it simple it seems. Not sure how I a sentence.
The overview of that site explains its purpose/necessity quite well. Some things are footguns, many are just confusing time-wasters. Nevertheless, they are frustrating and hamper the learning process.
But that is the learning process. What else is there to learn in a language if the syntax doesn't count? They're all Turing complete and all of them can do everything. All we need to do is learn the exact magic words.
I don't think it's honest to parade Go as a language that's the paragon of simplicity that's easy to learn when that's simply not true. I also don't think it's honest for people to argue that addressing any of Go's countless warts would somehow make the language more complex or harder to learn.
But Go's real strength is in its readability, not writability. I think it's very much possible to learn Go in a week, then read clean Go code like the standard library and understand exactly what's going on. At least that's my interpretation of what it means for a new grad to be productive in Go in less than a week. Nobody is expecting someone new to write production-grade libraries with intricate concurrency bits in their first week, but they're already productive if they can read and understand it.
As a rule of thumb we spend 10x more time reading code than we do writing it (code reviews, debugging, refactors). So why not optimise for it?
I generally do the second, because doing the first is extremely tiring when reviewing code, but I dislike it immensely.
Uncaught exceptions -> panics, like what this nil catcher is aiming to solve
Places far away -> easy goroutine creation with no origin tracking makes errors appear sometimes very far away from source
Use after free -> close after close
FactoryFactoryFactories -> loads of BuilderFunc.WithSomething
Lots of other pains I could add that are genuinely novel to Go also, but funny that for everything you mentioned my head went “yep, just called X”
This may be ok, as you say, if you allow errors here and there because you are fine dealing with those problems. But at the other end, it may be a user that is affected by the error. Which may be ok as well, but why should it be? We lament the quality of software all the time.
Compare this to other engineering fields: unless you study the knowledge of those who came before you may not even be allowed to practice in the field. I would not want to use a bridge built by someone who learned bridge building in a weekend.
Software is different though, it's rarely a matter of life or death. Given that, maybe it's ok to not have the highest quality in mind, because the benefit of productivity far outweighs the alternative.
I'm torn.
The line has to be drawn somewhere. I think everyone has certain things they'd put on the other side of that line, and strict nils are probably at the top of the list for many, but overall it's good that the Go team is stubborn about not adding new stuff. If they weren't, maybe there would be better nil handling, better error handling, etc. but compiles would also get slower and the potential for over-engineering, which Go now discourages quite effectively, would increase. At a high level, keeping Go a simple, pragmatic language with a fast compiler is more important than any particular language feature.
And how well did that work out for Haskell?
https://gist.github.com/graninas/22ab535d2913311e47a742c70f1...
Just because one person thinks Functional Programming is the right way to do it, doesn't mean another person has to agree. The same goes for every paradigm, and language feature under the sun. Different people want different things, different projects have different needs.
No single language gets everything right, no single paradigm solves every problem, no feature is a "must have" in every language. A functional approach might be great or a productivity killer depending on the use case. A GC may be the best thing in the world or a performance nightmare. OOP may be a really good idea or a path to unmaintainable crap depending on the implementation.
There are no silver bullets.
The only thing that is ABSOLUTELY certain: When people get told "Our way is better, you should use our way", despite the fact that there are no silver bullets, people will resist. And that resistance can lead to languages vanishing into obscurity.
You know this post is speculative fiction, right? It's actually about what could kill Haskell, not what could kill Rust?
Here is the articles title: "What killed Haskell, could kill Rust, too"
So no, it's not about what could kill Haskell. In 2022, ~0.3% of all code pushed to github was Haskell. To put that number into perspective: vimscript was ~0.25%
Yes, it is a speculative fiction, with a good reason, because the analogies are pretty clear.
https://news.ycombinator.com/item?id=38360177
So, according to what metric is Haskell "thriving"?
It was designed, specifically, as per Rob Pike, for _bad_ developers. Developers who couldn't be productive at Google because they weren't properly taught at unis [0].
Then it caught momentum and then here we are, discussing a bad language designed for bad developers as if there is nothing better we can do with our lives.
Pike's point is that peak PLT is too lofty to be productive or even useful for folks who are actually technically competent and literate relative to the rest of the industry. No one will get anything done if they're spending all their time teasing an advanced type system into inferring the required program.
A solution might be optionals, which might require sum types, which might require generics (which Go just learned), which most definitely requires a more complex type system, which almost certainly involves longer compiler times.
Is that all worth it? I don't know. The Go team certainly didn't think so.
Languages that I'm aware of that do solve this are Scala, Rust, Kotlin to some extent, Haskell... languages which do not have a reputation of being stable, easy to learn, easy to read and understand, compile quickly, etc.
Kotlin is definitely the odd man out in that list.
Though the reputation for those other languages mainly stem from their embrace of way more advanced concepts rather than null handling via optionals, I think that this specific concept makes it easier to learn/read/understand (though not compile quicker)
Just because person A thinks this is hugely important, doesn't mean person B has to agree, or that B is a bad developer.
Mind showing us the source for that?
Go wasn't made for incompetent developers. I'm fairly certain that people who land a job as devs at Google are pretty competent.
Go was made to facilitate rapid onboarding, easy digestion of large codebases, and working efficiently in large teams where people are guaranteed to have widely different educational backgrounds, experiences and ideas about programming.
That's why the language has to be simple, obvious, and be focused on readability. That's also why Go is strongly opinionated.
I think Go could still work without completely neutering the type system.
Algebraic Data Types and pattern matching are not difficult... python already has them.
You can't make up that other devs' opinions / preferences are identical to yours just because they use the same language, there are other important factors in play (e.g., if your company is using Go, then you'd be more productive in it and be more likely to choose to contribute in it even it Go is less productive as a language)
> code is straightforward and easy to read
I have to disagree. I don't want to read 3 lines out of four that are exactly the same. I don't want to read the boilerplate. I don't want to read yet another abs or array_contains reimplementation. Yes it's technically easy to read, but the actual business logic is buried under so much noise that it really hinders my capacity to digest it.
> the compiler is blazing fast
much agreed, that is my #1 pain point in rust (but it's getting better!)
> and it's concurrency model is probably the most convenient I have ever seen
this so much. this is what I hate the most with go: it pioneered a concurrency model and made it available to the masses, but it has too many footguns imho. this is no surprise other languages picked channels as a first class citizen in their stdlib or core language.
> Go trades "being-modern" for amazing productivity.
I don't think those two are incompatible. If we take the specific point of the article, which is nil pointers, Go would only have to import the sum types concept to have Option and maybe Result as a bonus. Would this translate to a loss of productivity? I don't think so. (oh and sum types hardly are a modern concept)
Also, there may be a false sense of productivity. Go is verbose, and you write a lot. Sure if you spend most of your time typing then yes you are productive. But is it high-value productivity? Some more concise languages leave you more time to think about what you are writing and to write something correct. The feeling of productivity is not there because you are not actively writing code most of the time. IIRC plan9 makes heavy use of the mouse, and people feel less productive compared to a terminal because they are not actively typing. They are not active all the time.
I get what you're saying, and I'm glad you are having such a good experience with it. Disclosure, I am not talking down to any language here...in fact I actually like Rust as a language, even though I don't use it professionally.
I am just saying that Go is incredibly easy to learn, and I don't think there are many people who disagree on this point, proponent of Go or not.
> I have to disagree.
We'll have to agree to disagree then :-) Yes, the code is verbose, but it's not really noise in my opinion. Noise is something like what happens in enterprise Java, where we have superfluous abstractions heaped ontop of one another. Noise doesn't add to the program. The verbose error handling of Go, and the fact that it leaves out a lot of magic from other languages doesn't make it noisy to me.
> I don't think those two are incompatible.
Neither do I, but that's the path Go has chosen. It may also have been poorly worded on my part. A better way of putting it: Go doesn't subscribe to the "add as much as possible" - mode of language development.
> But is it high-value productivity?
Writing the verbose parts of go, like error checking, isn't time consuming, because it's very simple...in fact, these days I leave a lot of that to the LLM integration of my editor :-)
Is is high value? Yes, I think so, because I don't measure productivity by number of lines of code, I measure it by features shipped, and issues solves. And that's where Go's ... how do I say this ... obviousness really puts the language into the spotlight for me.
This is my sense. "False sense of productivity" is an accurate statement - I've also found that it seems to be for a very specific (and not necessarily useful) definition of "productive", such as LOC per day.
It's not as bad as dynamic languages like Python, but very frequently Go codebases feel brittle, like any change I make might bring down the whole house of cards at runtime.
Isn't it basically just what Cilk did, but with fewer feaures?
Go, for whatever reasons, has gotten people using it.
Same. Started a company, onboarded just about everyone to Rust. It went very well.
At my employer we have a pattern of promoting people who have done things like write a proprietary application gateway. The dev got a couple promotions and moved on to another company and we got stuck maintaining a proprietary application gateway with a terribly messy configuration and poor observability.
None of the Google fresh hires I know personally are stupid. They are talented people who could be just as productive in a C++ or Java codebase. Maybe even better when you have features like Java's streams or C++ templates to throw at non-trivial problems. They might need more time, but it's something easily budgeted for. If new hires have to be productive from the first day, that's a problem the company has created and not the employee. If other languages have too many ways to do something, just enforce only using a few of them, teams have and continue to do that.
I use Golang in my current job. The library ecosystem seems fine. But even as a "new hire", the language frustrates me sometimes. Go's concurrency is "easy", but has a minefield of problems. Just off the top of my head, for-loop semantics [which to Golang's credit, is being fixed but it is absolutely a breaking change], just being able to copy a mutex by accident. These are bugs I've written and not had fun tracking down. In a year I'll have all these footguns memorized, but I could also have spent a year getting better at any other language. Even at my experience level, the Rust compiler gives me enough grief for me to know that when it's happy, whatever I've written will work. Nothing about Golang gives me that confidence.
And this is exactly what Go avoids, in my opinion and experience.
Unless someone shoehorns Go into an Enterprise Java style (which, sadly, is possible, and sometimes done), the problems you listed with the Java codebase either don't exist in Go, or are orders of magnitude less painful to deal with, even in large Go codebases. Plus, the toolchain is pretty obvious, because most of it ships with the language.
And while my argument mentions new hires specifically, because the impact is most visible with them, this is just as important for mid and senior level developers; yes Go is sometimes more verbose (although enterprise Java and lots of C++ code still runs circles around Go in that regard) than it's contemporaries, it is also obvious. There is little magic, there is little action at a distance, and the opinionated style of the language discourages superfluos abstractions.
I have used quite a few languages so far in my career. Go is the first where I was able to comfortably read and understand std library code within the first week of learning the language.
> Just off the top of my head, for-loop semantics [which to Golang's credit, is being fixed but it is absolutely a breaking change],
It is technically a breaking change. Practically, it isn't, because there simply are no examples of production code in the wild that rely on this unintuitive behavior (As mentioned multiple times in the discussion on Go's issue tracker, the dev team did their research on that), and code that implements the (very easy) fix, will continue to work after the upcoming change.
> Nothing about Golang gives me that confidence.
My experience is different. I know that most of the problems the Rust compiler complains about will be handled by the fact that Go is GC'ed, and most of the rest I avoid by relying on CSP as my concurrency model (Can't accidentially copy a mutex if there's no mutex ;-) )
All of those could be checked or irrelevant, I have no idea.
Looking through the first few pages, most of these panics are easy to audit, and are infallible or in contexts where it doesn't matter (internal tooling, etc). That's a pretty stark difference to every single reference being a potential landmine.
Like, one of the first files has only .unwraps in the comments (like a dozen of them in a file), some are infallible uses, some are irrelevant-to-runtime tooling, etc.
But anyway, "some" is a lot smaller than "all". Just like some of memory safety issues would also have happened since you can still use unsafe in Rust, yet it's still a big step forward in reducing those issues in the ugly real world
If you think about it a bit, given that bugs are relatively rare in a mature project, it's going to be difficult to find a use of unwrap that's definitely bad.
So let me charitably ask directly: have you looked through all the examples at least on the first couple of pages? And if you have, what % of instances is relevant to your point?
It would be similarly difficult to trawl through the source of the Go compiler and find definitely bad instances of pointer dereferencing. So does that mean that it’s not actually a problem in Go either?
Getting good type errors without requiring type annotations seems like a win over languages that are annotation-heavy. Normally I’d be skeptical about relying on type inference too much over explicit type declarations, but maybe it’s okay for this problem?
This is speculative, but I could see this becoming another win for the Go approach of putting off problems that aren’t urgent. Sort of like having third-party module systems for so many years, and then a really good one. Or like generics.
Is this just a symptom of having a lot of engineers and they keep churning code, Golang being verbose or something else. Hard time wrapping my head around Uber needing 90+ million lines of code(!). What would be some large components of this codebase look like?
There's a lot of overlap and some invalid combinations, but you're still left with a huge number of combinations where Uber must simply work. And every time you add a new thing to this list, the total number of combinations grows polynomially.
(Also, Go is slightly more verbose than most languages. I think that's a feature and not a bug, but it's one more reason.)
A lot of people seems to gravitate toward languages with less dense cognitive load. I have learned to love kotlin, but its also a super dense set of syntax to power it's very expressive language.
But if you really cannot afford to return more than one bit of information, do `func foo() (*T, bool)`.
Result<T,E> does this. I forget exactly why Result is actually different from, and in fact superior to, `func foo() (*T, error)` but IIRC it has to do with function composition and concrete vs generic types.
(*T, error) is either T (non-nil, nil), or error (nil/undefined, non-nil), or both (non-nil, non-nil), or neither (nil, nil). By convention usually only the first two are used, but 1) not always, 2) if you rely on convention why even have type system, I have conventions in Python.
Leaving aside pattern matching and all other things which make Rust way more ergonomic and harder to misuse, Go simply lacks a proper sum type that can express exactly one of two options and won't let you use it wrong. Errors should have been done this way from the start, all the theory was known and many practical implementations existed.
Because *T could be nil or non-nil, it seems like the analogy would be a nullable type in the Result<>. In Go, (T, error) would only have the states (non-nil, nil) and (non-nil, non-nil) if T is not a pointer. Still, the Result type seems better to me because the type itself is encapsulating all of this (and the error I guess cannot be null).
The second part is that it's reified as a single value, so it works just fine as a normal value e.g. you can map a value to a result, or put results in a map, etc... , language doesn't really care.
And then you can build fun utilities around it e.g. a transformer from Iterator<Item=Result<T, E>> to Result<Vec<T>, E> (iterates and collects values until it encounters an error, in which case it aborts and immediately returns said error).
Effectively,
instead of
result, err := doSomething()
if err != nil {
return nil, err
}
result := doSomething()?Types for which Try is implemented can Try::branch() to get a ControlFlow, a sum type representing the answer to the question "Stop now or keep going?". In the use you're thinking of where we're using ? on a Result, if we're Err we should stop now, returning the error, whereas if we're OK we should keep going.
And that's why this works in Rust (today), when you write doSomething()? the Try::branch() is executed for your Result and resolves into a Break or a Continue which is used to decide to return immediately with an error or continue.
But this is also exactly the right shape for other types in situations where failure has the opposite expectation, and we should keep going if we failed, hoping to succeed later, but stop early if we have a good answer now.
...and then they add more syntax sugar to partly sweep the complexity under the rug. I like Rust as much as the next person, but I'm apprehensive about how this will play out.
I have worked with Rust Option/Rust types and found them extremely unergonomic and painful. The ?s and method chains are an eyesore. Surely PLT has something better for us.
Hence why the language is full of gotchas like these.
Had it not been for Docker and Kubernetes success, and most likely it wouldn't have gotten thus far.
Kinda...
They made the language easier and quicker to write a compiler, but harder to write programs in, and it doesn't look like that will change in Go 2.0.
I appreciate both languages, and of course Swift feels like what you’d pick any day.
But, after using both nearly side by side and comparing the experience directly, I’ve got to say, I’m so much more productive in Go, there’s SO much less mental burden when writing the code, — and it does not result in more bugs or other sorts of problems.
Thing is, I, of course, am always thinking about types, nullability and the like. The mental type model is pretty rich. But the more intricacies of it that I have to explain to the compiler, the more drag I feel on getting things shipped.
And because Go is so simple, idiomatic, and basically things are generally as I expect them to be, maintenance is not an issue either. Yes, occasionally you are left wondering if a particular field can or cannot be nil / invalid-zero-value, but those cases are few enough to not become a problem.
Genuinely curious what's so much of business logic is for.
"All" a kernel does (for some very large value of "all") is schedule userspace programs and manage the system's physical resources (memory, disk, devices). You can reach a point where a kernel is done, in the sense that it meets those basic needs with an acceptable level of performance. Kernel developers don't make extra money for every new feature they add - if the system is good enough, then it's good enough.
As for "per locality business rules differ that's why so many lines of code.." seems like you can have a policy engine+DSL (JSON or YAML or custom policy language and engine) thus your code base shouldn't balloon to almost 100 million limes of code...
There are entire teams that are working on just internal services that connect some internal tools together.
There was also very little effectivity and efficiency in the era of cheap capital so there were tons of talent wasted on nonsense. Uber built their own slack for a while!! (before just going to mattermost)
People always ask who actually makes money on Uber... I think it's not the cab drivers, not the investors, who makes money is the programmers. It's a transfer of money from Saudis to programmers.
Well it was, anyway.
mad
What I would really like golang to have is way to send a “last gasp” packet to notify some other system that the runtime is panicing. Ideally at large scales it would be really nice to see what is panicing where and at what time with also stack traces and maybe core dumps. I think that would be much more useful for fixing panics in production.
There was a proposal to add this to the runtime, but it got turned down: https://github.com/golang/go/issues/32333 Most of the arguments against the proposal seem to be that it is hard to determine what is safe to run in a global panic handler. I think the more reasonable option is to tell the go runtime that you want it to send a UDP packet to some address when it panics. That allows the runtime to not support calling arbitrary functions during panicing as it only has to send a UDP packet and then crash.
I could see the static analyzer being useful for helping prevent the introduction of new panics, but I would much rather have better runtime detection.
I tried this with a medium sized project and some unexpected code that could panic 3 functions away from the nil.
Link to the source, or better yet, never link at all to anything related to Uber.
I do recommend the Go team to find a way to these tools to run before it complies, just doing go build while going through these tools first goes a long way than just using scripts