Sub-millisecond GC pauses in Go 1.8
groups.google.com
groups.google.com
So... not worst-case. :)
Regardless, very impressive achievement, especially given that they've been able to do it without a JIT or exotic tricks like Azul.
Worst case in absolute time always requires ignoring pathological cases.
We don't care about worst-case latency in practice, and average case is often ignored as well. We look at 99.9% latency numbers or things like that. Worst-case is for people designing pacemakers or rockets, that's not what we are doing here.
Considering this code will likely be used in IoT, yes, this is what we’re doing here.
Nor is my sous vide machine or anything else I've seen as an IoT device. I wouldn't call say my oscilliscope an IoT device even if it can connect to wifi. It's a scientific device with cloud upload.
There was an interesting talk at the Chaos Communications Congress a few years back.
Gp post was not talking about security we're talking about Real time computing. If it's running on IoT hardware, eg rpi running linux, it's likely not doing real time os stuff because linux isn't really a real time os.
And I still don't classify pacemakers as IoT. Just because you want to slap a vague acyronym on any and every "connected" embedded device doesn't mean I agree.
That stuff is IoT in every definition of the word.
Monitoring might be fine, but installing something in the body that can kill you and then running it on node.js doesn't seem like a good idea.
...maybe an anonymous usage program?
...maybe their typical use case? (which is definitely not everyone elses)
...maybe the typical use case of Google employees using Go for their single-computer tasks? (which would be closer to a "typical user" as in people who read HN)
...an average of those three, or perhaps even more, use cases? wighted in some way?
But if done correctly, optimizing the 90% typical problem by 20%, gives you a general 18% improvement, far more than even making disappear the least important 10% of the runtime (which some bad optimization efforts tend to focus on...).
Making software faster is always about knowing where to work, and then focusing exactly on that. Nothing wrong in attacking the "typical [anything] case".
Pretty harsh for an informal post on a mailing list, its not like this was a formal proposal or an announcement.
Talking about why "typical" lacks meaning is fair, but calling the OP negligent is not.
You can't very well tolerate an "atypical" GC pause when you're firing a laser or radiation pulse, doing engine control, or even updating a display in a flight instrument setting.
In my, admittedly maybe lacking, understanding amortization doesn't really work that way. "Amortized worst case" (for example) means that you can still bound the worst case[1], but it's just not necessarily going to be a very accurate bound. Obviously, amortized complexity doesn't tell you "<X ms" right off the bat since it deals in abstract "operations", but if you have known worst case bounds for all the "operations", then an amortized bound for a given operation will give you something equivalent to "<X ms".
[1] I mean, it's a common proof technique to actually have a non-negative cost function (+ invariant) and postulate/derive an upper bound on that, so... what gives? What's your reasoning here?
I somehow confused myself by thinking of it in terms of one of the proof techniques for amortized worst case where you derive a fixed upper bound for any "n". Of course this is a much stronger property than needed.
[1]: https://github.com/golang/go/blob/8f81dfe8b47e975b90bb4a2f8d...
I'm not sure that's what Go does, but I'm not aware of a faster way on standard hardware.
- https://github.com/golang/go/issues/10958It's much more expensive and complex to take an erratic latency operation and bring it down by throwing on more resources. As far as I can tell, the normal design course is making sure all your major actions are either pure or idempotent allowing parallel (and redundant!) requests to be made... which is a large (worthy, but large) engineering effort, and then we're talking about scaling to 2x or more just so you can make that redundant-request thing your default behavior.
For OpemJDK/Hotspot there will be a <100ms GC in Java 9: http://openjdk.java.net/jeps/189
Here is how Zing use flags:
-XX:+UseGenPauselessGC - Generational Pauseless GC (GPGC) optimized for minimal pause times.
Doing the manually triggered & staggered GC trick on a pool of machines you control can give you very low latency guarantees, since no production request will ever hit a GC-ing JVM.
However, 100, even 200ms pauses get "lost" during network coms. But users tend to notice if a page takes 30 seconds to load rather than 1 second (differences i've seen optimising Javas GC, don't know at all how Go and Java compare).
That kind of difference would be very expensive to solve using more servers.
Sever side zero memory leaks absolute requirement and "real time" responses rarely a requirement. triggering gc during init and/or shutdown of a component often enough.
Building a CNC machine - every tick is valuable, but as long as it can run a job for 2 days before crashing no one will notice if it leaks memory like a sieve when you run a calibration routine.
You can also "just" remove servers that are currently in GC from your pool, and have high throughput in a service with reliable latency.
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gc...
Here are relevant points:
1. If (a) peak application performance is the first priority and (b) there are no pause time requirements or pauses of 1 second or longer are acceptable, then let the VM select the collector, or select the parallel collector with -XX:+UseParallelGC.
2. If response time is more important than overall throughput and garbage collection pauses must be kept shorter than approximately 1 second, then select the concurrent collector with -XX:+UseConcMarkSweepGC or -XX:+UseG1GC.
From Mechanical Sympathy Blog [1]
" G1 is target driven on latency –XX:MaxGCPauseMillis=<n>, default value = 200ms. The target will influence the amount of work done on each cycle on a best-efforts only basis. Setting targets in tens of milliseconds is mostly futile, and as of this writing targeting tens of milliseconds has not been a focus of G1. "
So in Java world we are talking 100s of milli seconds of worst case which is 3 order of magnitude higher than Go.
1. http://mechanical-sympathy.blogspot.com/2013/07/java-garbage...
The HotSpot designers have (correctly, IMO) observed that server use cases typically prefer throughput to latency.
On a meta note, I wish people would focus on the actual algorithms instead of taking cited performance numbers out of context and comparing them head to head. The Go GC uses the same concurrent marking and sweeping techniques that CMS does. This change to avoid stop the world stack sweeps is something that no HotSpot collector does. But it's counterbalanced by the fact that Go's GC is not generational. Generational GCs change the balance significantly by dramatically improving throughput.
The question is what use cases can tolerate large throughput hits but not few msec pause times (G1 can also do pauses of a few msec in many cases, I see hundred msec pause times on very large heaps, but not desktop sized heaps).
I suspect there are very few use cases. The G1 team seems to be focusing on scaling to ever larger heaps right now, like hundreds of gigabytes in size. They're relatively uninterested in driving down pause times as Shenandoah is going to provide that for the open source world and Azul already does for the $$$ world.
In general Oracle JDK uses order of magnitude more memory and order magnitudes higher GC latency compare to Go. IMO it is quite useful to remember when deciding on language on new projects. If hotspot people think that these numbers are not true I am sure they would let people know.
* Go's GC is completely non-compacting while G1 is for all generations and CMS for all but the old generation.
* Shenandoah will offer lower pause times by doing concurrent compacting, at the expense of throughput.
* Azul already offers a pauseless compacting collector for hotspot
1. http://www.slideshare.net/RedHatDevelopers/shenandoah-gc-jav...
About the memory footprint: The runtime is larger, so there's a baseline cost you have to pay for the JITs. But baseline != linear cost.
If you have a long-running applications that run more than a few seconds and crunch more than a few MB of data those numbers would change.
So unless you're using the JVM for one-shot script execution those benchmarks are not representative. And if you do then there would be certain optimizations that one could apply.
> There must be some end user (positive) impact of memory compaction in Java.
Fewer CPU cycles overhead per unit of work, i.e. better throughput, which does seem to be an issue for gogc[0]. No risk of quasi-leaks in the allocator due to fragmentation. Reduced cache misses[1]
[0] https://github.com/golang/go/issues/14161 [1] http://stackoverflow.com/q/31225252/1362755
I do not think so. Here is explanation about Java memory bloat in typically used data structures.
https://www.cs.virginia.edu/kim/publicity/pldi09tutorials/me...
Also, `regex-dna` will probably benefit from compact strings in java 9.
And some of the longer-running benchmarks are in fact faster on java, so I think that supports my point about warmup.
Yes, apart from reverse-complement, regex-dna, k-nucleotide, binary-trees.
> Also, `regex-dna` will probably benefit from compact strings in java 9.
Do you actually know that com.basistech will re-write their library to use compact strings?
> I think that supports my point
I think that's called cherry picking.
-- Benchmarks are of limited use.
-- Are JVM startup costs significant?
http://benchmarksgame.alioth.debian.org/sometimes-people-jus...
go version go1.7 linux/amd64
OP "Sub-millisecond GC pauses in Go 1.8"
See http://mattwarren.org/2016/08/08/GC-Pauses-and-Safe-Points/ for all the gory details
100 microseconds is quite a long time in CPU time for a single-core these days, and proportionally longer with multi-core, or say in GPU time. However taking into account the VM runtime environment, this wouldn't make Go's feat any less impressive.
malloc()-style allocation gives you precision over how much you want, and this is most interesting to a memory-constrained system trying to allocate amongst many different processes(the original Unix use-case). But willy-nilly use of malloc() and free() leaves you with lots of fragmentation, as well as a larger attack surface for memory errors. What the actual allocation algorithm does is out of your hands, too, at least if you're depending on the OS allocator(you can always go write your own and supply it with a giant heap to munch on, and this may occur when you need tuning for individual allocation scenarios).
In the right scenario, a GC won't do too much differently from a manual allocator(there are lots of optimizations that could bring the allocation to same-or-negligible runtime), but as we all know, right scenarios are something you can't always count on. A GC does, however, greatly simplify the memory management of a long-lived process since it can do nifty things like automatically compact the heap.
IME, a mix of stack, GC, and some arenas in the form of growable arrays, is absolutely fine for the needs of most applications. Quite often this last requirement creates a sticking point, though, where the language disallows value semantics for arrays of objects, and then you can no longer assume they're allocated linearly, plus the GC is taxed with additional references to trace. In those cases, if I have them I use arrays of primitive values as crude containers for an equivalent optimization. Either way, it's very annoying and creates some awful code, because those huge batches that I'd like to put in arenas also tend to be the bottleneck for the GC.
Is that true? I thought nursery allocation was usually as fast if not faster than stack allocations.
Not because it is very high, but because stack allocation/deallocation is so very simple.
Mesa/Cedar, Modula-2+, Modula-3, Oberon, Oberon-2, Active Oberon, Component Pascal, Eiffel, D, Swift
I can list other ones that are actually quite obscure.
In any case, I also had D and Swift on the list, which are quite actual.
As for the rest, I don't have anything to add to weberc2's answer.
No, it's not. For short-lived objects, at least on the JVM, allocation is a pointer bump, and collection is free, because unreachable objects are not even walked. Stack allocation doesn't beat that by much.
The difference is if the memory management is manual, you have the ability to clean it up and reduce that overhead toward 0%.
If it's a system-enforced GC, you are limited in what you can do.
Given that the overhead become 0 in either.
Is it sometimes harder to write zero alloc code in GC'd languages? Sure but its not impossible.
In Java for instance the difference in performance compared to C++ comes from memory layout ability not memory management.
We need more modern languages with AOT compilation, ability to allocate on the stack and use value types.
I like C++ and have spent quite a few years using it, before settling on memory safe languages for my work.
Many C++ presentations seem to forget on purpose that there are memory safe programming languages, which offer similar capabilities for controlling memory layouts, thus presenting the language as the only capable of doing it.
Modula-3 is an example, or Swift and D for something more actual.
There are many use cases where C++ is still used where the use case at hand actually didn't any a real need for it.
For example, on my area of work, C++ has been out of the picture since around 2006. We only use it when Java or .NET need an helping hand, which happens very seldom.
On the mobile OSes for example, C++ is only relevant as possible language to write portable code across platforms, but not so much for those that decide to focus on just one.
There Swift, Java and C# are much better options.
For those in HPC, languages like Chapel and X10 are gaining adepts.
Also as an C++ early adopter (1993) I remember being told by C developers something similar to what you are saying in regards to tool maturity.
Now around 30 years later, their compilers are written in C++.
[1] shameless plug: https://github.com/duneroadrunner/SaferCPlusPlus
They could have avoided that whole problem by providing an API like DontCollectThisRoot, but instead they (and a lot of programmers) chose to pretend the GC was good enough to treat like a black box.
https://samsaffron.com/archive/2011/10/28/in-managed-code-we...
So now we are stuck with modern languages having to bring those ideas back to life.
Not everyone is writing games...
Perhaps I'm misunderstanding, but do many C programmers understand not only the current state of malloc at any given moment in their code but exactly how it works?
I think not.
A lot of the things you do in C++ to reduce memory management overhead are the same things you can do in Java, C#, and Go to reduce memory management overhead. That effort is neither special nor universal.
HLLs often have to be careful about using language features that unexpectedly create garbage, but in terms of actual management and collection it's not like ANY competent modern language is slow at it.
People often seem to neglect the fact that Java is still surprisingly fast despite spending lots of time on memory management just because many developers are so insensitive to how things alloc. Modern GC systems can manage allocs as well as deallocs, so with care from the programmer and the runtime authors you can reach the kind of performance people talk about as critical for "embedded systems" (even though in practice SO many things do not deliver on this promise in shipped products!).
This is unrelated to GC.
> The cost of having GC is that you pay with higher memory usage for doing GC in batches thanks to which you do not have to pay for single deallocations. So this performance advantage cannot be used in embedded space because there is no free memory for it, you would need to GC all the time which would kill the performance.
The same is true for C. You don't get to make frequent allocations for free in any language. You have to trade space for performance; in GC-land, the answer is tuning the collector. In Go, this is one knob: the GOGC env var.
A very well know, Aonix, now part of PTC, used to sell real time Java implementations for military scenarios like missile guidance systems.
For example, languages that use closures have to have very smart compilers or even innocuous functions can create implications for the working set size, which puts the allocator and deallocater under a lot more pressure.
And that's not even the most subtle problem you might run into! A common cause of memory constraints in Java 1.7 and earlier stemmed from subarrays of large arrays. Java made a fairly innocuous decision regarding methods like String.substring that ends up biting a lot of people later on, even as it is the right decision for a slightly different set of performance considerations.
Good programmers understand how malloc works. What, are you kidding, or am I misunderstanding?
Performance-oriented programmers do not use malloc very much. As you say, you can also try to avoid allocations in GC'd languages. The difference is that in a language like C you are actually in control of what happens. In a language that magically makes memory things happen, you can reduce allocations, but not in a particularly precise way -- you're following heuristics, but how do you know you got everything? Okay, you reduced your GC pause time and frequency, but how do you know GC pauses aren't still going to happen? Doesn't that depend on implementation details that are out of your control?
> even though in practice SO many things do not deliver on this promise in shipped products!
But, "in practice" is the thing that actually matters. Lots and lots of stuff is great according to someone's theory.
First of all, it's not like most mallocs DON'T have heuristics they're following. Without insight into what it wants to do it is equally as opaque to how Java or the CLR manages memory.
And your behavior absolutely can and does influence how much things are allocated, deallocated, and reused. If you think that the JVM cannot be tuned to that level, you're dead wrong and I can point to numerous projects written for virtual machines that reach levels of performance that are genuinely difficult to reach no matter your approach.
> Good programmers understand how malloc works.
"Good programmers know how their GC works. What, are you kidding, or am I misunderstanding?"
> But, "in practice" is the thing that actually matters.
"In practice" Kafka is the gold standard of pushing bits through distributed systems as fast as possible. "In practice" distributed concurrency systems (that are often the speed limit of anything you want to build on more than one computer, e.g., etcd, Zookeeper, Consul) are I/O limited long before their collected nature impacts their performance.
And if we can eventually liberate ourselves from operating systems that give priviledged status to C and C++, that ecosystem will diminish further because its performance benefits come at too high a cost, and are generally oversold anyways.
> "Good programmers know how their GC works. What, are you kidding, or am I misunderstanding?"
I think you are not understanding what I am saying.
You link your allocators into your code so you know what they are. You see the source code. You know exactly what they do. If you don't like exactly what they do, you change them to something different.
A garbage-collector, in almost all language systems, is a property of the runtime system. Its behavior depends on what particular platform you are running on. Even 'minor' point updates can substantially change the performance-related behavior of your program. Thus you are not really in control.
As for your other examples, apparently you're a web programmer (?) and in my experience it's just not very easy for me to communicate with web people about issues of software quality, responsiveness, etc, because they have completely different standards of what is "acceptable" or "good" (standards that I think are absurdly low, but it is what it is).
In my experience, most C/C++ devs know what malloc/free or new/delete does, but how? They don't care as long as it works and doesn't get in their way. Sure in larger applications, the allocator/deallocator can consume quite some time - but even then it rarely is the bottleneck.
I happen to have a more hands-on experience with allocators, I had to port one a long time ago, but in C or C++, I rarely knew how the one I was using was implemented (except for the one I ported). Seeing the source code? Sorry, that's not always available and even if it is, not too accessible - not that many devs actually ever look into the glibc code...
And linking your allocator? Most of the times you just use the default-one provided by your standard library - so that happens 'automagically' without most developers realizing this. I yet have to see a modern C or C++ app that specifically has to link it's own allocator before it could actually allocate memory. Most compilers take care of this.
For most stuff I do - I like gc's. In most real-world situations, they are rarely the bottleneck, most applications are I/O bound. For most stuff, a GC's benefits outweigh it's disadvantages by a huge margin. And if the gc could be become a bottleneck, you should have been aware of that up front, and maybe avoid something using a GC, although I'm not a fan of premature optimization.
Not going with a generational GC is a choice that runs against what has been accepted practice for decades. It may of course work out, but it's what I'd consider an experiment.
Compare that with Java where rather sofisticated escape analysis does not help much besides allowing to return several things cheaply from a function. Typical code there just does not follow stack-like allocation patterns.
See the "SustainedLowLatency" mode for something very similar to what Go does (although .NET's GC, unlike Go's GC, is generational, which is a significant difference).
Their experience was actually pretty instructive; with their first release of the new GC (a proficient but basic generational collector) they still weren't beating Boehm all the time, and usually weren't beating it by much. Given its constraints, Boehm's GC is impressively optimized when you run it in semi-precise mode.
See http://mattwarren.org/2016/08/16/Preventing-dotNET-Garbage-C... for the full details
http://stackoverflow.com/a/14277068
tl;dr - "arrays of structs are really the only bulk storage method available in C#"
E.g. Java vs Go vs C# vs Actionscript vs Javascript?
Also on some platforms it could even be Go-to-hardware.
AAA game developers only moved from Assembly to C, Pascal when forced to do it.
A few years after they moved from C to C++ when the SDKs started to be C++ only.
Similarly they will only move from C++ when forced to do so, and they only major complaint is related to build times.
I bet most are willing to wait until C++20 for modules than switching to another language, even if some studios manage to have some hits developed in safer languages.
On the other hand, Web and mobile games have lived with the consequences of a managed runtime for many years now. There are limits in how much processing power is available there, but it ultimately just diverts developer attention towards other things like a more robust backend, faster turnarounds, and other general workflow improvements independent of scene fidelity.
However, it may be good enough for games at well below 1% of your time budget for a 60 fps game assuming cache locality is good enough so you don't waste too much time fetching from main memory.
http://researcher.watson.ibm.com/researcher/view_group.php?i...
Already in 2012, so even if not yet there, I think Go can get there, specially since you can take care to just stack allocate and minimise GC use during the hot tub path.
In HFT systems in GC'd languages or not, you don't allocate on the hot path so GC times are immaterial.
That's not to say you couldn't potentially write an OS or an embedded system in Go (I mean, you can write OS's in Lisp if you really want) but I doubt it would be fun and I doubt anybody would recommend it. You definitely won't be writing idiomatic Go without a lot of extra pieces that you can't really afford in those situations.
Can you write an OS kernel in Go ? no Go's runtime still depends on a OS. And whoever talked about Go as a system language didn't have kernels in mind, but "network infrastructure".
Here you can learn how those guys at Swiss Federal Institute of Technology in Zurich did it.
http://people.inf.ethz.ch/wirth/ProjectOberon/index.html
Basically the runtime is done bare metal thus becomes the OS kernel.
Oh, and Oberon wasn't just a research OS, it was actually used across the computing department by many of its employees.
Similarly, can you write an OS kernel in ISO C? No, you still need some assembly or runtime support. For example, ISO C doesn't have any notion of making syscalls or returning from the kernel to a caller or for setting up page tables.
Any argument why go isn't suitable for systems programming along these lines should be about how much and what kind of runtime is acceptable for a systems programming language.
A (fairly popular, I think, but certainly not universally agreed upon) argument could be that systems programming languages cannot have garbage collection because it should be possible to implement a garbage collector in them, and doing that in a language that already has one is possible but silly.
That is necessary even with a concurrent garbage collector because a garbage collector that allocates in its own heap may hang itself (propaganda allocates; gc triggered; gc allocates; gc triggered to satisfy the allocation; new gc triggered; etc.) . Or do Go's developers accept this risk and live with it?
It's left to the compiler actually. Programmers can't be trusted.
The runtime does not implicitly generate garbage (like arbitrary Go code). It is compiled with a mode that fails compilation if something can't be stack-allocated. When heap allocation is necessary, it is requested manually. However, the memory returned is garbage collected, as usual, there is no free.
http://people.inf.ethz.ch/wirth/ProjectOberon/Sources/Kernel...
It has the same features as Oberon for systems, programming which was used to build quite a few Workstation OS at Swiss Federal Institute of Technology in Zurich (ETHZ), by Niklaus Wirth.
https://en.wikipedia.org/wiki/Oberon_(operating_system)
You can read the source code here in the 2013 revised edition of the 1992 book.
http://people.inf.ethz.ch/wirth/ProjectOberon/index.html
The only thing missing from Go versus what Oberon offered is register access on the unsafe package, but even then can be sorted out with an extension or a few calls into Assembly.
Oberon-07, which is even more minimalist that either Go or the original Oberon is sold by Astrobe for bare metal programming on cortex M boards.
http://www.astrobe.com/default.htm
Go is already bootstraped into itself, so the writing compilers is taken care of.
It is used by Docker and Kubernetes for managing containers.
It just needs someone writing an OS with Go for its Master or PhD thesis, or even just port Oberon source code, which is freely available into Go.
Go 1.8 is moving to a hybrid write barrier in order to eliminate stack rescanning. The hybrid write barrier is equivalent to the "double write barrier" used in the adaptation of Metronome used in the IBM real-time Java implementation.
Some benchmarks:
https://github.com/golang/go/commit/bd640c882a8dcb88a0497770...
Basically everything that's problematic with returning a status int in C, but all new, hip, and backed up by Rob Pike's pseudointellectual bullshit and a bunch of Silicon Valley 20somethings.
They could at least, you know, have an Either type or something. Anything?
When did this separation become law ? What if the "application logic" requires recovery ?
>They could at least, you know, have an Either type or something
(int64, error) in func ParseInt() (int64, error) is your Either type. And checking if you got the "left or the right side of the Either" is IMHO much shorter and clearer than in Scala.
https://golang.org/pkg/strconv/#ParseInt
http://www.scala-lang.org/api/rc2/scala/Either.html
>backed up by Rob Pike's pseudointellectual bullshit and a bunch of Silicon Valley 20somethings
Why the ad hominems ?
How ecactly is having to check twice the amount of cases an improvement (note btw, that checking for Left/Right is doing it wrong)?
(int64, error)
Either gives you exactly the two you want.
(value, no error)
(value, error)
(no value, error)
(no value, no error)Error as type or error as values ? the std lib promotes error as values (i.e. check equality) instead of errors as type (i.e. check the type). Go error system WAS written with errors as value in mind. There is no point having errors in place of exceptions if errors were intended to be used as types (which they are not, as said previoulsy). Basically developers are implementing their own mediocre exception system on top of Go errors.
The error as value thing made sense in C 30 years ago, it doesn't in a language created less than 10 years ago.
There are a lot of C inspired patterns in Go that make the language half modern/ half dated in strange ways. That's fine when one comes from C though, that isn't when one comes from anything remotely modern. But I guess it's why Go is successful, it's basically C with garbage collection.
Wow, again, that's not the problem here. Errors in the standard libraries are defined at values. There is no point testing it as interfaces, it will not give you the nature of the error, since they are all defined with fmt.Errorf . Do you understand now the problem ? the problem is being consistant across codebases between errors are values and errors as types.
I have already used libraries that catch the exceptions internally and return some kind of error value instead. :(
No, an error implements the error interface. It means that it can be a value of any type that implements the constraint of having an Error method.
It doesn't matter what a value implements if you don't test its content by type. Std lib errors are meant to be tested by value, not by type. It has nothing to do with interfaces again. When you get a std lib error in order to know what error it is you compare it to an exported value, not its type. I don't know why you keep on insisting on interfaces that's not the core of the issue here.
https://golang.org/doc/faq#nil_error
Of course, there are ways around this such as returning a struct, but then that's no longer compatible with the error interface.
Which your struct can easily fulfill. That's the beauty of Go interfaces.
This is perfectly valid and doesn't cause the nil issue:
return someErrorStruct{}
There's nothing absurd about it; interfaces are a reference type. If you have a reference to a reference, then checking the "outer" reference for nil doesn't tell you anything about the nullity of the "inner" reference. The advice is just a special case of "don't needlessly use pointers-to-pointers".
I say this as someone who uses Go daily for my work and mostly likes it (despite the many warts).
It's exacerbated by the fact that Go chose to make things like maps, slices and channels pointers, too. It has an excellent philosophy about zero values (so you can't get nil strings, even though they are pointers internally), yet goofed when it came to something as intuitive as "var foo []string", and claimed this behaviour to be a feature. The (nil, nil) interface is just icing on a crappy cake.
The fact that such a new language as Go doesn't have a way to express missing values safely should be disappointing to developers.
> The (nil, nil) interface is just icing on a crappy cake.
The same problem exists with languages without nil. For example, if you choose to do the stupid thing and return Option<Option<error>> when you only need Option<error>, then checkout the outer Option<> for None is not going to magically guarantee that the inner Option<> is not None.
> It has an excellent philosophy about zero values (so you can't get nil strings, even though they are pointers internally), yet goofed when it came to something as intuitive as "var foo []string", and claimed this behaviour to be a feature.
What are you talking about? nil slices are every bit as safe as an empty slice or an empty string (which is just an immutable empty slice).
> The fact that such a new language as Go doesn't have a way to express missing values safely should be disappointing to developers.
I agree, but I'm mildly annoyed by it, but as it's the least of all of my problems, I find words like "absurdity" to be too heavy-handed.
As for Option: Not sure how that's an argument. And anyway, a language with real sum types will never allow going down a branch that doesn't receive a correctly matched value.
It sounds like you're again confusing nil interfaces with an interface holding a nil value (in particular, there is no way to get a nil interface from a nil slice). Here's an example that demonstrates nil slices do not cause problems: https://play.golang.org/p/tSA_otqg3-
> Another is that it leads to inconsistencies: json.Marshal(([]string)nil) returns "null", for example, not "[]".
1. This is unrelated to the language; it's the behavior implemented by the JSON library
2. This behavior is correct; a nil slice is not the same as an empty slice:
fmt.Println([]int(nil) == nil) // true
fmt.Println([]int{} == nil) // false
> As for Option: Not sure how that's an argument.
You posited that Go's nils are bad because you can satisfy an error interface with a pointer type, and then when you return (*ConcreteType)(nil), your error handling code executes. The problem here is unrelated to nil or to interfaces; the problem is that you're nesting one binary type inside another (binary in the literal sense of having two states, either nil or not nil in this case). You would have the same problem in Rust had you done Result<Foo, Option<ConcreteError>> or (in the case of a single return value) Option<Option<ConcreteError>>. You would fix the problem in either languages by recognizing that you only have 2 states (error or not error) and removing the nesting (in Rust: Result<Foo, Error> or Option<Error>; in Go `return ConcreteType{}`).
> And anyway, a language with real sum types will never allow going down a branch that doesn't receive a correctly matched value.
I agree, and it would be nice if Go had this, but this is also not a very significant problem--this problem is blown way out of proportion.