Three bugs in the Go MySQL driver (2020)
github.blog
github.blog
Tcp is full duplex, but net.Conn (the main std API that everyone is using), doesn’t have a “CloseWrite” method. That bubbles down to all downstream uses, such as tls. Another example would be how deadlines and timeouts are handled, and how they (don’t) interact with context, which is the blessed way to do timeouts for all non-IO purposes..
Still, Go is one of the best languages for network programming imo, because it allows you to check most edge cases (like timeouts) without wasting like 4 threads to do so.
But that’s just like.. my opinion.
One trick to unify the context and io timeouts is roughly:
conn = ...// Wherever you get your file or con.
ctx, cxl = context.WithCancel(ctx)
defer cxl()
go func(){
defer conn.Close()
<-ctx.Done()
}()// Do reads and writes knowing that if your context times out the fd will close and your io op will stop.
The only caveat is if you need to be aware of partial data read/written when a timeout occurs - then you need to tell the os about your timeouts :(.
However, once you need idle timeouts or heartbeats you have to increase the complexity further. Setting deadlines before every op (which is recommended) is (a) super easy to forget and (b) a perf issue if you write a stream of data, due to the several syscalls involved, which can be noticeable if you transfer at bandwidth capacity.
Even std (such as net/http, tls) is full of these workarounds. If Go could get the last remaining pieces right, it would be near perfect for userspace network programming imo, as well as benefits in other domains as well. I want this Pony.
I don't understand this - isn't the pool supposed to stick around for the duration of the program?
Did you mean that the connection sticks around for the duration of the program?
I did not know it did that, and scanning through the docs, they don't appear to mention it either.
Is this only on client programs? Does it do the same for server programs too (create a pool for each incoming src-ip:src-port connections)?
defer rows.Close()
/* do things */
err = rows.Close()
if err != nil {
}
1. If the explicit Close() is called first, the error from the deferred Close() is ignored.2. If the explicit Close() is never called because of a panic, the deferred Close() will kick in and cleanup the resources.
Like you said, the performance gains (if any) are probably not going to matter when a database call is involved.
I think if there’s a panic somewhere that you’re not explicitly expecting you can say goodbye to any notion of safety you might have had.
Most panics are small stupid things localized to a specific piece of code, like nil dereference or slice out of bounds, and not indicative of some global process state being messed up.
In my deployments, a server is just one in a pool of servers, so losing one is not a problem. The whole system is designed such that any server can die at any time and the overall service will continue to be available.
If you have a different architecture and design philosophy to me then I can see why you would want to recover in your handlers. It just doesn’t suit me at all.
Cases where it panics and leaves an incorrect state seem very rare; panics due to an incorrect state are slightly less rare, but still fairly rare.
func fetchFoo(ctx context.Context) (_ Foo, mErr error) {
conn := pool.Acquire(ctx)
defer errs.Capture(&mErr, conn.Close, "close conn for fetch foo")
}
I have a couple of useful variants: defer errs.CaptureLog(logger, conn.Close, "close conn")
defer errs.CaptureT(t, conn.Close, "close conn")Is there anything GitHub haven't copied from GitLab? They did it before it was cool multiple years ago. Same with CI/CD, Kanban project management, package management, etc.
Competition is good for users, but it must be annoying for GitLab to get copied in everything by the incumbent competitor that remained stale for years before being bought by unlimited money Microsoft.
On this topic, maybe finally this will make GitHub Enterprise self-hosted an acceptable alternative to GitLab? As far as i know it has always been extremely painful to manage.
CI was omni-present on GitHub in the form of Travis; integrating it in GitHub was kind of a no-brainer, and Kanban boards have been around long before GitLab. It seems more "convergent evolution" than anything else.
> incumbent competitor that remained stale for years before being bought by unlimited money Microsoft.
GitHub already got $200-300 million in revenue before they got bought by Microsoft, roughly equal to GitLab today. I don't think money is really the big factor here (and AFAIK GitHub finances are still independent from Microsoft; they don't "pump money" in it).
I don't know why GitHub was relatively stale before Microsoft bought them. I suspect the reasons are more organisational than finance-related.
Three bugs in the Go MySQL driver - https://news.ycombinator.com/item?id=23254045 - May 2020 (87 comments)
These days though it's genuinely pretty solid, haven't had a problem in a couple years. They've largely worked the kinks out.
But I think it’s a major design error to use FFI for a thin client library. You’re bringing in a whole other network stack for what should be some simple serialisation/deserialisation. Plus even if you were, say, linking against libmysql you’d still have a bunch of wrapper code to make it a nice API in whatever the host language may be. Not worth it.
2. One of the major pitfalls with linking against c libraries that do their own i/o is that they don't play nicely with the go scheduler. That means the goroutines blocking on a c i/o operation will occupy it's thread for the duration of the call.
This article however is about Go, where linking to C libraries is often not the best approach, so the situation is a little different.