Implementing Go's defer keyword in C++
oded.blog
oded.blog
Sorry for pooh-poohing someone sharing some ideas, but being someone who has to maintain long lived code in quite a few languages, code like this immediately sets off my alarm bells.
Don't get me started on the dude who use #define magic to make C look like Pascal.
Some places ban templates, some ban RAII. Some ban operator overloading. Some ban lambdas or C++11 features. Some ban macros.
There's a lot of variety, and just because you use something like this in your library or project doesn't mean you can't come out and play. C++ is the most "be yourself" language I've encountered. There is no idiomatic code.
Now, I get where you are coming from, and I'd love C++ a lot more if there was less feature abuse, but I've accepted that the ship has sailed.
If you tell people that the only way they can do something "on scope exit" is to write a whole class with a destructor and manually capturing variables, the result is they won't bother. They'll just put the code at the end of the function, and say "I'm too lazy to make this exception-safe."
This defer() macro is almost exactly what I've been using for the last 5 years or so:
https://github.com/capnproto/capnproto/blob/master/c++/src/k...
And I can say pretty confidently that it's a really good idea to have. I'm sort of surprised, actually, that not everyone is already doing this...
I did it for memory allocation, because C designers weren't able to create something as simple as new, rather requiring us to keep using sizeof on malloc calls.
That was only for an year, until I got hold of Turbo C++ and only dropped pure C when my teachers required me to deliver assignments in C.
Thankfully most of them accepted C++ as well.
with open(oldfile, 'r', encoding='utf-8') as infile:
with open(newfile, 'w') as outfile:
....
If the implied close generates an exception, everything still works reasonably. __exit__ has a "traceback" parameter, which indicates an exception is in progress. An exit operation should do its closeout, then re-raise the exception sent as the traceback, if any. This works even when a destructor raises an exception. (A typical case is closing a network connection. If the other end dies, close will generate an error, which should not be lost.) The default use of "with" in Python thus comes with correct single and nested error handling.
That's not true of RAII or Go's "defer". In C++'s destructors, you can't return an error code, and an exception in a destructor is usually a big problem.
It's hard, but not impossible, for a deferred call in Go to report an error. The deferred function can muck with the parameters of the function which called it. This only works if the deferred function was written specifically for the context in which it was called; just deferring a "close" won't do this. So you don't get proper error handling by default.
(Languages seem to need exceptions. Both Go and Rust started with an always-aborts panic mechanism. That was too drastic in practice, and in both languages, panicking was made recoverable and provided with unwinding machinery. So both languages now have the heavy machinery for exceptions, without the language support for them. This results in kludges to simulate exceptions.)
What's wrong with indenting? You're entering a scope; indenting is appropriate.
The primary use case ExitStack is not to reduce indentation levels; it's to support "a variable number of context managers and other cleanup operations in a single with statement".
You wrote "The easiest way to handle opens and closes is thus exception-safe".
Consider if you need to merge up to 50 files containing sorted lines, where the result must also be sorted, and where you should not load all of the data into memory first. (Sorted according to UTF-8.) How would you do it? Not so easy, is it?
With ExitStack you could write it as something like the following sketch:
filenames = [ list, of, file, names]
with ExitStack() as stack:
files = [stack.enter_context(open(fname, "rb")) for fname in filenames]
with open("merged_lines.txt", "wb") as output:
output.writelines(heapq.merge(*files))
A side-effect of making ExitStack work is that it might help if the indentation depth is a problem, which mappu apparently has experienced.
As another example of where Python registers a cleanup rather than use a context manager, see https://docs.python.org/3/library/unittest.html#unittest.Tes... .
from contextlib import contextmanager
@contextmanager
def tag(name):
print "<%s>" % name
yield
print "</%s>" % name
>>> with tag("h1"):
... print("foo")
...
<h1>
foo
</h1>Remember, "with" isn't always required - you can also write the code using a try/finally.
It does not, it works like a C++ destructor, the context management code is static, and anything it needs is provided as a parameter.
While "ad-hoc" context managers are possible (and quite trivial in fact):
1. I don't remember ever wanting one
2. they would not be convenient owing to Python's limited lambdas so
3. you'd probably implement them using a ~decorator on a nested function instead which
4. may look quite odd to most Python developers
For example
File.open(args) do |file|
# use file
end
def self.open(*args)
file = File.new(*args)
begin
yield file
ensure
file.close
end
endAn other is that Python's "context managers" are reified which means you can pass them around until you ultimately use them. It's a tradeoff.
@contextlib.contextmanager
def open(*args)
# file open
try:
yield file
finally:
file.close()
with open(args) as file:
# use fileBut as nice as it is, to me it's just syntax sugar over the wrong abstraction instead of something beautiful in it's own right.
def self.open
File.open(...) { |f| yield f }
end
I ended up an OnExit() context that does:
if conn is not None:
ctx.call(conn.close)This is completely incorrect with respect to Rust, both historically and practically. Rust started with unwinding, and then grew opt-in abort machinery for those use cases where it matters. And, for what seems like the millionth time, unwinding does not equal exceptions; this is like saying that C has exceptions because of setjmp/longjmp. Unwinding exists in Rust solely to give one the ability to fail gracefully, running destructors that might have valuable side effects and allowing Rust-written subsystems to avoid taking down the entire system. For an example of the latter, Skylight, who ships a Ruby gem written in Rust, "catches" any unwinding so that an errant failure in their gem does not bring down a client's entire Ruby process. I have been writing Rust for years and, without exaggeration or (heh) exception, have never once seen the ability to "catch" unwinding used as a basis for constructing ad-hoc recoverable exception-handling, especially since abort-on-panic is an entirely valid mode for Rust applications (this is how Dropbox's backend Rust is compiled, for example) and completely obviates any sane attempt to use panics for recoverable error handling.
- Exception -> Rust's panic
Reality:
- Runtime assert -> Rust's panic
- Exception -> Rust's Result (ie error monad)
[0]: https://github.com/urfave/negroni/blob/master/recovery.go#L1...
Ehhh... `with` is a crude substitute for destructors. C++ is a whole language designed around RAII. Python really is not.
You do make a good point that the handling of exceptions in destructors in C++ is problematic. But you can actually throw from destructors, as long as the destructor wasn't called during unwind for another throw. I'm of the opinion that the inability to throw again during unwind is a glaring bug in the standard, since it presumes people can know if their code is going to throw, which is like presuming people know where their bugs are. But the reasonable way to work around it is to squelch exceptions from destructors when another exception is already in-flight. This is usually fine, as the "secondary exceptions" thrown during an unwind are usually side-effects of the "primary exception" anyway.
If you write C++ with exceptions, you need this. If you simply place tear-down code at the end of the function instead, you are creating exception-unsafe code. Of course, ideally state is encapsulated in classes with destructors and tear-down happens there, but writing a whole class can often be overkill for a one-off use case. If you don't have a defer macro, you're going to get lazy and introduce bugs.
If you don't use exceptions, then you still need this, because your code almost certainly has lots of "if (error) return nullptr;" or whatever. Without `defer`, you need to stick the teardown code before every one of those returns, or you need to resort to "goto fail" error handling.
No one is suggesting this. They are saying that you use an RAII class such that it doesn't need tear down. In the example in the article, if std::ofstream calls close() in its destructor (which it does) then you don't need to DEFER a call to it. If you lock a mutex, use a RAII locker class so that you don't need to DEFER a call to unlock(). As someone once said of C++: "my favorite feature is the close brace", because it calls destructors for all objects on the stack, even during exception unwinding.
Of course, the article is just using RAII. But I disagree with the style; rather than re-writing a DEFER(myfile.close()) in every single function that uses a file object, you should make the file class have an appropriate destructor, or at least write a wrapper that does. That way you re-use that code rather than having to re-write it over and over.
> If you write C++ with exceptions, you need this.
You absolutely do not. You need to use the core language feature that it is using: deterministically-called destructors.
At least one person in this thread is, in fact, suggesting this: https://news.ycombinator.com/item?id=15524405
> They are saying that you use an RAII class such that it doesn't need tear down.
Yes, as I said in my original comment: "Of course, ideally state is encapsulated in classes with destructors and tear-down happens there, but writing a whole class can often be overkill for a one-off use case. If you don't have a defer macro, you're going to get lazy and introduce bugs."
I do know how destructors work, thanks.
In C++ the destructor handles this as part of the lifespan of the object. Making users of the object encapsulating the resource handle resource cleanup is an antipattern, which is why the standard library does this for you.
Otherwise it seems a lot like you can just make the call at the end of scope yourself for the same line count and _much_ more readable code.
I guess if you have multiple return paths it's more tenable to use scoped dtors for this but it all really seems questionable to me -- I can't think of a single use-case where it'd be preferable to use something like this rather than a dedicated class.
(Obviously, classic RAII has pretty much the same benefit. But it requires a whole class per type, whereas this approach has less boilerplate.)
If your application uses exceptions at all, then the time to be super-defensive about exceptions is all the time. If you don't form the habit, then you'll forget in the cases where it matters.
Plus, most programmers are pretty bad at identifying where exceptions might be thrown, because it tends to reduce to the question: "Where will my code have bugs?" Obviously, if you knew, you'd have fixed them...
> _much_ more readable code.
I'd argue that it's more readable to put state build-up and tear-down next to each other, so that future readers can see and modify them together. If you put them far apart, it's easier for them to skew.
I think life is simpler if you just adopt a blanket rule: all cleanup is done in destructors/scoped guards. Occasionally code might be slightly longer than it needs to be, but your life is simpler overall. Plus you can forget the old C rule of only one return from a function that you mentioned. (If you use exceptions anywhere then you already stopped using that rule anyway.)
typedef fdwrapper = unique_ptr<int, close>;
fdwrapper fdhandle(open(filename)); int fd = socket(...);
if (fd < 0) return false;
SCOPE_EXIT { ::close(fd); };
if (ioctl(fd, ...) != 0) return false;
if (ioctl(fd, ...) != 0) return false;
...
return true;
Which, BTW, is exactly what folly::File does :)
Yes, and that RAII class would be reusable all of the places you need to close a file descriptor. It's even a handy place to hang more methods that work on the descriptor... for instance if you wanted to abstract those ioctls.
That's why I personally never use ScopeGuards in C++. You are correct that writing a single-use RAII class requires a few lines of extra code. However I have found that after I write such a class I often find myself using it over and over.
Also, defer in Go always uses the function exit, not the scope. Not sure you can do that in C++. Everything seems scoped.
Maybe don't do this :-)
Typically though there are custom delete/free API calls you need to make, and to use them with unique_ptr you have to pass that function pointer in as the second argument to the unique_ptr constructor.
At the very least you should have a C++ function that gives you the unique_ptr back that does the right thing; but at that point you almost might as well just make a class and give it member functions to map the C calls.
Of course, in my C++ code I would just write a class/struct to wrap any RAII-like logic, not use a “defer”. A marginal amount of extra work that pays off in consistency and readability.
#define DEFER(...) ScopeGuard CONCAT(_defer, __LINE__) = [&] () { __VA_ARGS__ ; }
It’s this type of constant minor language-lawyering and clerical work that makes me both very good with C++ and very averse to using it anymore. (Though to be fair this particular case is also a problem with C, which I like nevertheless.)
e.g. in Go can you can defer inside a for loop.
pro: you can push more than one "defer" in the same function.
con: "finally" puts the cleanup code at the end of the function. With "defer" you have it scattered at the beginning.
Quite commonly an advantage, e.g.:
f, err := openFile('...')
if err != nil {
...
}
defer f.Close()
Same with locks and release. Now you don't have to scan across the entire function body to make sure your inits and cleanups match up.
Defer gives no guarantees about when that thing is going to run -- just that it is after the function returns. Because of this, you can get erratic behavior if you open and defer closing a lot; essentially openFile just stops working at some point.
This can happen with any limited resource and isn't theoretical; I had a friend who ran into this in long running process with a lot of IO.
A "defer" statement invokes a function whose execution is deferred to the moment the surrounding function returns, either because the surrounding function executed a return statement, reached the end of its function body, or because the corresponding goroutine is panicking.
https://golang.org/ref/spec#Defer_statements
There might have been something else going on with your friend's code. Maybe the function he was using didn't return before running out of file descriptors?
There must have been some other leaking going on. He was using a largish library when he saw the leak and eventual crash so maybe it was due to improper pooling or C api calls? I'll have to ask him more about it.
I would consider this a 'pro' as each action and defer are paired, executes in reverse order and a conditional action/defer doesn't need a second matching check.
func a() {
i := 0
defer fmt.Println(i)
i++
return
}Honestly, am I missing something?
So, the version that prints 1 is
func a() {
i := 0
defer func(){
fmt.Println(i)
}()
i++
return
}Although this approach would probably not work like this in Go given Go's lack of exceptions.
{
std::lock_guard<std::mutex> lock(some_mutex);
...
}
But yeah, it has downsides. If you would not name the variable, and do `std::lock_guard<std::mutex>(some_mutex);`, then its scope is limited to a single line, and the lock is immediately released.
In Rust however sync::Mutex::new takes a value, which the lock owns, and acquiring the lock yields a MutexGuard which acts as a reference to the wrapped value. You must acquire the lock to get access to the value (since it's owned by the lock itself), and once the lock is released you have no way to access the locked value (unless you created a copy).
> If you would not name the variable, and do `std::lock_guard<std::mutex>(some_mutex);`, then its scope is limited to a single line, and the lock is immediately released.
OTOH you can pretty trivially create a wrapper/extension which takes a lambda, acquires the lock, calls the lambda then drops the lock. It could even return the lambda's result while at it.
> C++ has a neat feature called Resource acquisition is initialization, a.k.a RAII.
Is this really a feature of C++, or is this a design pattern/best practice that developers have discovered and use to write safer C++ code?
While RAII was coined by the C++ community, it is not unique to C++, there are other languages that support it.