Comment by dasmoth | Hacker News Reader

Comment by dasmoth | Hacker News Reader

Parent Full thread

I agree regarding the symptoms, but am less convinced that run-time efficiency is the sole (or perhaps even major) cause. If it were, I'd argue that we'd see less usage of, e.g., Python.

I don't know the true cause -- I wish I did. But I do do see a trend towards an ever more "hands-off" style of software development: large volumes of automated (especially unit) tests in preference to interactive approaches (or hybrids, like running subsets of tests via a REPL), and running test instances on remote servers (often via extra layers of indirection, such as CI servers) rather than running on your own machine. I'd love to see a resurgence of interactivity, but if anything it seems to be going against recent trends.

Edit: The subjective impression I've got is that doing things the indirect, hands-off way is being presented as somehow more "professional" while hands-on interactivity is the realm of the self-taught and hackers. Do others see this? What can be done to change these perceptions?

This decision was made and become deeply entrenched long before Python came along. Python is pretty good, but even it is constrained by the decisions that went into the design of C, since Python is implemented in C. Its underlying data structures are C data structures. Its memory management is C memory management. The GIL is there because C.

> The GIL is there because C.

Oh please.

> The GIL is there because C.

That's not at all true. There are very, very few platforms/libraries/languages that both a) allow you to manage concurrent access to complex, arbitrary data structures in a sane, predictable, and not-dangerous way and b) provide the capabilities (including performance) necessary to implement a general-purpose programming language on top of them. Even fewer of those tools existed when Python came about.

Was it technically possible to make a GIL-free Python or a Python not based in C? Sure. But it wasn't in any way likely, or a reasonable-at-the-time decision. If you look into the history of the GIL things will make more sense; it has next to nothing to do with the implementation language.

I'm not saying that trading the GIL for the benefits of C was not a reasonable tradeoff. Nonetheless, it is simply a fact that the GIL is needed because of the constraints imposed by choosing to work in C.

Citation needed. “Because there existed languages with different concurrency models” doesn’t really explain why the choice of C for Python’s implementation necessitated a GIL (there are plenty of languages and tools built on C with non-GIL-ish concurrency systems, and many languages/tools built on not-C that don’t expose the concurrency features of their underlying language at all). Look into how/why then GIL was added; it has much more to do with not wanting to reimplement the language/break compatibility than it has to do with what the language was implemented in.

> it has much more to do with not wanting to reimplement the language/break compatibility

I don't know what "language/break compatibility" means.

The GIL is needed because Python's reference-counted memory management system is not thread-safe, and this can't be fixed without compromising either portability or performance. It's really that simple.

> The GIL is needed because Python's reference-counted memory management system is not thread-safe.

That is correct. What does that have to do with C? Thread-unsafe code exists in all languages. The GIL's lock is itself a pthread mutex and condvar underneath, and equivalent constructs exist in all (to my knowledge) modern threaded programming environments.

"not wanting to reimplement the language/break compatibility" is a reference to the successful efforts that have been made to remove the GIL in CPython. Those efforts have not (yet) moved towards merging into mainline CPython because they require a) lots of reimplementation work and added complexity in the language core, and b) would very likely break the majority of compiled extension modules.

I think that's additional evidence that the GIL isn't a C problem; they removed it, in C, without fighting or otherwise working around the language.

> That is correct. What does that have to do with C? Thread-unsafe code exists in all languages.

Yes, that's true. But thread-unsafe GC does not exist in all languages. When GC is provided natively by the language it can be implemented much more safely and efficiently than if you try to shoehorn it in afterwards.

> the successful efforts that have been made to remove the GIL in CPython

That's news to me. Reference?

https://youtu.be/P3AyI_u66Bw

Or search “gilectomy” on LWN. Or check out stackless etc.; it turns out removing the GIL technically is historically one of the easier parts of removing the GIL entirely.

It sounds like your main problem is with python’s GC model. Reference counting doesn’t have to be thread-unsafe, but in scripting/interpreted-ish languages that want their threading story to involve seamless (I.e. no special syntax unless you want it, it’s on you not to blow off your foot) sharing of state between threads at will, like Python and Ruby, a GIL or equivalent is the norm. Sometimes it’s not as intrusive as python’s, but it does seem like a necessary (or at least very likely) implementation pattern for languages that want to provide that seamlessness. You can have thread-safe reference counted GC in a traditional scripting language, but that tends to come with a much less automatic threading/concurrency API. Perl 5 is an example of that category, and it is implemented in C.

> a GIL or equivalent is the norm

Exactly. And why do you think that is? And in particular, why do you think it is the norm when it is decidedly NOT the norm for Common Lisp and Scheme? The norm for those languages is completely seamless native threading with no special syntax.

It seems like you're arguing in favor of a different language rather than a different implementation of Python. If you wanted to implement, say, the Python class/'object' type in a thread-safe way that didn't rely on a GIL and still freed memory according to approximately the same GC promises as the Python interpreter, I suspect you'd end up implementing something GIL-like in Scheme or Lisp (though my experience extends only to some Clojure and Racket in school, so I may be unaware of the capabilities of some other dialects).

If you wanted to implement a language that looked a little like Python but had your favored Lisp's GC semantics and data structures, I'm sure you could. But it wouldn't be Python.

That's without getting into the significant speed tradeoffs--you can make these languages fast, and I get the impression that there has been a ton of progress there in the last decade. But when Python was being created, and when it had to deal with the implications of concurrency in its chosen semantics? Not so. As I originally said: was it theoretically possible to build Python or equivalent on top of a non-C platform at the time? Sure. But I doubt that would have saved it from tradeoffs at least as severe as the GIL, and it definitely would not have been the pragmatic choice--"let's build it on $lisp_dialect and then spend time optimizing that dialect's runtime to be fast enough for us to put an interpreted scripting language on top of it" seems an unlikely strategy.

> If you wanted to implement, say, the Python class/'object' type in a thread-safe way that didn't rely on a GIL and still freed memory according to approximately the same GC promises as the Python interpreter, I suspect you'd end up implementing something GIL-like in Scheme or Lisp

Nope. CLOS+MOP, which exists natively in nearly all CL implementations, is a superset of Python's object/class functionality.

The GIL is not there because C. It's there because of refcounting and Python guaranteeing that many nontrivial operations are thread safe.

Yes, but refcounting is there because it is impossible to build a proper GC in portable C.

Can you elaborate on this?

You can't trace the stack in portable C.

Sometimes I do work this way in the repl - usually when I have little confidence I even understand how to fit together a new library or API call. But if you spend more time in the repl building a data structure and interactively iteratively testing your function, when it is all finished all you have checked into source control is a function, not the tests. You have to then write the tests. In a lot of cases its faster to just write the tests and interact with the code through those. In some language environments this is not exclusive of using a repl - in Haskell for instance it is pretty easy to interactively run a module and tests saved to disk in the repl.

Perhaps if great interactive development and debugging tools were more pervasive, there would be less demand for lots of tests written at the same time as the actual function.

(The fine-grained "unit" stuff, anyway. System/integration tests come with different trade-offs).

Reply on news.ycombinator.com