Hacker News Reader

Comment by Retr0id | Hacker News Reader

There's one caveat that I don't see explicitly mentioned in the article - CPython's bytecode "API" is not stable, and availability and behaviour of particular opcodes can vary between two different CPython release versions.

In practice it's stable-ish, but I'd be very afraid of putting it in a critical path like this.

Further, contrary to what you might assume CPython is not memory-safe. Most of the memory-unsafety is difficult to accidentally reach from regular Python code, but it's a distinct possibility if you're emitting bytecode directly. For example, LOAD_CONST is not bounds-checked.

View on HN

Stable-ish probably turns into full unstable due to rapid changes in the faster cpython work. Lots of bytecode changes have been done.

I encountered this when a coworker got random segfaults trying to run some Cython code. Somehow, he was using an old beta version of the interpreter. After stepping through with GDB (I think it was a call to LOAD_CONST) and finding the offending seemingly benign code, on a hunch, I updated the interpreter, and the random segfault disappeared.

> memory-unsafety is difficult to accidentally reach from regular Python code

Can you give an example of Python (source) code that isn't memory safe?

I assumed Python would be similar to Lua in this respect - the interpreter is memory safe no matter what source code you give it, but when it comes to custom bytecode, all bets are off.

My favourite unsafe one-liner:

    eval((lambda:0).__code__.replace(co_consts=()))

This one is slightly cheat-y because it does modify the code object, just not the bytecode itself.

"Purer" unsafety issues are treated as bugs and patched, but that doesn't stop them from existing periodically. I don't have a specific example on-hand but I bet you can find some in here: https://github.com/search?q=repo%3Apython%2Fcpython+segmenta...

Edit: This one is a decent example (and hasn't been fixed for a long time) https://github.com/python/cpython/issues/74957

> "Purer" unsafety issues are treated as bugs and patched, but that doesn't stop them from existing periodically.

Yeah, the same is true for Lua (search for "segfault" here: https://www.lua.org/bugs.html). The difference is that Lua tends to have far fewer bugs overall than Python, mostly because it's so much simpler.

That was intentional. And intentionally quite a lot of safe thing can be broken.

Whoever opened issue #74957 appeared to have encountered it accidentally.

But my initial point was, if you're writing raw bytecode, you're much more likely to hit memory unsafety issues relative to normal Python code.

Anything using codecs, especially CJK. The amount of scary stuff we found when reimplementing it was... Well, scary

The most typical alternative to this is that you write the hot code in an unsafe language. And your code already has dozens of dependencies on third-party modules in unsafe languages. So it's something to be aware of, sure, but it's not opening up a whole new surface of vulnerabilities.

I get what you mean, but everyone already knows C is a sharp implement. It's less obvious that writing CPython bytecode has similar risks. (You can also write native python extensions in rust etc., but most people aren't doing that, yet)

I'm not saying it's necessarily a reason to write off the approach, but definitely something people should be aware of.

Yep.

If you need to process data fast, just write the component in C that runs as a separate process, and send data to it with pipes.

Reply on news.ycombinator.com