Using Protobuf with PyPy | Hacker News Reader

Using Protobuf with PyPy

omerkatz.com

20 pointsthe_drow9 comments

I didn't quite follow how cppyy/Reflex works (and the Reflex link is broken). But I work on Python Protobuf at Google and wanted to clear up a few things about that.

> Protobuf's compiler generates Python code which relies on a CPython extension in order to interface with the Protobuf implementation and is I mentioned before this is not going to be as fast as it should be for PyPy. Google provides a pure python implementation of Protobuf in version 3.0 which is not yet release and breaks compatibility in some aspects from Protobuf 2.x.

Protobuf 2.6 has both a pure-Python implementation and a C++-accelerated one, just like Protobuf 3.0. The C++-accelerated one is only necessary if you want the performance benefits.

Also, Protobuf 3.0 does not break any compatibility with Protobuf 2.x. Protobuf 3.0 does contain a new protocol version "proto3", which differs in some ways from the previous protocol version "proto2". But Protobuf 3.0 supports both proto2 and proto3, and the two are completely interoperable with each other. Basically, each message you define is either proto2 or proto3. Any proto2 messages you define with Protobuf 3.0 are 100% compatible with proto2 messages in Protobuf 2.x.

So would it be possible to use the pure-Python implementation with PyPy without having to resort to Reflex et al?

To my best understanding, the pure Python implementation is much slower. Even on PyPy.

I did not know that Protobuf 3 is compatible with Protobuf 2.

I read that as Protobuf 3.0 supports proto2, which is different than proto3 being compatible with proto2.

I fixed the link to reflex. Thanks for that.

I can extend the explanation about cppyy/Reflex if needed.

Hi. We had tons of success using protobuf-c and wrapping the resulting wrapper using cffi. The pure-python protobuf wrapper generates really terrible code - don't use it for anything performance-sensitive.

Yeh I know. We discussed this approach on the #PyPy irc channel. As far as I recall the wrapper is not open source. We needed something that can be done quickly and therefore Reflex code generation based approach worked better for us.

there is no wrapper ;-) at least no wrapper generator, we just hand-wrapped some stuff.

Reflex generates the bindings for us. That's why we chose it instead of hand-wrapping the C API ourselves.

Reply on news.ycombinator.com