Working with Binary Data in Python

Have you tried bitarray[1]?

[1]: https://pypi.python.org/pypi/bitarray

Also https://pythonhosted.org/bitstring

Very cool. Any conclusions from the experiment?

Disclosure: I'm the primary author/maintainer of suitcase (https://github.com/digidotcom/python-suitcase). In your example, you can shave off a line by doing the following:

    example = FIDOAttestation.from_data(a)

Here's an example of fairly extensive use of construct:

http://amnesia.gtisc.gatech.edu/~moyix/minidump.py

Explanation:

http://moyix.blogspot.com/2008/05/parsing-windows-minidumps....

Thanks for the link, I didn't know about construct!

A colleague of mine recently developed a similar (from the looks of it) library, in Python and Javascript:

https://github.com/jfjlaros/bin-parser

Unfortunately it's not under active development for several years. Lots of good ideas though.

> when manipulating real binary or in memory data, a little bit of C would really simplify

In what sense? Accessing an octet in a buffer and performing basic bitwise operations on it is equivalent:

   # C _and_ Python
   some_buffer[idx] & 0x80

Reading a buffer in Python is slightly less error-prone, as errors are automatically converted to exceptions, which can be caught at a more centralized location in the code; in C, one must check for errors (and manually propagate those errors) at every read; this applies a bit less in C++, though even there, the stdlib leaves some things to be desired (but generally speaking, they can be wrapped away).

Text encoding/decoding, struct packing, and base64 aren't in C or its standard library.

Now, in terms of speed, C wins hands down. (I wrote a minecraft file parser in both C++ and Python, and the difference was several orders of magnitude.)

Distributing that becomes much more of a pain, though. Unless it's a major component of the software, I'd really rather keep it in pure Python.

This seems intuitively obvious, but can you think of an example that wouldn't be answered with a competitive implementation using Cython or Numba?

If you upvote a story, it will be saved on your profile page under the "saved stories" link.

It's still very cumbersome, especially compared with state of the art. You may want to look at how it works in Erlang.

Agreed. I remember unpacking binary data in Python 2.7 during my late undergrad and found it much more cumbersome. I bookmarked the article in celebration the next time I'm doing something like that again.

All python 3 does is add an onion layer around unicode. Fortunately it looks like these techniques will work for python 2.7 or 3.x. With 2.7 obviously being the preference for pretty much everyone who isn't a python 3 developer.

I disagree. The cases where I would do so voluntarily are limited, but I've been forced to deal with binary data on several occasions. HDF5 is also my goto if I am in control, but sometimes you're given data that is only in some obscure binary format.

Unless you were implying that such work is best done in another language. I can buy that, but I found Python to be a convenient choice for it.