Comment by magicalhippo | Hacker News Reader

Comment by magicalhippo | Hacker News Reader

I was looking at MessagePack for communicating to and from my STM32F1-based microcontroller project from the PC controller software I'd be writing. At least the official C library was not optimized for memory usage and code size. I also considered BSON, but it also lacked suitable libraries.

So I ended up using JSON. Yes the message sizes are larger in byte size with JSON but using the jsmn[1] parser I could avoid dynamic memory usage and code size was small. The jsmn parser outputs an array of tokens that point to the buffer holding the message (ie start and end of key name etc), so overhead is quite limited.

For JSON output I modified json-maker[2]. It already allowed for static memory usage and rather small code size, but I changed it to support a write-callback so I could send output directly over the data link, so I didn't have to buffer the whole message. This is nice when sending larger arrays of data for example.

Combined it took about 10kB of program (flash) memory, of which float to string support is about 50%. Memory usage is determined by how larger incoming messages I'd need, for now 1kB is plenty.

A nice advantage of using JSON is that it's very easy to debug over UART.

Though having compact messages would be nice for wireless stuff and similar, so does anyone know of a MessagePack C/C++ library that is microcontroller friendly?

[1]: https://github.com/zserge/jsmn

[2]: https://github.com/rafagafe/json-maker

My MessagePack implementation is designed for embedded:

https://github.com/ludocode/mpack

It can be built to a very small code size, especially when you disable libc, allocations, etc. There are some people using it on embedded devices like Arduino. There's someone working on a port to 8-bit microcontrollers that don't have a 64-bit float, so you may want to look into that as well; see the open issue for it on the GitHub link above.

Protobufs/nanopb would be my go-to for minimal message size.

If you want small code size, CBOR seems like a good bet:

> The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. [1]

This [2] C-implementation fits in under 1KiB of ARM code.

[1]: https://cbor.io/

[2]: https://github.com/cabo/cn-cbor

I, too, love cbor. https://github.com/RantyDave/cppbor

CBOR is also used on WebAuthn, usage in a web spec means to me that someone smart considered it a sane choice -- and more importantly that the format is here to stay.

CBOR’s RFC: https://tools.ietf.org/html/rfc7049

And Amazon has picked it up as a first class citizen in some of their IoT Core features. It’s definitely here to stay.

It's great CBOR is accepted in wider area, but I am personally curious why WebAuthn choose CBOR instead of JSON. WebAuthn is a web browser feature, and why W3C would introduce a new data exchange format in their specs? Maybe WebAuthn needed a binary data type?

I'm guessing a binary format is nice when interacting with a device..

Anybody know if (and why) U2F uses CBOR?

Yes. CBOR is designed for IoT especially in mind.

Ah yes I looked at CBOR too, but I dismissed it for reasons I can't recall right now. Will have to take another look.

That’s strange because CBOR is almost literally msgpack that got an RFC and has extensions. I cant remember what MsgPack does for online streaming and indefinite lengths.

They’re extremely similar.

Looked at it again, seems memory management is a bit of an issue, it supports memory allocation callback but not just handing it a buffer to work with (though I guess allocation should be predictable).

Also I don't know how they got "code sizes appreciably under 1 KiB". On my STM32F1 release mode with -Os it adds about 12kB.

But yeah, maybe I should reevaluate CBOR.

For reference, I’m using TimyCBOR because it’s include with Amazon FreeRTOS.

You’re on your own for malloc, which for me is great because FreeRTOS Heap4 management is quite good. So I malloc an object I’m decoding into and parse away.

There are two options parsing arrays and strings/bytestrings and I just chose the option where I specify the pointer to use, vs them using normal malloc then free() later.

I really like this setup. I made a deinit(bad_message) that works anywhere it failed (parse, validate, eval, etc), goes through and looks for pointers that I previously would have malloc’ed.

There is another popular library but I forget what it’s called.

CBOR was originally part of the MsgPack project, by the way, before its designer forked it and renamed it after himself.

> MessagePack ... official C library was not optimized for memory usage and code size

But libmpack is: https://github.com/libmpack/libmpack

- libmpack serialization/deserialization API is callback-based, making it simple to serialize/deserialize directly from/to application-specific objects

- libmpack does no allocation at all, and provides some helpers to simplify dynamic allocation by the user, if required.

- C89

ArduinoJson https://arduinojson.org supports MessagePack. I haven't looked at its static or runtime memory requirements.

Not messagepack, but if protobuf is ok, then nanopb has given me good results on uC projects.

Ah, that looks pretty spiffy, thanks!

I'll second nanopb as pretty good. Used it on an STM32F4.

It’s good. But it’s a one man show and you are very much in this is how it’s done realm.

I threw away NanoPB in favor of TinyCBOR and haven’t looked back.

No idea if it suits your needs, but here's my pet project for microcontroller friendly communication protocols: https://github.com/jean-roland/LCSF_C_Stack

This library is very small and you need to implement your own I/O via well-defined functions. The parser itself does not use any library (including libc):

https://github.com/camgunz/cmp

Reply on news.ycombinator.com