Anyone know of an efficient binary format similar to MessagePack that supports deserializing only whitelisted keys, and not paying the penalty of parsing data that isn't needed?

MessagePack is great, but libraries generally only support deserializing the whole thing. I have an application where these structured documents can be very large, and scanning code sometimes only needs a very small subset of keys.

I believe Cap'n Proto has this feature, but unlike MessagePack it's not schemaless.

For example, given a struct like this:

  {
    "id": "123",
    "name": "Developers",
    "members": [{
      "id": "567",
      "permissions": [
        {"type": "read"},
        {"type": "write"}
      }
    }]
  }

Let's say I only want the name, the ID of each member, and whether permissions.read is set. I may want to do something like (Go):

  StreamingUnmarshal(b,
    func(keypath string, parse func() interface{}) {
      switch keypath {
        case "id", "members.id":
          value := parse()
          // ... use value ...
        case "members.permissions.type":
          if parse().(string) == "read" {
            // ...
          }
      }
    })

Random access could be also work, as long as it didn't need to sequentially parse from the beginning of the data to get to the right value each time. Something like:

  id := GetKey(b, "id")
  memberIDs := GetKey(b, "members.id")
  permissionTypes := GetKey(b, "roles.permissions.type")

The trickiest bit is treating arrays of nested structs (as in "roles.permissions.read") correctly, although efficient scanning of keys becomes an important optimization point, too.

Probably the best method would be to store the keys pre-sorted at the beginning of the data, so that they'd better fit in the CPU cache, and have pointers to the offsets of the values:

  KEY1,KEY2,KEY3,VALUE1,VALUE2,VALUE3

Arrays of structs are tricky here, again, but this is solvable.