In memory Key/Value store in go using gRPC | Hacker News Reader

In memory Key/Value store in go using gRPC

github.com

52 pointsknrt28 comments

Live design review: never use type “string” for keys and values in protocol buffers. Always use “bytes”.

hey can you add why?

Why is that?

My google-fu only turns up references to invalid utf-8 data, but that didn’t sound like what you were referring to.

string and bytes are the same on the wire but generate different application code. string is defined to be utf-8, so if you are planning to put arbitrary bytes in there, you can't use that type. in some languages string is checked for utf8 conformance, which is expensive. In some languages allocating strings is more expensive than allocating bytes.

Only use string when you really mean utf8.

Isn't that still ok for keys ? You typically want keys to be human-readable and -writable strings anyway

I recently had this debate in a professional context. My correspondent argued that string was good for filenames, because it's a human-readable string. I pointed out all the different ways that a filename (on a variety of platforms!) can be constructed such that it will not conform to utf8. In a new system I would want to see an iron-clad reason for string in this case, because it's not obviously correct and the efficiency story isn't on the side of string, either.

Luckily you can change from string to bytes at any time, since the wire format is the same, so this isn't the kind of mistake that gets cast in stone.

Maybe your key is hash of something. Converting it to string is extra effort and it takes more space.

> never use type “string” for keys and values in protocol buffers. Always use “bytes”.

What are "keys and values" with regard to protobuf? Surely you don't mean maps as bytes can't be "keys" anyways. And why shouldn't you use string as a field type?

I think they're referring to this app, which uses Protobuf strings to represent its keys and values.

I'm more interested in DB's that store protocol buffers, and can expose their innards (like spanner, bigtable, etc.). What's good to use/try on the outside?

Protobuf has well-defined JSON semantics, so leveraging the JSON encoding seems like the practical way there and many databases can query/index/manipulate the contents (and if space is a concern, often JSONB and similar types are quite compact).

> (like spanner, bigtable, etc.)

Neither Spanner nor Bigtable has this feature. If you store a protobuf in a Bigtable column, that's your own business. Bigtable itself won't notice.

Internal Spanner does support protobuf columns.

That's neat. Just it just blow it up into a struct?

I'm not sure what the internal representation is. Spanner is aware of the proto type through some mixture of DDL and build rules. Updates to proto types get pushed when updated schemas are pushed. You can use deeply-nested fields in various query clauses, create indexes, etc. But I believe updates must replace the whole object. Much like JSONB columns in other RDBMS.

This is from memory. I'm not at Google anymore.

My memory too ;) - Left middle of 2017, here is some article that mentions it - https://www.nextplatform.com/2019/01/15/spanning-the-databas...

also some bare mention here - https://static.googleusercontent.com/media/research.google.c...

"... The effort to standardize our own implementations both covers gaps in the SQL standard where details are unclear or left to the implementation and covers our own Google specific extensions. Before defining our own SQL dialect, we did consider other options. Using the front end of an open-source SQL processing system, such as Postgres, would have accomplished several of the same goals including filling the gaps left by the SQL standard and ensuring uniform syntax/semantics across our systems. We ruled out this approach because internal systems at Google make heavy use of Protocol Buffers in their applications, and we wanted Protocol Buffer message and enum types to be first class types in our SQL language"

FoundationDB's Record Layer?

Internally, google had best-practices (but could be outdated), that each RPC service function should take unique input message, and unique output message, and never reuse one, even if empty, or having same structure, at the cost of more writing.

Later this allows one to change things independently.

Wait but what about EmptyMessage and Any? Yes they're both terrible choices for any circumstance, but also widely seen.

I dunno. I'm simply recalling what I remember as best-practices by memory, could've been wrong about some of them, but for my own protobufs that I use I keep the same practice.

But mainly for that reason, I would like to keep as much as backward compatibility, and if I have need to add a field, I can do it without changing some other function. (If need to, I'll add it in both places). Hard to tell really, for some things it might not make sense at all.

No, you're right. EmptyMessage exists but you shouldn't make it an input or output to an RPC if that RPC might ever in the future accept or return arguments. Since people are super bad about predicting the future, just call it WhateverRequest and WhateverReply and if there are no arguments today, don't put any fields in those types.

I would also not use Any as a request or reply type, but Any, in general, is fairly common in practical use.

https://google.aip.dev/ gives lots of advice for structuring a gRPC API. What you say is mostly in line with that, with some exceptions. For example https://google.aip.dev/135 suggests that a delete operation on a resource should normally return google.protobuf.Empty. Of course, this is implicitly a sum type; you either get an empty response or an error message. If you needed to return anything else, you're not just implementing a delete. Two exceptions they call out are soft-deletes (not just a delete...) returning the resource itself, and a delete that's a long-running operation.

I have been poking around building an eerily similar example[1] service. I am trying to come up with a good pattern, using go-micro, where a team can quickly and easily leverage the protobuf IDL for gRPC/REST interfaces across large teams. It sounds like it should be easy, right?

The tooling around pulling in protobuf at compile time is a huge pain in the ass. I've been spending an hour or so every night over past few weeks searching issues to figure out why things aren't working. This is a great reference point for me, thanks!

[1]: https://github.com/johnbellone/cache-service

I am hearing more and more about gPRC lately, can anyone give me a run down on why its useful and what it solves?

It allows you to execute code on another machine. Your code makes a function call that returns a status / error, except the function runs on another node. It's another RPC framework and there are more caveats like what guarantees a framework makes. Did the function execute exactly once, at most once or at least once.

We have tons of in memory KV stores, I assume this is implemented for fun? If not advertise the killer feature better

this was implemented as a take home assignment I guess

Reply on news.ycombinator.com