HNHacker News
TopNewBestAskShowJobs

spimmy

355 karma · joined November 19, 2012

CTO/cofounder at honeycomb.io
submissionscomments
spimmy··on Instrumentation: The First Four Things You Measure
yea that's not good advice, but "don't rely on overstuffed dashboards" is good advice.

you should only have simple dashboards (and alerting) for KPIs and end to end checks. Everything else should be instrumented and debugged using a real time sorting and slicing tool, esp if you have a complex system (microservices, distributed system, polygot persistence).

spimmy··on Scuba: Diving into Data at Facebook [pdf]
Yay!! So happy to see Scuba get more visibility. I can't even count the number of times I said, and heard other engineers say, that the thing they would miss most about Facebook was Scuba.

That's the whole reason why we built honeycomb.io. If you're a FB expat, check us out.

spimmy··on Sleepless nights with MongoDB WiredTiger and our return to MMAPv1
Oh, I just realized that by "company's engineering" you meant CleverTap, not MongoDB. Got it.
spimmy··on Sleepless nights with MongoDB WiredTiger and our return to MMAPv1
I'm agreeing with your comment, not arguing with it.

The problems we had moving from mmapv1 to rocksdb were almost entirely based on the rewrite of the query planner in 2.6, not the storage engine itself.

The WT storage engine is far less mature than RocksDB. Only a very tiny amount of code in the Mongo+Rocks integration is actually new code, it's just the thin layer of glue between mongo and the storage engine API.

It was a tricky migration because we made the decision to do it early while so much code was still new in mongo for that version. We wanted to help give feedback on the storage engine API because we had pushed so hard for that feature, and because we were pushing the boundaries of what the mmap engine could ever reasonably be expected to do.

(Side note: OMG, anyone who measures their request rate in anything other than per-second is putting up a giant red flag that they don't know what they're doing.)

spimmy··on Sleepless nights with MongoDB WiredTiger and our return to MMAPv1
The article was written by people who don't understand very basic things about databases.

To be fair, this is part of Mongo's pitch: come use a database, you don't need DBAs or people who know things about databases. Which is always gonna be kind of a lie. So they invite this on themselves.\

I hate when I get reduced to sounding like a mongo apologist, but it's their own fault for not following any best practices with data.

spimmy··on Sleepless nights with MongoDB WiredTiger and our return to MMAPv1
I have plenty of rants about the company's engineering, priorities, etc. I just hate people saying dumb shit. :)

Workload QPS is a meaningless number unless you know what the workload is. At Parse, we used RocksDB (a LSM tree write optimized storage engine) to get something stupid like 9x faster writes. I don't know where you're seeing 600 QPS ... (we got 100x that if we were optimizing for writes) ... but it still doesn't mean anything unless you know the size of the document, whether it's inserts, queries, updates, whether it's hot spotting on a row, whether the lock is at the collection, document, database, or global, etc.

If you're going to switch storage engines, you have to use some sort of capture replay technology or you can not have any confidence in the change. Use what we wrote at Parse: https://github.com/ParsePlatform/flashback. We spent ~6 months teasing out bugs in rocksdb, mongodb, the 2.6 query planner rewrite, etc before we even started to make the switch.

If you don't do that, you don't get to complain about the results. Sorry.

spimmy··on Sleepless nights with MongoDB WiredTiger and our return to MMAPv1
No, you're being dumb. :)

Was mysql "ready for production" when google used it? facebook? Lol.

Storage engines are hard. I'm boggled that they recommend that you keep the old replica around for ten days. When we did the MMAPv1 migration to RocksDB at Parse, we kept at least one MMAP replica around for 6-9 months, as we worked through the integration of Mongo + rocksdb ... even though RocksDB the storage engine is powering fucking Facebook.

I get so fucking annoyed at the Hacker News armchair brigade that wants to talk shit about MongoDB, when they don't know a goddamn thing about databases -- or, from the looks of it, Facts About Stateful Services In General.

Eyeroll.

spimmy··on How to Survive an Acquisition
I think you're right, and I completely intended it in a practical, non-incendiary way. People would constantly say they weren't there to pursue personal, political agendas, they wanted to have maximum impact and do what was best for Facebook. It's like the #1 catchphrase you have to memorize to get ahead. It makes sense. In most cases, it's even true.

I may have a separate set of issues with Facebook People policies, but there are a ton of really passionate, amazing engineers working there who deeply believe in the mission (or at least their local specialty). I miss many of them very much and wish them all well.

spimmy··on How to Survive an Acquisition
"Try me."
spimmy··on How to Survive an Acquisition
Yeah, or just completely oblivious. Most of the technical "advice" they insisted on giving us was just completely wrong.

Also, gotta say, I was never as self-conscious about being a woman in tech until we got acquired by Facebook. I've never had so many experiences of walking in to a room (as a tech lead) and having a dude look me up and down and say, "ummm, how technical are you?"

spimmy··on How to Survive an Acquisition
This completely describes our experience. Thank you.
spimmy··on How to Survive an Acquisition
I am absolutely venting emotionally, I thought that much was clear. :) But I heard a variant of this in dozens or hundreds of meetings, generally when a project was getting shut down or canned or reorg'd, so it's not true to say that it's not true.

It's not even a bad slogan. Company priorities should come before territorial interests. This is why it's important to work in a place that shares your values at a high level.

spimmy··on How to Survive an Acquisition
OP here. Engineer/tech lead.
spimmy··on Facebook is closing Parse
Nice. These guys are legit. (RIP, Parse :( )
spimmy··on How We Moved Our API From Ruby to Go
You are totally right, most of the stuff that really hurt us was Rails middleware magic, not Ruby itself. I should have been more precise -- grumpy rails bashing, not grumpy ruby bashing! FWIW we still use Ruby on Rails for our website and it's great for that.

I'm hoping to get some followup posts from the backend eng team on specific interesting problems we ran into during the rewrite.

& yes deploys with go are the freaking bomb :)

spimmy··on How We Moved Our API From Ruby to Go
Hard to say. Ruby really did let us move and ship products insanely fast, without having to sink precious engineering time into boilerplate and standard libraries. Most startups fail, and it's often because they couldn't move fast enough. So I don't think it was a bad choice at the time. We were able to do the rewrite once we had grown up a bit, gotten acquired, hired more engineers, and had a proven business model that was worth investing in.
spimmy··on How We Moved Our API From Ruby to Go
Gustavo is the best. ^_^
spimmy··on How We Moved Our API From Ruby to Go
(we can, and do, do a lot of things to limit the impact any one app can have ... but there are limits. Async is much better for our API model.)
spimmy··on How We Moved Our API From Ruby to Go
Keep in mind that we're a platform, not a website, and we have very little control over things like schemas, slow queries, inefficient looping requests that our developers send us. We can't optimize 500 thousand apps' queries for them.

So for example if an app does something bad like performing a full table scan on every request against a 300 million collection, so every request to that backend is timing out at 30 seconds, and there are thousands of them per second, well -- pretty soon your fixed pool will be full of requests timing out to that backend.

spimmy··on How We Moved Our API From Ruby to Go
Good points. Technically we still use rails for our website, and have no plans to move off of it. Rails is great for websites! It's just our API that we rewrote in Go.
spimmy··on How We Moved Our API From Ruby to Go
optimizing won't help you here, unfortunately. the process-per-request model is fundamentally flawed past a certain scaling point.
spimmy··on How We Moved Our API From Ruby to Go
ahh, thanks for the clarification. :) (edited the post to be more accurate)
spimmy··on How We Moved Our API From Ruby to Go
it's more like, getting paged in the middle of the night over and over and over is a known mental health problem :)
spimmy··on How We Moved Our API From Ruby to Go
we had to way overprovision to handle even momentary spikes in availability from any backing store. we aimed to run at around 20% unicorn worker utilization under normal conditions.
spimmy··on MongoDB and RocksDB at Parse
We have a lot of edge cases in our data set and usage model, no question. Glad it's working well for you. :)
spimmy··on MongoDB and RocksDB at Parse
There's no article, sorry, I just happen to know them.
spimmy··on MongoDB and RocksDB at Parse
Amen brother. You should never base your db decisions on either marketing copy _or_ HN know-it-all complainypants. :)

We actually evaluated TokuMX extensively last year, pre pluggable storage engine. We might have pursued it if they had implemented a compatible oplog at the time, but with a migration path that consisted of dumping and importing production data -- and no way to re-elect the old primary if there were any production problems -- that made it simply a non-starter for me.

They did eventually implement a compatible oplog, which was a good product decision, but the entire TokuMX engineering team recently quit Tokutek en masse so it's still not a great option in my book. Too bad.

spimmy··on MongoDB and RocksDB at Parse
tszming -- will take care of it, thank you!
spimmy··on MongoDB and RocksDB at Parse
ah but those particular compression / write rate stats are compared to mmapv1, yeah
spimmy··on MongoDB and RocksDB at Parse
We're testing with both. Somewhat hindered by the fact that WT often dies trying to import our data, or crashes, or loses data in weird ways. :( I believe WT will eventually get there, but rocks has been unfathomably more solid for us so far.
← PreviousPage 2 of 3Next →