alexgartrell

You can't just throw out a platitude like "it's all unix when you break it down." That doesn't actually make any sense.

When you use a bunch of different languages the primary difficulty you run into is interop. Some languages work well together by default and interop is easy, but this is almost never the case. Generally you're stuck making the decision between easy interop (RESTful APIs or your favorite RPC framework) and fast interop (JNI, swig, etc.).

So how do you guys actually deal with interop, other than to limit the need for it?

I don't think latency is the primary issue here, as you're probably really far from your users and the hop is a mild cost compared to transiting the internet.

In my mind the bigger issue is limited throughput, packets per second are fundamentally limited by this approach (tun devices, at least in linux, are generally limited to a single packet per read). Ted even alludes to a 25% reduction in traffic, though it's unclear to me what the link speed is (likely 100M or 1G). In my experience, even with the latest and greatest in linux, you're not going to see reasonable packet rates, and you're still better off doing this type of stuff in the kernel.

That said, tun devices are super cool, and a great tool for anyone to have if they need to do weird stuff like low-level network testing, so I think this is a really cool post. You can read more about it for linux here: https://www.kernel.org/doc/Documentation/networking/tuntap.t...

""" A clean-room implementation of TLS v1.2 by Ashwini Oruganti (an especially timely project given recent events). It's an ambituous undertaking, but she's narrowing scope by focusing on designing and implementing a "TLS API for humans" and building on top of existing lower level primitives. The project will be written as part of Python's cryptography library. """

Not to be a hater, but nothing I've ever experienced in my career has lead me to believe something like this is possible unless you're starting with openssl. Even then, the lack of a code reviewer raises some eyebrows.

Why did you guys need a /10?

Sorry, to be clear I'm more curious about how it's integrated with version control and how things like MVC are done. I really haven't got a clue.

For that matter, are people mostly using HHVM via fast-cgi?

So coincidentally I work at Facebook and know a bunch of these guys pretty well, but I don't actually interact with the Hack stuff at all (as I work in infrastructure doing flash and network stuff, primarily). I recently played around with the Open Source hack stuff and I installed it on my VPS.

Has anyone else played with Hack in a productionish environment? I'm just wondering what deployments and stuff are actually looking like in the world outside of Facebook.

p.s. tutorial that really exposed me to hack is here: http://hacklang.org/tutorial

The short, uninteresting answer is that it's a work in progress. Initially we built FIFO into our caches because it was easy to build and didn't interact badly with flash disk craziness (write amplification specifically). You can read more about our old caching system mcdipper at https://www.facebook.com/notes/facebook-engineering/mcdipper...

Dear any Stack Overflow Developers,

Can you describe the network infrastructure in finer detail? Specifically what type of load balancer are you running?

And what's peak RPS? Where are your network peaks? (I'm guessing major peak US Pacific and minor US Atlantic?)

It's a meritocracy because anyone can grow up to be a banker.

Edit: not 100% sure why this Got down voted but I was being serious. Meritocracy fits as long as the basis of assignment is merit. Just because the system is broken doesn't mean it's not a meritocracy.

Intrusive datastructures were the thing I missed most from C, but boost::intrusive satisfies my desires when it's absolutely necessary.

In general, I'm anti-boost, but I think it's a personal bias and we use the hell out of it at work to great effect. The one thing I'll give boost::intrusive over sys/queue.h is that the type system helps you a lot more to catch issues and the common case is a bit simpler (a struct that exists in a single linked list and a single hash table, for example).

I've been burned many times by something like the following

  struct foo {
    TAILQ_ENTRY(foo) lru_entry;
    TAILQ_ENTRY(foo) hash_entry;
  };
  ...
  void some_magic_func(struct foo *f) {
    TAILQ_REMOVE(static_tailq, f, hash_entry); // oops, should be lru_entry!
  }

Boost intrusive templatizes on the member as well (IIRC) and eliminates this whole class of problem.

> write a very simple UNIX-based operating system in C++ (does anyone else smell a contradiction?)

I'm going to use this as an opportunity to champion C++ for systems programming despite the fact that it wasn't really your argument. I feel obligated to do this because I agreed with you for a long time on this but have changed my mind over the past year or two.

The argument against C++ as a systems programming language has always been that it gives you too much freedom to hide complexity, but I've written systems code in both C and C++ professionally and it's been my experience that the extra burden of having to write every damn thing over again in C (or use a relatively poorly tested container library, as compared to something like the STL) is not worth the "obvious performance characteristics."

I'd further contend that in non-trivial software all perf characteristics are non-trivial, so even though a + b might be doing all kinds of crazy things in C++, you're much more likely to run into problems invoking store_value() which might interact with a number of underlying libraries and system calls. At the end of the day you're hopelessly screwed without profiling and the utility perf works just as well on C++ programs as it does on C programs.

Finally, the majority of code in my experience is either slow path or at least not the bottleneck anyway. You can almost always optimize a systems program by making smarter system calls (or hardware invocations), while optimizing CPU is rarely worth it. Even if you're CPU bound, it's likely that it's in doing something like SSL, wherein you're almost definitely wrong to try to jump into a project like OpenSSL with micro-optimizations. With that in mind, why not allow yourself to move faster by creating a std::vector? When you use the proper conventions, it's generally as fast anyway.

Just because people can (and have, a lot) misused C++ and created way too much complexity with it doesn't mean that it's not the preferred language in the hands of someone who knows what they're doing. (And yes, it's much harder to learn how C++ works in a meaningful way to avoid these pitfalls, but if it's your job, learn it anyway).

I'm an engineer on Facebook's CDN and Edge network, and, in my experience, the hardest places to serve people quickly are South America, Africa, and Asia. You should try to get some timing data for those places.

I chose Facebook over Google two years ago (despite a more lucrative offer from Google), because I wanted to go where I could have more of an impact. The interview processes are nearly identical. One of my go-to questions was a question I got asked at Google, actually.

Anyone have any thoughts on the difference between this 87 key keyboard and the leopold tenkeyless? I've been using an ancient wired 87 key apple keyboard that I love but I'm intrigued.

http://elitekeyboards.com/products.php?sub=leopold,tenkeyles...

I never understood Google's desire to keep things like GGC tight to the chest. Another search engine isn't going to win or lose based upon their ability to run caching http proxies in ISP data centers.

It's really hard to not come across as ad hominem when you're disagreeing with OP's take on a personal experience, but I have very little patience for someone who sees or experiences something negative and does nothing more that cry into the wind. Fix the bug or find a work around. Lots of people managed to have great and very useful and applicable college experiences (I did) so maybe it's the OP who is wrong?

With equity packages, base and bonus, many Google/Facebook employees crack the 200k figure.

So I was a memcached engineer. The memcached team barely works on memcached anymore, because there are no real returns. Being product focused means that it doesn't make much sense to really increase the perf on something when there's other stuff to be done, even if you could do it in a really glamorous way (user space network drivers spring to mind for this -- that'd be pretty cool)

Instead, they focus on superior libraries and different use cases that make the stack more efficient. (Mcdipper was my project here). In infra, you are measured by two things: 1) dollars saved and 2) features made possible (in a very global way, sometimes this means business deals). You also get points for the site continuing to work

Technology doesn't go away. In 10 years, you'll be able to build this rifle with a 3d printer, an arduino kit, and plans from the internet. However you feel about gun control, you've got to acknowledge that strict gun control will be unrealistic short of ridiculous, draconian measures.

My first reaction when I saw this submission was "Holy shit, who leaked kiln?"

It's actually all SSD by default, but you can pin the buckets into memory if you'd like. We generally do this in the memcached-replacement use case and specifically for CDN. However, we do have another use case not emphasized in the article that has a mix largely consisting of smaller values which we store in the bucket cache as direct records on SSD.

In general, if it fits in memory, it's faster to keep it there. If you don't have space for whatever reason, you leave it on the SSD.

Excuse me for stating the obvious but [citation needed].

Apologies to good HNers for feeding the troll here, but I've always been curious about how holocaust deniers could have the balls to carry through with their opinions despite a preponderance of evidence to the contrary.

C++11 is the superior option because it is easier to write correct code with it, plain and simple. And we aren't talking about orders of magnitude difference in throughput or latency, we're talking about a slight increase in CPU idle.

And layers upon layers of object oriented crap is also a problem in C (I've seen it). At the end of the day I've just been burned more by the complexities of building large things in C (particularly when people do reference counting in C) than I have been by the complexity of c++ in general.

So I used to agree with you, but I've done extensive performance critical C development at Facebook (memcached, a new thing we are about to talk about) and I have also done extensive performance critical c++11 development (our layer 7 load balancer for http) and I'd have to say that c++11 is the far superior option if you really, truly understand what the compiler is doing to your code (a huge caveat).

Unique_ptrs are a total game changer, and the ability to use closures and lambdas when you want to set a callback function instead of a function pointer with a context pointer you have to cast and decode is absolutely huge for readability. maybe we aren't getting every last bit of performance out of it that we could with C, but it works at our pretty ridiculous scale, so I think C might have been a premature optimization for us.

These are probably off-net cache servers for Google, which means that to save money on transit costs, Time Warner has installed special servers and Youtube references them directly for Comcast subscribers (lot's of things give this away, but I wouldn't be surprised if it was DNS based and they know based upon your resolver. Try switching over to a global but not eDNS based resolver and see what happens).

If that is the case, it's not unlikely that the off-net boxes are simply overloaded for people in this guy's usage area.

Fault, error, failure, hazard. Technically speaking, nothing actually matters until it becomes a hazard (if you're a cynic, this means "costs you money"). I think that this is one way of thinking that engineers could benefit a lot from, in that framing issues within these boundaries gives you a lot more perspective.

Write patterns still matter until you're talking about like megablock granularity. Mmap will swap out pages at random (relative to disk layout) and page granularity is far smaller than a megablock. It's certainly possible for controllers to handle this properly, and I don't want to tell you that they never will, but even the very expensive PCI-E flash we use at fb demonstrated this "bad behavior".

If I had to guess, I'd say better cache locality. It's tremendously more expensive to go to main memory for something than to go to the cache.

I'm sure the auditing and control is more than enough to stop your average creeper employee from reading normal people's inboxes, but I very much doubt that it's enough to stop a very smart, very determined spy from doing the same. At the end of the day, someone has root, and that guy can do pretty much anything.

I'm confident that Google is doing a better job than pretty much anyone else, but this problem is a more or less unsolvable one.

Edited to add that another interesting idea is that the people who man the DC's are actually pretty sparse (relatively few people for a lot of servers) so it's not inconceivable that one could trigger a failure on an important box, take down a replica of the figure's mailbox, swap out the drive for RMA and then do a quick copy. I bet this would be easy.

I guess my point is that no level of internal controls at any company can actually stop a determined government. If that were true, governments, which are much more paranoid than tech companies, would have eradicated spying a long time ago.

I know that this is a pot shot against the American legal system (especially with regard to copyright stuff), but it really makes no sense. A spy would likely be no more expensive than a legal team (in fact, he'd be subsidized by his salary at Google) and would be monumentally more effective and secretive.

Beyond that, the hard part of training a spy to get into Google would be getting a good enough computer science student involved. From there, it's really just a matter of teaching them to cover their tracks semi-intelligently. However, given what I'm sure is a mountain of completely legitimate reasons to look at user data (for example, to resolve data corruption, investigate malicious users, etc.) and an inconceivably larger mountain of user data to look at, I don't actually think it'd be that hard to get away with it.