acroback

Oh yes, we were asked to write positive reviews on Glassdoor.

Guess heat I did? :)

People get laid off all the time. Business is not hostage to your performance.

In the end we all are just numbers of the XL sheet. Treat your company just like one.

All the talk about mission reeks of naivety. Seriously, I can understand the emotional outburst but it is unnecessary.

As someone who went through this couple of months ago.

1. Code by writing utilities to help your engineers.

2. Never code something which is critical for the release, you will end up blocking everyone.

3. Let engineers make mistakes( which are not critical) without blaming them.

4. Facilitate communication not lone wolf culture.

5. Avoid taking decisions without consulting your team, which invloves them.

6. Keep practicing leetcode or puzzles or programming or whatever you do, because you will be replaced sooner or later.

7. Don't get emotionally attached to people, it's not your company. Work hard and fair. But look out for yourself.

8. Manage expectations clearly with upper management, don't let team feel unnecessary pressure from hyperactive product teams.

9. Clearly set expectations, chalk out a plan and work towards it with each Engineer.

10. Relax, it is not a 1 week task, it will take years if not months to see some visible change.

Get some SSDs

You give me hope.

I have been putting off a Google interview for a whole year now.

I have real experience building Cluster Filesystems, Distributed Caches, TCP/IP Control plane software and low latency Ad platform for more than a decade.

Sounds good on paper, but I can guarantee I cannot solve most of these puzzles without actually solving them beforehand. Why? Because I do not have cycles or time to solve these on my own. I have a system to build, Engineers to help and a product to ship. Family obligations do not help.

Seriously, Is this normal? What does this tell me? I should not appear for Google interview and waste their and my time.

Some of the replies here suggest that Googlers just push protobufs? Damn, that is one hell of an interview to push protobufs around :(.

Glad, atleast they are upfront about puzzles, so that's good.

So basically what ZeroMQ does at far better speeds with better latencies and reliability with a fraction of cost and it doesn't crash or stops operation out of nowhere.

Why is everything SaaS, when it is fairly easy to use existing proven solutions?

Is it just money?

We uses ZeroMQ on production as part of a message layer which provide very tight and predictable latencies sustaining more than 300K QPS across multiple services and across 3 data centers(GCP, AWS and private Datacenter). This is a stellar testament of how good ZeroMQ is as a mature solution.

I think the reason we are successful with our design is because ZeroMQ is just great for what it does. It never crashed or gave problems to us, only issue we saw was because we did not understand how it worked internally.

Had it been written in C, we would have been able to hack it a little bit to our tastes but I am not complaining. I can understand why someone would use C instead of C++ for high performing libraries.

Oh on the subject of C:

This is what we have :- Main service which is written in C does around 8K QPS at peak hours(on a 2 core 4GB Amazon EC2 instance), so we have around 35-40 instances in the cloud. The cost to run this service is low for us, we love it.:)

Biggest issue, people are bad at C and writing correct multithreaded code in C is an Art.

TDD need to die. This is a curse.

There should be integration tests along with some property based tests and fuzzy tests. Usually catches a lot of things.Invest in monitoring and alerts too.

TDD is like relying on debugger to solve your problem. Is debugger a good tool? yes,it is a great tool. But using it as an excuse to avoid understanding what happens under the hood is plain wrong.

The problem lies in industry where software engineering is not given any value but whoteboarding and solving puzzles is.

Software engineering is a craft honed over years of making mistakes and learning from them. You want code asap, kick experience engineers get codemonkeys in and get a MVP.

Quality is not clever algorithm, but clear conscise logic. Code should follow the logic, not the other way around.

Clear > clever.

Half of the times it just sits there blank. Sometimes it loads in 2-3 seconds. Rest of the times it loads within 1 sec. Rarely, it also shows a default ad( I work in same industry, hence the term).

How is that great?

PS: Not talking about native ads.

How is this monstrosity going to understand that it is not just practice which makes humans good at team work in Dota2?

AI has long ways to go before it can defeat humans in a complex game like Dota2.

Start with writing Unit tests for basic functionality.

Do not do the cardinal sin of writing unit test for every goddamn function in the codebase like a rookie.

Start fixing code, one by one only if it can be tested by Unit tests.

Remember you need to first learn the code base and context because harping "complete mess". Maybe they did not have time, maybe timelines were more imp. That is what industry is about especially when you do not have enough Engineers. Plus this new wave of PMs who want features at any cost, doesn't make things any easy.

Good luck

I see a disturbing trend in industry especially in Bay area from my experience.

Young engineers are bad at debugging, bad means real bad but good at programming puzzles.

This points to the fact that people value puzzles over experience and in turn miss the big picture.

Sure you can code up a balanced RB tree in 5 mins, but what will you do when packets start showing up in bursts on your service? Can you find out that on a live system?

I despise this new Bro culture of mugging programming puzzles from leetcode and hackerank. Faltering at first real world challenge is not only disappointing, but an insult to craft of engineering.

/Rant over

New functionality, breaking existing perfectly fine and working functionality is a big no.

One of our core services is written in C, it never crashes, it sustains QPS no other service can in the entire ecosystem, why change it to something new unless there is a solid reasoning to it?

I kid you not, one of the companies we integrated with gave us 2 64 Core VMs in their private cloud to run our service. We run 3 instances of our process on each VM along with Redis taking about 60GB of memory in total, with 59.996 MB of memory taken by Redis. And it was written in Go. Apparently, we miscommunicated that we will be using a GC language for this service. They assumed Java and based on their experience allocated 2 64 core VMs with 256GB memory each :D.

I think I did not present this correctly, I apologize for that.

What I mean this this. Any IDL interfaces live at their functional boundaries. e.g all proto definitions shared internally by java services live under jumanji/proto/, all proto definitions shared internally bu Go service live under goat/proto/ and so on so forth.

But anything which is shared in a public manner e.g any proto definitions between Java and C live outside either of these repos in top level common_proto/ repo. The build system is configured to take care of pulling the correct proto version from common_proto repo.

We use maven for this, but nothing much we can do about that. Build systems are ugly and we have to live with maven in near future.

EDIT: Oh that said, stop using required field in protobufs if you are stuck using proto2 like us, whoever decided to add required field to proto2 grammar made a mess. Things change and deprecate repeatedly, gazillions of required fields are annoying and recipe for disaster.

No, we do not want to pay extra to run Java services bloating our measly VMs. Also, we found that it is far easier to get consistent latency with Go code than with Java code.

Again, YMMV benchmark for your use case.

That said, Java doesn't have go routines, doesn't provide low level access to memory, benchmarking framework is easy to work with in Go. Go gives access to sync Pool which when use correctly gives extremely good gains.

One of our measly dual core VMs, can do more than 40k QPS with predictable latency at 99 percentile. With JAVA oom panics were common and latency percentile was all over the place. Sure, you can perhaps tweak the Java code, but it was far easier to do with Go.

This means either your contract agreement between services is not language agonistic or someone did not do their job when Interface schema was changed during reviews.

Just like any other software, interfaces live in separate repos e.g protoplasm/ for proto-buf definitions. avrobber/ for avro and so on so forth.

So, any change to protoplasm/ triggers automated tests on all other services irrespective of boundaries.

Haha, I made that up to gain free internet points ;)

Totally technical considerations which engineers must present and justify.

e.g When we added Go, we had to present what it brings? A good testing framework, light weight(subjective), goroutines(which fit our use case), in built benchmark support, mature support of third party packages etc etc.

This is a classic case of not understanding micro services and trying to fit a problem around a tool.

At work, we have close to ~50 services(no one calls them microservices), but they do not suffer from this brittleness. We segregate our services based on languages. So, all C services go under coco/ , all Java services go under jumanji/ , all go services go under goat/ , all JS services go under js/. This means, everytime you touch something under a repo, it affects everyone. You are forced to use existing code or improve it, or you risk breaking code for everyone else. What does this solve? This solves the fundamental problem a lot of leetcode/hackerrank monkeys miss, programming is a Social activity it is not a go into a cave and come out with a perfect solution in a month activity. More interaction among developers means Engineers are forced to account for trade offs. Software Engineering in its entirety is all about trade offs, unlike theoretical Comp Science.

Anyway, this helps because as Engineers we must respect and account for other Engineers decisions. This methods helps tremendously to do this. No one complains, everyone who wants 1000 more microservices usually turns out to be a code monkey entangled in new fad, or who doesn't want to work with other Engineers.

You want to use rust? There is a repo named fe2O3/, go on. Accountability and responsibility is on your shoulders now.

If you think about it, an Engineer is tied to his tools, why not segregate repos at language level instead of some arbitrary boundary no one knows about in a dynamic ecosystem?

Why the fk do these stupid companies then ask really hard Algorithm puzzles in interviews?

This is insane, what is imp? Giving too much info or not, asking more questions or not?

Oh man, sometimes I hate this process of interviewing.

I agree with you.

I work for a Startup in valley, and I agree. Our founder CEO was fired recently. Why, you may ask?

Because he was a really poor Manager, so poor that even Senior Engineers could see it clearly. Luckily board came to senses and replaced him with a proven CEO.

Sad but that is the truth, to scale you need different skillsets and off course being a good Manager is just one of those.

I think people are misinterpreting support for try catch family in some high level languages as lazy approaches.

But truth is it is the programmers who are lazy, we should not blame language for providing a feature which some programmer abuse.

I have never seen a Java programmer profiling their code for Memory or CPU at work and I am here running my shitty C code under Valgrind and a profiler all the time.

I guess, it is programmers and not the language.

Go's error != nil approach though rudimentary actually forces lazy programmers to complain, since they cannot abuse it.