scaleout1

from the article they are thinking of fining up to 3.x million USD

I read your comment but it's not clear to me how what you are suggesting would work practically. My friend and I are bootstrapping a startup that would primarily cater to Pakistan audience. We are hosted on AWS and use Stripe for payment processing etc, like most startups. Even if we wanted to host something in Pakistan (which we don't), no such infrastructure exists and we are in place to hire any dev resources in Pakistan either.

Per the article, if you are not hosted in Pakistan, Govt will fine you 3.x, which will probably kill our project, but it's not clear how Pak Govt can find us if we have no presence there?

Last time I work at a massive Monorepo, half of my team was running got fetch as a cron job. It was an extremely painful experience

Yes thats the pattern I ended up using with the help of `custom_error` crate which automates the implementation of `From` for your custom errors. I guess the disconnect for me was that in Java/Scala all custom exceptions extend `Throwable` so types always line up, in rust custom Errors are disjointed so you have to wrap everything yourself to align the types.

Does it help with Error enum coming for third party crates? I can see how it will help with my own code though

I can see your point for unit testing but I am struggling to see how it will work for Integration testing. For example I would like to initialize a connection pool (or other heavy object) only once for the entire test, I .. havent figured out how to handle that except for creating a new one in each test. Any pointer on how you are handling those? Thanks

Recently started introducing Rust in my team and here are my notes on some of the issues we have run into so far. We are primarily a Scala/JVM shop with a little bit of Python.

- `Error Handling` is a bit of a dumpster fire right now. Rust has something called a `Result` enum which is similar to `Either[Error,T]` monad in Scala however every single Rust crate I have used so far its creating its own `Error` enum which makes it really hard to compose `Result`. Ideally I would like to chain `Results` as `e1.and_then(e2).and_then(e3)` but its not possible due to incompatible error enums. Ended up using `https://docs.rs/crate/custom_error/1.3.0` to align the types.

- A lot of basic things are still influx and community wide standards have not been established. For example: I needed to externalize some environment specific settings in a Config but couldnt figure out where to put non-code assets in a cargo project and then how to reliable read them. In JVM world `src/main/resources` acts like a standard place for stuff like this but that patterns has not been established yet.

- Distributing code inside the company is hard because there is no integration with Artifactory or similar tools. We are directly referring git sha in Cargo right now and waiting for better solutions.

- Rust comes with a default unit test framework but it pretty bare bones. I havent seen examples of test fixture, setup/teardown support and loading test configs etc.

- I really like Rust compiler because of really good error messages it produces but its really slow and you start to notice it as you add more code/external crates

-IDE support is good but not great. I am using IntelliJ with Rust plugin as we use IntelliJ for Scala/JVM and it is nowhere as good as even Scala plugin which is pretty mediocre in itself.

Overall I am pretty happy with the language (except for Error issue) and most of my gripes are around the ecosystem and tooling around the language. Hopefully these will be resolved as language gains more momentum

wait till they break Google voice into Google talking and Google Listening

Unfortunately in Bay Area and specially at FAANG you need 0% of job "Requirements" on your resume as they will leetcode the shit out of you in the interview without asking one relevant question pertaining to your resume or even to the actual job. One of the reason given for this is that the stack used at these companies is completely homegrown so your experience using framework $X for $Y years has no relevance

Thanks for the reply, got a few more additional questions for you :-)

Lets say you are counting distinct ips used by `users` using HLL. Lets say you start getting DDOSed by certain users since I am assuming you are not doing s shuffle before writing to FDB, you will be locking the user, reading HLL, deserializing, merging and writing back to FDB from multiple machines which will results in a lot of rejected transaction and retries. My question is whether retries unwind fast enough or you will end up dropping data on the floor as you will exhaust the retry count

Hey man, thats pretty cool and we do exactly the same using Cassandra instead of FDB. Since Cassandra doesnt support transaction at high volume (100K tps) we do a shuffle so that all the same key do read/modify/write from the same machine. It seems like with FDB you can get away with it as it supports transactions? My question to you is what is the volume your system is operating at? Also how does it work for skews? Lets say you need to update HLL for a key that is heavily skewed, does your FDB transaction unwind fast enough not to slow down the whole system?

Why are almost all the replies by ex Google engineers? I thought I would see an even distribution from all FAANG companies but replies are pretty heavily skewed toward ex google employees. Is this because 1) Google culture is completely different from other FAANG companies? 2) Engineers at other FAANG companies dont quit their job? 3) or they dont browse HN in their free time?

This maybe offtopic so I apologize in advance but when people say "Asian-Americans" does it include South Asians as well or just East Asians? As a south asian this has caused me a lot of confusion while talking to people in bay area

Interesting project although cant say I am happy to see SQL being used in Streaming Systems like this. In my last two jobs I had to write frameworks and tools to enable "Data Scientists" and "analysts" to write production jobs and problem I have run into with exposing SQL to this class of user is that every job end up being its own special snowflake with deeply nested SQL with custom UDFs mixed in for good measure. Due to "unique" nature of each its significantly increases the support and maintainability cost. I have to come the conclusion that a typesafe api with map/filter/flatmap is much better API to expose that Stringly typed SQL. I am curious to know whether Uber is running into similar support issues?

Finagle/Finatra based microservices at Twitter. Using the same stack at a different company now

Spring vs. JavaEE is a topic that stopped being relevant five years ago. I cant believe anyone seriously considering any of these heavy duty teach stack in 2017. There are much much better choices out there even for Java developers, and if they can come out of their comfort zone a bit there is Clojure, Scala, Kotlin etc with MUCH nicer fraemworks

Few things that comes to my mind

- Have clarity around what time people are expected to arrive/leave. If your company offers free food, dont schedule breakfast super early and dinner super late.

- Dont schedule meetings at 5:00 pm

- Dont change product direction/design based on meetings you had at an after work party

Basically acknowledge that people in general and women in particular have responsibilities outside work. Strive to have work life balance for everyone in your team and not just fresh out of college kids who can afford to sit at house till 9 pm.

when I was college I thought all the data structure and algorithm courses were complete waste of time. Primarily because I was working as an intern on the side, making crud applications, slinging xml, writing DAOs etc.

For first five years of my career I never had to touch any of the stuff I learned in school and I was particuarly happy that I mostly mailed it in in those classes. Eventually my career evovled into dealing with data at massive scale and working on some of biggest services on the planet and the way I have taught myself to program completely changed. No longer it was possible to just sling code and hope that it will just scale to million of users. All the stuff that I slept around in class was relevant again and I had to go back to coursera and take those classes all over again. So moral of story, if you will be slinging webapps rest of your life you probably dont need to know Big O, different search algo, linear algebra and statistics etc but if you think you will be working on stuff thats coming around like automanous cars, IoT, augmented reality etc, you should definitely read up on it

Problem with gradle is that every build script is its own special snowflake as Gradle does not impose build structure. I am glad that they are moving toward Kotlin as Groovy is awfully slow

Thats an interesting approach. One question though, in our use case user often travel from city to city and country to country. How do you model that if you are only using local DC and local replications?

the trick is not to go through the recruiter. Instead ask Committer on their OSS project or your internal contact to vouch for you to the hiring manager.

This article makes me sad. Interviewing in our industry is so broken. I have been out of school for a while and switch jobs every few years and this is the technique I use to beat the bullshit interview process

. Make a list of companies that I would apply to and sort them from most interesting to no-way-in-hell-i-am-working-here order

. spend a weak reviewing typical algo/data structure questions

. For the companies that I absolutely want to work for, I review every single glassdoor review and write down the interview questions. Remember, most companies have question banks and most interviewers have favorite questions which results in same questions being asked over an over again. You want to exploit that

. Then to get over my interviewing jitters, I interview at a few companies where I would absolutely not work at. This results in no pressure interview practise and you can literally laugh at their asinine interview questions and walk out

. Finally, for the companies i actually want to work at, I try my best to get rid of phone screen. This is usually accomplished by dazzling them with my decent size github profile, contributing some fixes to their OSS project or finding someone who already works there that is in my alumni network

. Then when you finally arrive for the interview, you have real world interview practise, they are already impressed with your github profile/references and biased toward you versus some random joe off the street and you have made sure you have a pretty high probability of getting a question that you have already seen or is similar to a question you already know.

This technique has helped me get Jobs at top 5 employers in the valley along with a few startups. The reason I am posting this here is to demonstrate how broken, unfair and easy to game this whole process is

Yes this is based on teams of 1000+ developer at a non google company

Please dont do it unless you are google and have built google scale tooling (gforce/big table for code cache)

benefits of monorepo

* change code once, and everyone gets it at once

* all third party dependencies get upgrade at once for the whole company

cons (if you are not google)

* git checkout takes several minutes X number of devs in the company

* git pull takes several minutes X number of dev

* people running get fetch as a cron job, screwing up and getting into weird state

* even after stringent code reviews, bad code gets checked in and break multiple projects not just your team's project.

* your IDE (IntelliJ in my case) stops working because your project has million of files. Require creative tweaks to only include modules you are working on

* GUI based git tools like Tower/Git Source dont work as they cant handle such a large git repo

Google has solved all the issues i mentioned above so they are clearly an exception to this but for rest of the companies that like to ape google, stay away from monorepo

As someone who has used Heron (along with MillWheel, Spark Streaming and Storm) I feel like this announcement is too late. The biggest thing Heron offer is raw scale but since they decided to use existing Storm API, it has the same shitty spout /bolt API that Storms offer. In contrast, Spark streaming/Flink/ Kafka Streaming are all offering map/flatmap/filter/sink based functional API. At twitter most teams used SummingBird on top of Heron to get the same functional API but summingbird didnt get a lot of traction outside twitter and I am not sure how actively maintained OSS version of summingbird is. Even if you bite the bullet and decide to use SB with Heron, you will still miss out on a lot of usecases as SB was mostly focused on doing read/transform/aggregate/write whereas most streaming problem that i have noticed outside of twitter involve doing read/transform/aggregate/decision/write. I suppose you can implement decisioning in SB but i havent seen it done.

Comparing Heron to google millwheel is interesting because of the design choices they made. Heron support at least one and at most once message guarantees but at Twitter most job ran with acked turned off so it was at most once with acknowledged data loss ( they had a batchjob doing mop up work to pick up missing data). Google on the other hand implemented exactly once semantic by doing idempotent sinks/ watermarking and managing out of order messages plus deduping support. Since both Flink and Spark will be implementing Apache Beam (millwheel's predecessors) model, only reason I see someone picking heron instead of Flink/Spark is that they are operating at massive scale that flink/spark dont support yet

When you suddenly realize that your "big" data is not really that big!. Who needs a Hadoop/Spark cluster when you can run one of these bad boys

Here is a pdf going briefly into kafka comparison and design motivation http://goo.gl/J9XdsG . During my time at twitter I remember when Twitter switch from Kafka to distributedlog. They have an internal layer that adapts Kafka API to distributedlog, I am not sure if they have open sourced that

Heron is half Java and half c++ and could have been written on top of finagle had a different team inside Twitter had implemented it