2,256 karma · joined August 4, 2011
1. Figure out which Kafka broker you're using. The concept of a consumer and consumer APIs 0.7 is different from 0.8 which is different from 0.9 and different still from 0.10. Ranging from non-existing, to quirky, to finally a good design that's working - but you need to make sure offsets are committed in time.
2. Offset management is a thing. If you're lucky and using recent brokers you're good, but you'll still have to make sure timings for submitting offsets and heartbeats interleave in a way that Kafka doesn't think consumers are dead. If you had problems imagining this scenario - exactly. This was a very hard race condition to find, that's solely up to you and your motivation to fix it and not scratch it of in favor of "a random glitch".
3. Kafka clients are still radically different, supporting different versions. You need to be lucky to use a language and a platform that is in harmony with latest consumer APIs. However - I'm certain it'll converge. The ecosystem will converge slower.
4. Out of reasons Xorlev mentioned, you will find yourself against the wall, making sure each job task is idempotent. Suddenly - this becomes a people management problem too.
All of these can (and probably _will_) be solved, however I feel that (2) and (4) will always be there, because that's part of why Kafka is so great.
In addition, I think Kafka is one product which you _must_ read the "whitepaper"[0] for before you want to build consumers for it. The first reason - because it's an innovative design, that might come in handy in every day life if you're an engineer, and the second reason - is to understand the founding context in which it was created - logs and why there are so many tradeoffs that were done for it to be amazing at that, and to realize that this original founding context was _not_ transactional jobs.
Switching gears now. Many organizations find Kafka as a much needed cure for data processing pipelines, and pushing events and messaging as a first class citizen in the organization from a _data_ point of view. For that, Kafka is amazing. With it, you can realize the dream of having an "event mart" where groups, teams, consume and publish their view of the world, processed, as a message stream, and someone can pick up that stream and build a completely different product on top of it (not a perfect example but one we can all relate to - think about Twitter's firehose).
The perception problem is, that once this floods the organization, there's little to do, to use the same mindset to build _operational_ and transactional queue systems, where you don't process events or data, but perform tasks. Unfortunately that's not true. I'd be happy if there were stronger education about this from Kafka's side.
For the kq project - I wish best of luck and I'd be interested to see it unfold. Code is very clean and I feel it's inviting to just read and learn from it - kudos!
1. There's a bunch of objects that need to pass down the ffi boundaries py->go 2. compute 3. There's a bunch of objects that need to pass up the ffi boundaries go->py 4. Python now continues as usual with a bunch of processed objects
In that case, yes this would be a problem. The way I'd resolve it is by planning the ffi boundaries accordingly. I'd make python do as less as possible, and pass just declarative "instructions" to go. In this case where's the file location and where's the sourcemap file location (and perhaps where to dump output to if that's the case). And go doing as much work as possible to make sure there's only a minimal number of objects passed back if any.
It may _feel_ like a hack but ultimately its the same approach if you were to make a "sourcemap server" making python code communicate with it over RPC.
If this is not the problem then I'd love an example of what you meant
IMHO the end result is more maintainable, readable, and accessible from FFI point of view. Regarding the performance, so Go has a GC, but I'm wondering if that would affect things dramatically at all.
Here is the Ruby side FFI code: https://github.com/jondot/scatter/blob/master/lib/scatter.rb
And here's the "native" part: https://github.com/jondot/scatter/tree/master/ext
Every now and then I keep looking at Rust and how it can integrate with higher level languages, the last time I really wanted OpenCV to work well with Rust. I think that's a big selling point. So far, to me, it's not perfect yet but it may get there.
From a pragmatic point of view, I imagine Sentry getting more bang for a buck with Go as there would be less wheels to invent from an ecosystem POV, and from a maintenance POV it would be closer to Python. But that wouldn't advance any of the Rust ecosystem at all, and we do need that as a collective.
First, here's two reasons why interview processes suck. One, is that no one wants to help recruiting, or HR, do their job. Helping them is an errand and is done in a robotic way. You end up with an idiotic checklist that is a shame to great engineers. Second, is when the team finally realizes idiots joining the team are hurting the team - they will go about and have more motivation to support the recruitment process. Then, when you let a group of people devise such a task you get: an average process designed to leave out the human parts (shift bytes in array) and fit each person's interview style, or a crazy idiotic processes made by people mentally jerking off to each other (implement raft. in assembly). A team can be a company or an actual software development team.
The right way to do it, is this. I believe a single person, who cares about developers, who is passionate about developer experience - should build his/her team's interview process. He/she can take feedback from the team, but that person should eventually build the process and make the calls.
In this specific case I want to believe there is an accurate answer, and a correct answer, and they wanted the correct answer, or at least to see him negotiate his way to the correct answer even though it is subpar (although his accurate answers are impressive). That being said, I'm giving Google credit here. There is a small chance it might just be one of the first options I've described here.
Obviously the conclusion that follows is that I think React Native is going to be a revolution for mobile development. Not only will it succeed, it will get copied, and you'll have many variants of the same like-minded platform (open source, developer-oriented, low-barrier language).
Regarding the article, my point of view is that some points in this article are valid, but are trivially obvious:
- Javascript is Javascript, take it or leave it. Flow and Typescript is an evolution that to me looks good. It is one language that was adopted to be user-facing in React Native. According to Apple's Developer terms 3.3.2 and 3.3.3 only interpreted languages that run on JavascriptCore are allowed to be pushed to market.
- You can't know what the future holds. I was part of the Ruby community at the start, then Node, then Go. I was hard core into .NET, and before that Java. I saw platforms and technologies rise and fall, and was surprised over and over. I was much into Silverlight, a technology that Microsoft wanted to take over native development on the Web. Guess what happened to Silverlight? THAT was very nasty. We all know how that feels, but that's nothing to attribute just to React Native.
- Regarding patents: I believe we will have alternatives to React Native long-term, with the same properties but we'll still choose React Native :). However I also believe Facebook doesn't use patents in a bad way that the author hints at, as far as we know and as far as we've seen. I also met with core React Native developers, and they're awesome.
Practical advise: I'd suggest for the author to meet with the React Native core team.
Being fond of .io's I naivly googled my <brandname>.io, and found that park.io owns it - this happened last week. I immediately sent an email to inquire. We considered the price, and then when I came to buy it today, a week after, the price is tripled. This was a fixed price domain, NOT bid.
That's clever price manipulation. Detect when someone wants something, let it sit, and when they're ready - triple the price. Maybe that's a hint for how he made so much money? In any way we'll just do the get<brandname>.io or something like this, as a compromise. Thanks for being a douche, park.io!
And then, magically, this is now on HN :)
However C++14/17 is getting a ton of new features and a fresh effort to push it towards Rust's innovation. I wouldn't be surprised that if this energy continues, we'll see a lot of Rust in C++.
What is completely certain, that 3 years into the future, if I look at this very comment I'm writing now, C++ will still be as common and valuable. Rust - not sure.
I know I sound like a conservative dinosaur now (but trust me, I'm not), but this kind of thought process you get only after 20 years in the field, and I do apply it only for topics that deserve it (systems programming).
It's been a scary thought; I'm dreading to realize all the know-how's I once had known, that I've managed to forget. So in the meanwhile I've been looking at starter-projects, big C++ projects (somehow, there aren't small, simple ones) to understand how modern C++ projects should be _structured_ these days.
To those looking at this negatively I say - this projects is great for what I've been looking for. You can take it and build a differently flavored server from it, and still have modern tooling (CMake, GTest). It is simple enough.
The domain dictated that we needed a 'heavy lifting' framework like .NET or the JVM.
We did also work with Ruby, nodejs, and Go for certain microservices.
So now, decision making time:
- Do you reimplement in .NET ? - Do you live with both .NET and JVM bases? - Do you migrate existing systems from .NET to Java?
The .NET system was aging in any case, people needed a fresh breath of air, and it was nice to get rid of all of the Microsoft licensing and closed garden. It is then when I decided the new-gen stack would be Java8 and Kotlin, side-by-side. Kotlin was (and still is) amazing.
What subset of Scala? what libraries? what build system? apparently if things worked well, there would be at just one true - and well respected - answer for these.
So now I ask, honestly, why not go with Kotlin (https://kotlinlang.org/)? a language that IMHO didn't try to rule the world from day one (a Good Thing), but scores high on aesthetics, abstraction, power, IDE, compilation speed, and Java integration.
I have been following (and playing with) Scala since a bit before the Yammer thing exploded, I've been following Odersky as a student listening to the OOPSLA podcasts, and I never got to convince myself Scala is worth it. Can someone convince me to still use Scala when Kotlin is around?
A few of the motivations:
* A lisp with an amazing startup time
* A lisp that can tap into a big and vivid ecosystem, like Clojure is tapping into Java's and its own
* Cross compilation and self-contained binary (I've personally tried doing this with various lisps, and none of them really support it)
* Go as a platform is improving version by version and is very performant already
Another way to think of it - if only Clojure compiled to binary directly, then this would be pointless. Because chances of that happening are virtually zero, is why this project made my day.
Go is the assembly language of cloud infrastructure, just as Javascript is becoming yet another "assembly" language. I believe Go is perfect for this, because it is also simple (as opposed to ES6, for example).
Any chance you can share how you got to such polish? the workflow of modeling the animations and UX? How you concluded SVG would be best? tooling you used?
Amazing!
:)
And now for feedback: the one thing that I don't miss is the Scala fanboys hateisms, such as that one. To improve as a platform, as a community, you must be willing to be open to opinions of others rather than mindlessly tag and bag honest opinions into stereotypes. When this comes into the Scala world, I'll give it another try.
Every year I evaluate Scala for adoption. I'm a pragmatist, a polyglot, and usually never say no to seeing how things can improve my execution; which is why I chose to leave the door open for Scala.
And every year it doesn't make the cut for me (various reasons, probably well-known - complexity, compiler, tooling).
This is happening for the last, oh, 4-5 years now. Each year, I take away two weeks from my free time to try and force myself into it.
This year I made a special effort knowing that Spark and big companies adopted Scala. And it lucratively looks like data science is adopting Scala (or does it?).
Knowing LinkedIn is saying "enough" is a real breather for me. At this point, I can stop. Hopefully it will trigger a chain reaction, and leave Scala for the academics.
Middleman has impressive workflows and markdown processing (I'm guessing parallel to that of the Github/Jekyll solution or better). Also conrefs can be implemented by simple partials (which makes less contention for the probably huge conref file)
Though I have to be convinced by trying the Github/Jekyll stack, this does open my mind regarding Jekyll 2.0. I'm happy to see Github tell us their Jekyll story :)
Unix pipes to test your solution (and you can keep piping other tools as well as pipe Crunch processors over and over again)
Eventhough streaming is 30% slower, Go's speed makes up for that and gives me up to 6x speed than the JRuby based solution.
The Ruby/JRuby solution has its place though, you have tens of thousands of valid ruby gems to use and it makes a very rapid development experience.