This is definitely an area where Pulasr is trying to improve, getting started is not easy. That said, the progress I have seen in the last year since I have been involved with it is really promising.
883 karma · joined July 23, 2013
[ my public key: https://keybase.io/addisonj; my proof: https://keybase.io/addisonj/sigs/Hqo1zUyFuPvsvswnLL1gCDIjNJ-vE8N29D9dnXumuq4 ]
loginwithhn.com one-time login token: [OqpofBS60H]
email: addisonj at gmail
If you are moving to a more "event-sourced" architecture, usually two main concerns (beyond basic operational stuff of uptime, scale, etc) are routing and long-term retention.
RabbitMQ has the routing but not the retention. Kafka can have the retention and the routing, but it can be complex/expensive. Apache Pulsar really shines here as the API is pub/sub but it is underpinned by a log structure that gives you long-term retention (that doesn't need to be manually re-balanced) but it's flexibility does come with some operation complexity when compared to RabbitMQ.
If your needs is pretty much just moving large amounts of data, Kafka is definitely the most mature and has a big ecosystem, but long term-retention is difficult and there are some sharp edges around consumer groups.
If you really really don't need long-term retention and need complex topologies, RabbitMQ is your best bet and is fairly reasonable to operate even up to fairly high message rates (~10k msgs/sec shouldn't be too hard to achieve)
There are a TON more options these days though, older more java solutions like activeMQ and rocketMQ or more "minimal" implementations like NATs, not to mention the hosted services on cloud providers.
Personally, I am a big fan of Apache Pulsar for it's flexibility and some nice design choices, but I don't think there is any silver bullet in this space.
The tl:dr; appears to be "we built a blockchain based query logger" where essentially the consensus appears to just be that it is valid SQL, and defines an ordering of the statements. Then once everyone agrees to apply the change, then it is applied... With (AFAICT) no interaction with the DB during the consensus process?
This seems to be a bit naive of features like locks and transactions that are critical to most DBs, so while maybe you can define a global ordering of statements, the execution of those statements may be nonsensical and result in a lot of failed transactions. For example, it seems like you could grind the usability of the system to a halt by flooding the validators with 'valid' SQL designed to create as many conflicts as possible that would cause a ton of writes to fail to commit.
I agree with the desire for more services to become decentralized, and this is a novel idea, but (IMHO) more and more the technical hurdles of blockchains often bring more challenges than they solve.
It might be worth chatting with Pulsar devs on their slack community (https://apache-pulsar.herokuapp.com/).
Most commonly what I hear people doing for this is either one of two approaches (or a combination of both): - encrypt the user data and delete the key, eventually the user data will get removed - regularly compact the topic (pulsar has a compaction feature) and write in a tombstone record which will remove any user data after compaction
Bandwidth is certainly a concern and that is one of the nice bits about Pulsar is not everything is replicated. You mark a namespace by adding additional clusters it should replicate to. We don't expect to replicate everything, just the things teams care about.
When we did this, Transit Gateway was just within the same region. At re:invent they announce the cross region transit gateway which we will look at moving to as well, but for now, it is just a full mesh of VPC peers, which for 8 regions isn't bad... but certainly gets worse with each new region we need to add.
For exposing the service into other VPCs in the same region we use private-link endpoints as to avoid needing to do even more peering.
Being that the project is a top level Apache project and also has some adoption by quite a few different companies and a number of corporate sponsors the future of the project is pretty safe bet.
1a. You are correct we didn't require geo-replication for existing use cases, however, initially, we saw geo-replication as an easy way to improve DR and we have an internal requirement for a DR zone in another region. Now that we have done the work, we are starting to see multiple places where we can simplify some things with geo-replication, so we think long term the feature will be really valuable
2. We split up auth into two main components: auth of users (where we use Okta) and auth of services. For okta, we just wrote a small webapp that users can log into via OKta and generate credentials. For apps/services, we already had hashicorp in place and wanted to just piggyback of our existing form of identity (IAM roles). Essentially, a user just associates an IAM role with a pulsar role and we generate and drop off credentials into a per-role unique shared location in vault that any IAM role can access (across multiple AWS accounts)
3. Once again, geo-replication wasn't really a hard requirement initially but more of something that we really like now that we have. I think the biggest reason why not postgres is that we have combined message rates (not everything is migrated yet) on the order of 300k msgs/sec across a few dozen services. Pulsar is designed to scale horizontally and also has really great organizational primitives as well as an ecosystem of tools. While I think you could maybe figure that out with some PG solution, having something purpose built really can pay big dividends for when you are trying to make a solution that can easily integrate into a complex ecosystems of many teams and many different apps/use cases
I am interested to see where NATS goes but for where we are today Pulsar was a much more obvious choice.
- ordering is really hard, you don't get guaranted ordering unless you write one message at a time or do a lot of complexity on writes (see https://brandur.org/kinesis-order) and the shards are simply too small for many of our ordered use cases
- cost, we just don't send some data right now because it would just be too much relative to the utility of the data (we would need like 250 shards)
- retention, long term, we want to store data in Pulsar with up to unlimited retention so we can rebuild views. There is still some complexity there (like getting parallel access to segments in a topic for batch processing) but it is much further along than any other options
- client APIs for consumer. We are a polyglot shop and really the only language where consuming Kinesis isn't terrible is Java (and other jvm languages). For every other language, we use lambda and while lambda is great it is still distinct deploy and management process from the rest of the app. Being able to deploy a simple consumer just as part of the app is really nice
For example, Kinesis is really limiting with the limited retention and making it very difficult to do any real ordering at scale due to the really tiny size of each shard.
Similarly, SQS does pub/sub well, but we keep finding that we do need to use the data more than the first initial delivery. Instead of having multiple systems where we store that data we have one.
As for why we didn't go with Kafka, the biggest single reason is that Pulsar is easier operationally with no needing to re-balance and also with the awesome feature that is tiered storage via offloading that allows us to actually do topics that have unlimited retention. Perhaps more importantly for the adoption though is pub/sub is much easier with Pulsar and the API is just much easier to reason about for developers than all the complexity of consumer groups, etc. There are a ton of other nice things like being able to have topics be so cheap such that we can have hundred of thousands and all of the built-in multi-tenancy features, geo-replication, flexible ACL system, pulsar functions and pulsar IO and many other things that really have us excited about all the capabilities
While there certainly is some aspects you need to be aware of, generally, Pulsar is much more "cloud native" and maps quiet well to k8s primitives.
We have practiced some disaster recovery, but it isn't 100% exhaustive (is it ever?), however it is also aided by how Pulsar is designed. We have killed bookie nodes as well as lost all our state in zookeeper. The first is pretty easily handled by the replication factor of bookkeeper data and for zookeeper we do extra backup step and just dump the state to s3 and can restore it. What we haven't tested in practice but now how to do theoretically is to restore a k8s stateful set from EBS volume snapshots. However, we see that as a real edge case. In Pulsar, we offload our data to s3 after a few hours, so we only need to worry about potentially losing a few hours of data in BK, as the zookeeper state is very easy to just snapshot and restore from s3. In other words, we are still working on getting more and more confident with data and don't yet recommend teams use it for mission critical non-recoverable data, but there are a ton of uses cases for it now and we can continue to improve on the DR front
We have done multiple upgrades and deploy all the time. Because bookkeeper nodes are in a stateful set and we have don't do automated rollouts, we manually have a process to replace the BK nodes. However, they don't trigger a re-balance as it closes gracefully and then re-attaches the EBS volume from the stateful set
We use EBS volumes, we use a piops volumes for the journal and a larger slower volume for the ledger store. THis is one of the great parts of bookkeeper design is that the two disks pools are separate so we just need a small chunk of really fast storage and then the journaled data is copied over to the ledger volume by a background process. We figure for really high write throughput we could use instance storage for the journal volume and EBS for ledger, but that would have some complications on recovery but still easier than having to rebuild the whole ledger data.
We use the pulsar proxy and expose it via a k8s service with the AWS specific NLB annotations.
We haven't had any issues with the k8s plugin and haven't really had any issues with EKS version upgrades. We just add new nodes when we migrate the kubelets
Yes, we have automation (via terraform) to allow us to add many different pools of compute and we use labels and taints to get specific apps mapped to specific pools of compute. For Pulsar, we run all the components multi-AZ
It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to get auth tokens for an AWS IAM role.
Overall, we are very pleased and the rest of the engineering org is very excited about it and planning to migrate most of our SQS and Kinesis apps.
Ask me anything in thread and will try and answer questions. At some point we will do a blog post on our experience.
First things first, Blackboard is some of the worst of enterprise software and a huge part of why they have lost so much market share is not only because they had a difficult and obtuse to use product, it is also because they were extremely aggressive in buying out any competition and EOLing their products and forcing schools against their will onto Blackboard.
And I think this is the bit that is often the case about the worst enterprise software is the it gets to a point where customers are so entrenched that the company behind the product has market forces that drive it to grow through unhealthy means that are often hostile to customers.
To his conclusion about all software tending towards these same problems, I can't agree more. It is a HUGE challenge to continue to deliver software that remains easy and friendly to use in light of the real need to add real features. You simply can't just ignore the market that thinks differently about a given problem and the workflow to solve it.
I think the right answer to this problem is not just really great product and UX (though that is critical), the real challenge is to figure out the underlying technology platform and engineering process that makes it cheap to build new features and compose them together without the complexity becoming overwhelming for your average engineering team. From what I can tell, this sort of architecture looks vastly different from how most software is started and how we think about software in general. Making that transition is one of the hardest things to do in engineering and it is what sets apart companies who continue to deliver well liked software and those companies that tend towards being defined by worst tropes of "enterprise software"
The people who demand change are wrong too, I am super happy people like you are doing what you do, otherwise this OSS thing really would never work.
But to be clear, while I think your stance is completely reasonable and justified, it is just a hard pill for some people to swallow. That doesn't mean that those who are jerks about it are justified, but it also doesn't mean that those who raise concerns, that those concerns aren't warranted.
Yes, Nvidia are not great when it comes to OSS, but like it or not, for doing real work in ML, scientific computing, or video and photo editing, Nvidia with CUDA is simply a much better experience and much more broadly supported.
While I don't think it should be the job of wlroots/sway to fix Nvidia's less than ideal driver chain, I think it is important to remember that a lot of people get a lot of productive work done on with Nvidia cards and would love to get rid of some of the pain of X while still being able to use their high end GPUs for both work and graphics.
It seems fairly clear with with MacOS focusing less and less on high end machines, there is a gap for high end unix workstations and hopefully desktop Linux can fill that gap, but right now that likely means Nvidia support... I think the best answer is just to buy amd and market forces will get Nvidia to change, but that still doesn't mean everyone can just go that path right now to get their day job done.
From the little that was said, going to guess this uses something like Beringei (https://code.fb.com/core-data/beringei-a-high-performance-ti...) under the hood
Tl;Dr: with Oracle changing the release versioning of Java, LTS releases are now essentially a paid product. If you run a large business with a lot of Java that is now a significant new burden to deal with.
I have been writing scala for 2 years, mostly in spark and flink and as much as I love parts of it, overall, the language is just not the most approachable. Some of the features are super valuable every once in a while, but I might gladly give some of them up for a streamlined language that is easier to get people started on (while not just writing imperative style Java with Scala syntax)
My other curiosity is compatibly. If I can compile a subset of my project much faster that doesn't need all of Scala, that seems like an okay place to be in.
For an area that in many ways is punching above its weight class in terms of tech companies, I occasionally have nagging (possibly unfounded?) concerns in the back of my head that one day a lot of these companies will be a hindrance on the continued growth here. I know some people already view Utah mostly as the home of MLMs and I don't know if that does much for attracting forward looking tech investment. That said, these companies have brought a ton of jobs and has certainly helped fund the growth in the area and I have worked with and met a lot of people who seem pretty decent.
While many of these companies aren't "evil", I think the MLM is probably one of the most insidious dark design patterns around. I think the best way of how these companies primarily operate is like the Cassette/CD clubs of a few years past. A small fee gets you a decent amount of "up front value" (Join the club for $30 and get 30% off your first order!) with the expectation that you won't order anything else or forget to cancel the yearly membership fee. This is, of course, all made worse by the fact that is often friends and family who are peddling the high margin health and dietary products with questionable efficacy. Like I said, I don't know if I can call it an outright scam (I know a few people who really like there shakes and supplements and swear by them), but it does drive strange incentives...
Once flink (and other systems) support this, I think it will really be a game changer. It will allow for doing things like sending an increment downstream rather than needing to keep all that state. Instead of mostly seeing stream processing as a thing we do for analytic use cases, it can really become the backbone for streaming applications.
Event Sourcing/CQRS has been an idea for a while, but in practice, is difficult to do because to the inherent difficulty in dealing with message semantics and consistency (see https://data-artisans.com/blog/drivetribe-cqrs-apache-flink for a good write up about such an app). The ability to independently optimize for both reads and writes while also not having to make all messages be idempotent, in my mind, will make this feasible for a broad range of teams and won't require a huge amount of work in thinking up clever message formats or working through as many failure scenarios.
That being said, I fully expect this to bite hard when the caveats aren't understood (such as when interacting with external databases) and there are still other hard problems. Like creating a consistent down stream view of an app in terms of business events rather than hooking into a database transaction log.
Still though, I think these are the sort of solutions needed to make distributed systems easier, even though there are lots of caveats :)
While I think I still might do an LVM for adding capacity, the biggest change here would appear the ability to scale provisioned IOPS. While gp2 disks IOPS are primarily tied to storage capacity, which often leads to overprovisvioning for peak loads, the alternatively high cost of smaller and faster storage with PIOPS often times isn't worth it when peak loads only happen occasionally. That changes with this and I am sure this will make a lot of DBAs happy.
AWS in the last year has really impressed with giving you even more knobs to turn to get exactly what you need. While it's still not quite as flexible as GCEs custom instance sizes, the GPU and now storage elasticity sort of make up for it. While it's still pretty expensive, AWS feels more and more like I can use an API to build a custom spec'ed server instead of trying to shove my workloads into only a handful of options.