I once built a toy document store using SQLite and Python using almost similar idea, https://maxpert.tumblr.com/post/47494540287/a-document-store... if done correctly the advantage of that approach IMHO is:
- ACID (Powered by SQLite)
- Complex and efficient Index (Powered by SQLite)
- CouchDB like API
I've been playing around with Rust recently maybe I will do a simple implementation in Rust-Lang which will keep it memory safe and efficient.
I have read so much about CockroachDB but I am still waiting for that one hero app that puts it on the map. The hard part to convince your company or somebody to put his serious stuff like on something CDB is the fear folks have that what if this company/technology is gone (hint hint RethinkDB). I believe even with all the cool stuff and the hype there are gonna be some paper cut scenarios. Now these cuts might be fatal for someone but for others it won’t be a big deal.
I believe if Apple can replace the stupid touch-bar with a row of e-ink keys. That will solve a lot of my problems. I myself have been imagining such a keyboard for longest time; happy to see somebody is thinking in same direction. This can be awesome thing for POS, and domain specific software getting rid of those finger twisting combinations.
I personally believe these businesses have lagged seriously in upgrading themselves to what was coming for them. They would have paid a certain percentage to cooks, and staff from the prices they were charging, definitely lower than they deserved (hence tipping). Nobody ever complained about them not paying enough and not buying insurance for them. Now once these delivery companies have forced their hands into paying commissions (BTW same items are expensive 10-30% on Uber/DoorDash and people still order) they have suddenly came to a rude awakening. Not that I don’t sympathize with small business owners going through trouble, but this sound hypocritical when Amazon in retail space has done exact thing and nobody cares, but over here everyone sees opportunity for political point scoring.
Threaded IO, Server assisted client caching, and Redis proxy in my opinion is an important milestones. Threaded IO will fully utilize multiple cores; and improvements are amazing. Server assisted caching will reduce client round-trips. Redis proxy will remove the debate of which client library to use, Jedis for example has not allowed reading from replicas or doing multiplexing distributed commands; proxy is gonna solve all those gripes. I can't wait for it to be stable and production ready.
So I have not used Java in production until recently. I was working with languages like C#, Go, Python, Ruby, etc. and built significantly large services in those (10 - 50 MAU). Coming back to Java 8 I was surprised that language made little to no progress, and was significantly painful to use with modern async eco-systems. I still had to review IDE vomit, and people were happy doing so. This has changed recently but records/sealed types is one such example of something that should have been done way back in time. Unfortunately that has been not the case; project loom is still exploration phases. I wonder how many people are bothered by this and moved on to different platform/language due to these shortcomings.
I’ve used LZ4 and Snappy in production for compressing cache/mq payloads. This is on a service serving billions of clicks in a day. So far very happy with the results, I know zstd requires more CPU than LZ4 or snappy on average but has someone used it under heavy traffic loads on web services. I am really interested trying it out but at the same time held back by “don’t fix it if it ain’t broken”.
So some time ago when I was playing around with my toy project (RaspChat) I noticed creating 2 channels and a go routine for every incoming websocket connection is not the answer. I was designing RaspChat to work on a 512MB Raspberry Pi; and I was bottle-necked by GC, and memory consumption around 3 - 4K connections. After loads of optimizations I got it around 5K. Digging deeper and found well I have to maintain a pool of go routines (like threadpool) and I have to write event loop. I was instantly pulling my hair. I was sacrificing so much simplicity, and flexibility of Node.js just because I was trying to avoid event loop and wanted to use channels (I did too much Erlang months before starting project and couldn't think anything other than process and messages). I got a backlash on my release (https://github.com/maxpert/raspchat/releases/tag/v1.0.0-alph...) from go community telling me how I was using desierializers/leaving loop holes in file upload and I didn't know shit about language.
At that time I found uws (https://github.com/uNetworking/uWebSockets.js) that easily got me to 10K easily, and I was like "I would rather bet on a community investing on efficient websocket event loop rather than me writing my own sh*t". Don't get me wrong; I love Golang! Seriously I love it so much I have been pushing my company to use Golang. I just don't want to glorify the language for being silver bullet (which it's fanboys usually do). I would never implement complicated business logic that involves many moving pieces. When my business requires dealing with shape of an object and mixing matching things to pass data around; I would rather choose a language that lets me deal with shapes of object. Go has it's specific use-cases and strengths, people advertising it as move it to go and it would be faster than Java/C#/Node.js etc. have not done it or have not dealt with complexity of maintaining it.
I've been following Crystal for long time (Shameless plug I've been running a geo-location service in production for years now https://gitlab.com/maxpert/crlocator and it works flawlessly). Despite the setbacks in past Crystal has continued to evolve and I am waiting for fully baked version 1. I am not surprised that Crystal can pull of something like running a complete operating system. Modern language landscape is getting populated and brings novel ideas (Rust, Zig, Golang, Julia); and Ruby's nice syntax with efficiency of compiled language IMOH still remains desirable for a lot of us.
This reminds me of similar debate I had with one of my friends falling for NodeJS FOMO. The actual codebase was RoR and he wanted to go for Node for no apparent reason. I was able to talk him out of it but what I understood from discussion was peer pressure of “hey all of this could have been async”. While I admit blocking threads might not be permanent solution, they can still take you pretty far. Not to mention the folks wanted a MERN stack and I believe leaving Postgres for Mongo without a damn good reason is just crazy idea.
Love the fact people are adopting Opensource solutions but corporates won’t budge. I think the hardest penetration for anyone right now is Microsoft Teams. Since they started bundling teams with Office they have been able to boast about numbers and actually hurt Slack. I wonder what is roadmap for Mattermost. Shameless plug I myself did an opensource version of a chat server called raspchat https://github.com/maxpert/raspchat which was able to handle almost 5K very active chatters on a Raspberry Pi Model B (Just 512 MB of RAM). My original dream was to build a drop and run server on cheap raspberry pi for local areas but man has to feed his family and I had to make a tough choice. It’s harsh IMHO out there to get attention and then convince folks to develop integrations for you. I hope this space doesn’t endup with duopoly and products like Mattermost take off.
I keep seeing articles showing up on debugging network issues like DNS or something else. Makes me wonder how much engineering effort we are putting in to fight the tool. Have these people considered/investigated other tools like Nomad (hashi corp)? How much value Kube actually adds vs these issue?
So I've my self used Lombok but I've been using Kotlin lately. IMHO Kotlin if far far superior in terms of how it provides appropriate constructs that are patches to a stale language (hopefully new release cycle fixes it).
So far I've observed with libraries like Lombok there is a smaller community (compared to language community) that vets the idea; and the features are opinionated to that particular community. Which results into fragmentation; just as example https://immutables.github.io/ and https://github.com/google/auto are two libraries providing features like factories and immutable value classes. Then you get articles like https://codeburst.io/lombok-autovalue-and-immutables-or-how-... for improving your code.
This in my opinion doesn't hurt as long as you are a small company. But when your code is going to be used across teams with different opinions, and language doesn't enforce a single way to define construct these kind of byte-code mangling ideas fall apart. I've personally people complain about Lombok and they hate it every bit. Doing these constructs at language level removes debate, and unifies how people in larger community. I hope newer versions of Java move fast enough to make something simple like `@Cleanup` a thing of past. I will highly encourage everyone to try Kotlin too; it's almost transparent drop-in with better syntax.
Who knew Wintel combination will become the curse for Microsoft. Microsoft paid heavy price including a complete OS flop, and losing mobile war; just because developers were not ready to leave Win32 API and step down to WinRT (buggy and half baked). Current emulation for 32-bit is slow and reviews are terrible. I wonder if Intel will ever be able to develop something revolutionary that runs x64 and energy efficient as ARM.
> A queue implemented in the RDBMS will never match the performance of a fast dedicated queueing system, even one that makes the same atomicity and durability guarantees as PostgreSQL. Using SKIP LOCKED is better than existing in-database approaches, but you’ll still go faster using a dedicated and highly optimised external queueing engine.
People only realize these warnings when they have built a toy system that is rolled out to a production/product that takes off. By that time you will realize "ohhh I don't need all the ACID guarantees for every job in my system", and I want to run my workers as lambdas/elastic scaling workers (you need more connections). That is where companies will be spending efforts from their best engineers to move away from Postgres as queue. Which comes down to question; why do it in first place? What makes it cheaper (other than local dev) to deploy a system that is bound to fail in future? Don't get me wrong I love Postgres, I just don't believe that it's the right tool for doing this job.
Aaaaah the memory of making the an efficient self extracting archive. What is brilliant about the idea is that even to this very day many "new ideas" live around same event loop idea.
I think the rich and wealthy valley brains need to solve this problem too. This company is definitely done, they already have caused a fire despite outages.
So this year on RedisConf Antirez demoed threaded version for Redis (only transport needs to be multi-threaded, core remains single threaded). Numbers were already amazing. I will pick the community version of Redis any day over forks.