https://www.usenix.org/system/files/conference/hotos15/hotos...
I’m a C++ veteran btw and understand the point but big data is about how to process petabytes of I/O not how to consume CPU.
> I’m a C++ veteran btw and understand the point but big data is about how to process petabytes of I/O not how to consume CPU.
I'm not sure this is cut-and-dry. Back when I was working on Spark workloads, there was some interesting research being done on where the bottlenecks were for jobs. I think it turned out for a lot of jobs, infinite disk / network io didn't give as much of an improvement as you'd expect.
https://databricks.com/session/making-sense-of-spark-perform...
These are not big data problems, however, by definition.
There's an incredible number of gigabyte and even terabyte scale problems that are consuming racks of blades when a little thinking and understanding of the problem being solved can be done pretty nicely on far fewer resources.
What's really happening is that to many people they think it's "easier" to simply rack more equipment into the cluster and end up shifting the complexity into cluster administration rather than programmer time.
Sometimes, that is the right thing to do. The problem is when the cluster solution also adds to the programmer-time complexity.
In my experience these problems are definitely not always toy problems while some of them can, easily, be ran on 1 server for the expected lifespan of the company because the company will never get that much users/clients/data even though immensely profitable.
Not everyone (almost no one) is a FAANG and I find it offensive to use the company money (which can be investor money) to realise the wet dreams of the cto while the case for that architecture and the cost of it, makes 0 sense to the business and bottomline.
Obviously in life, things are nuanced and differ case by case but nah, real world problems are, generally, not more complex. They are more complex in some (rare) cases, like the one you named. But most companies are not doing anything like that, but they do have their cto’s gearing up for that incredibly unlikely future.
1) We always wait for database, so performance of our code does not matter. This comes from places where ironically the whole database can fit into RAM. 10+ gbps connectivity - well it is almost commodity for business so the latency is not much of a bottleneck. Fast IO to store data - well imagine array of Optane drives. Not very cheap but really peanuts for normal business. All this means that purposely built data server which is a heart of many type of business residing on decent computer, can be blindingly fast and most of the businesses will never ever outgrow it. I've written quite a few of such servers serving loads of businesses in NA so hopefully my experience is not irrelevant.
2) Scripting languages are so convenient to use and save so much time and this is what matters as the salary is a main expense. - Personally I use those mostly for management/deployment/etc.etc type of scripts and maybe to quickly test some small ideas. I clearly see their benefits there. Anything that resembles a product that would actually run business. Sorry but experienced developer can implement those just as fast as in any scripting language and it will save a ton on maintenance.
3) And finally do not do premature optimization mantra . - In my opinion switching from scripting language to something like C++ has nothing to do with premature optimization. Developer can be very productive with compiled languages as they also have megatons of libraries for any imaginable task.
This is just a no true Scotsman argument. For 10 years I’ve watched python/ruby shops drastically outpace projects in C++/Java shops. What you’re failing to realize is how trivial most apps are and how fast fully functional back ends can be created with frameworks in those languages (django/rails).
Your bit about maintenance is also bullshit. I’ve spent entire days peeling apart complex C++ code bases to make a small change to some core abstraction. Ability to maintain code is entirely up to how well organized and documented it is. It has nothing to do with “scripting”.
A vast majority of web dev is incredibly simple. People tend to easily convince themselves that their app is very special and requires a lot of complex solutions to make it work. There’s a lot of incentives that lead people to that conclusion, but most of the time following the golden path laid out for you by an existing framework is going to produce a better product with much less effort.
Sorry but I have the exact opposite experience. First of all I would not call line of business applications incredibly simple. They're rather quite complex business rule wise. I saw floors filled with web developers constantly writing /rewriting endless stream of scripts often without any meaningful attempts to organize the code. In one example I was writing Python (yes guilty it is good for this type of tasks) scripts to process literally thousands of files of their source code to find and properly replace database access methods of which single app had not less then 5.
Ability to maintain code is entirely up to how well organized and documented it is. It has nothing to do with “scripting”
This I can 100% agree with
My guess, essentially holding their finger to the wind. Most orgs can't afford to do real engineering.
We were going to go all in on Snowflake with everyone on the team being for it. I sat down, read the original whitepaper, wrote a simulation of what the costs would look like with the current read/write statistics and tested a small batch of data on it to double check.
Turns out we would have paid between x500 to x10,000 what we were currently paying for using Postgres. I moved on three months ago, and last I heard they were trying to use Snowflake again.
Or perhaps not only there is market, but that's more or less what everyone is already doing.
I could manage my own server and fine tune everything, or I can throw it on snowflake. Snowflake means I've spent almost no time managing anything and it was costing less than an aws box running postgres but absolutely blew it out of the water performance wise. Depends on your workload but it's been perfect for one of my use cases. If they were just using postgres under the hood and I got the same en experience - fine.
Big things are time sharing resources and management/updates/etc. Can you charge me 10x the underlying cost but let me pay for two minutes on a wildly powerful machine? Great, that's a net win for me.