While annoying technically, for early stage startups, performance problems caused by an overly large number of users are almost always a good problem to have and are a far rarer sight than startups that have over-architected their technical solution without the concomitant proven actual users.
You can get by with one strong lead defining services and interfaces for a bunch of code monkeys that write what goes behind them.
Given an existing monolithic codebase, you can’t specify what high level services should exist and expect juniors to not only make the correct decisions on where the code should land up but also develop a migration plan such that you can move forward with existing functionality and customer data rather than starting from zero.
Could you elaborate? I'm interested in the specific reasons.
If you use a relational database, the worst-case outcome is you hit tremendous scale and have to do something special later on. The likely case scenario is some wins from the many performance lessons databases have learned. Best case outcome is avoiding a very costly excursion relearning lessons the database community has known about since 1970 (like reinventing transactions).
Managing data with a csv (without a great reason) is like programming a complex GUI in assembly without a reason - it isn't going to look like a good decision in hindsight. Most data is not special, and databases are ready for it.
It is great for data safety -- chown/chmod the file, and you can be sure your scripts won't touch this. And if you are accessing live instance, you can be pretty sure that you won't accidentally break it by obtaining a database lock.
Now "csv" in particular is kinda bad because it never got standardized, so if you you have complex data (punctuation, newlines, etc..), it you might not be able to get the same data back using a different software.
So consider some other storage formats -- there are tons. Like TSV (tab-separated-values) if you want it simple; json if you want great tooling support; jsonlines if you want to use json tools with old-school Unix tools as well; protobufs if you like schemas and speed; numpy's npy if you have millions of fixed-width records; and so on...
There is no need to bother with SQL if the app will immediately load every row into memory and work with native objects.
I have experienced pain with both characters (tab and comma), particularly when I am not the one creating the output file.
Commas are _way_ too common.
CSV is an awful format anyway.
You use this tsv with Unix tools like "cut", "paste", and ad-hoc scripts.
There is also "tsv" as defined by excel, which has quoting and stuff. It is basically a dialect of CSV (Python even uses the same module to read it), and has all the disadvantages of CSV. Avoid it.
Oh, the irony! Text with punctuation and newlines is complex data.
CSV is doomed but the world runs on it and pays its cost with engineer tears.
I hear this kind of thought-terminating cliche a lot on here and it makes absolutely no sense.
If # of users is a rough approximate of a company's success and more successful companies tend to hire more engineers ... then actually the majority of engineers would not have the luxury of not needing to think about scalability.
With engineering salaries being what they are, why would you think that "most people" are employed working on systems that only have 1000 users?