I think it's a bit more nuanced than that https://vondra.me/posts/are-we-reverting-patches-because-of-...
1,651 karma · joined July 8, 2016
So I'm very much advocating for people to "[u]se whatever tools work best".
(That is - now I'm doing this. In the past I taught a course on pandas data analytics and spoke at a few PyData conferences and meetups, partly about dataframes and how useful they are. So I'm very much guilty of what all of the above.)
So I'm definitely a fan, IF you need the DataFrame API. My point was that most people don't need it and it's oftentimes standing in the way. That's all.
Yes, I did write tests and no, I did not write 1000-line SQL (or any SQL for that matter). But I could see analysts struggle and I could see other people in other orgs just firing off simple SQL queries that did the same as non-portable Python mess that we had to keep alive. (Not to mention the far superior performance of database queries.)
But I knew how this all came to be - a manager wanted to pad their resume with some big data acronyms and as a result, we spent way too much time and money migrating to an architecture, that made everyone worse off.
Sure, Python code is more testable and composable (and I do love that). Have I seen _any_ analysts write tests or compose their queries? I'm not saying these people don't exist, but I have yet to bump into any.
Give an analyst AWS Athena, DuckDB, Snowflake, whatever, and they won't have to worry about looking up what m6.xlarge is and how it's different from c6g.large.
Some migrations are easier than others.
Some years I built a data grepping tool, but didn't want to build a UI around it, so I started looking into the Postgres protocol, so that I could plug in existing tools. Getting the basics right was not that difficult, the docs are pretty clear. https://www.postgresql.org/docs/current/protocol.html
Sure, get inspired, make something similar here or there, but this is an outright copy of the whole product design.
(I speak from experience. I've seen a lot of credentials leak together with source code. This meant I saw "secret" paths where Adminer was hosted on a given site. Having database credentials meant I didn't have to somehow circumvent network security to get to the database itself. [Yes, I disclosed all of this to the server operator and even got a measly $50 gift card out of it :-)])
I imagine 99% of installations (and certainly CI pipelines) would be fine with their wsgi webserver and sqlite instead of Clickhouse, Relay, memcached, nginx, Postgres, Kafka and a host of other services. I wanted to take a stab at this myself, but given the complexity of the system, uncertainty of being able to merge it back, and, last but not least, their license, I decided against it.
- Do NOT migrate to new tools, even if they are really shiny and popular - this may end up being a big hurdle you'll regret down the line (looking at you, Notion, Todoist, Google Keep, the list goes on). Just stick to your tools for at least a couple of years before exploring alternatives.
- Try search instead of organisation. I used to have elaborate folder structures for everything... but have since given up, still keep some basic structure, but I find most things via search anyway.
- Help your non-technical friends and family. You may have discovered the best clouds, NAS solutions, software, methodology... but they still use post its for passwords and keep family recipes in that one notebook in the cupboard. Guide them, share your subscriptions with them, give them pointers.
- Use fewer services, even if they are not the best tools for the job. I find it much easier to reason about a handful of storage media (and their backups, pricing etc.) than having an app for every single activity in my life (e.g. I stopped using apps for recipes, I use my synchronised note taking app instead. Same for shopping lists, use my todo app. Etc.)It was a pretty safe guess, though, the number of bugs related to this is pretty epic.
But the analysis only uses the latter subset. So in theory there could be a valid 5-letter that could be a good opener, even if it cannot be a winning word.
While they can crunch large datasets, they are laughably slow for the datasets most people have. So while I did propose we use these solutions for our big-ish data projects, management kept pushing for us to migrate our tiny datasets (tens of gigabytes or smaller) and the perf expectedly tanked compared to our other solutions (Postgres, Redshift, pandas etc.), never mind the immense costs to migrate everything and train everyone up.
Yes, these are very good products. But PLEASE, for the love of god, don't migrate to them unless you know you need them (and by 'need' I don't mean pimping your resume).
Has anyone run into the same issue? I'm considering reopening this investigation (even though I'm very happy with Snappy).
Unless the attitude becomes "all facts in infoboxes and most tables come from WD", the two "datasets" will continue diverging. That is obviously more easily said than done, because relying on WD makes Wikipedia contribution a lot more difficult... and that pretty much defeats its purpose.
It’s sad, really, I’d love to contribute more, but the whole data model is so clunky to work with.
I haven't dug into the whole v3/v4 modularisation, so that might answer it, but is there a way to minimise the dependencies that Plot brings? You say it needs D3, but what of it does it need specifically, the whole thing? It's just that it's 250K or so, so I was wondering what the minimal setup here is.
Cheers and congrats on the release!
Good times.
Yes, it's very good if you don't like setting up clusters (few do/can) and the UI is rather useful for getting up and running (not so much for writing code though). But you need to really understand the platform before adopting it. Please, please don't just adopt it because it's popular.
Does a CLA solve this? (Genuine question, I don't know.)