Another positive case of using ChatGPT! It is not good at math but it's good at the dirty work that you need before you start calculating. That's how you use it to make life easier!
"Our old data warehouse consisted of the most popular components of the time, including Apache Hive, MySQL, Elasticsearch, and PostgreSQL. They support the data computing and data storage layers of our data warehouse."
"As you can imagine, a long and complicated data pipeline is high maintenance and detrimental to development efficiency."
Maybe a little off-topic here, but if they use Twitter and Reddit as datasets to train their LLMs, how can they guarantee that the AI models will be unbiased. Prejudice, hatred, and ill intentions can all be very delicate to identify.
According to the post, the whole system costs half a million pounds, but perhaps it forgot to mention how long the system can last. If it can be used long enough, that it wouldn't matter how expensive it is to build, it will be the most environmentally friendly product in the world, like subways.
Apache Doris claims to be over 10 times faster than ClickHouse in keyword searches. The link in the tweet points to a technical writeup that explains the work they done to achieve that. Inverted index was included in ClickHouse as a new feature a few months ago.
Kisses goodbye to linear algebra way years ago. I think if I knew how it is applied in real life, I would be way more into it. Always respect people who can teach difficult or abstract stuffs in a fun way.
Same way for exams, too, especially when essay writing is included. Good ideas comes in a flash, you don't know how, you don't know when. Most of the time, if you can come up with a great solution and can express it in a clear and structured way the moment you hear the problem, that just means you're familiar with the topic or it is something you've solved before.
"The best component for you is the one that suits you most. In our case, we don't have to too much data to process but simply want a data platform easy to use and maintain."
"One prominent advantage of Lambda architecture is the stability it provides. However, in our case, the processing of real-time data and offline data sometimes intertwines. Such interaction becomes a root cause of instability. Thus, instead of pooling real-time and offline data after they are fully ready for queries, we use an OLAP engine to share part of the pre-query computation burden and make things faster, simpler, and more cost-effective."
New features of this analytic database ready for early adopters to try and test:
1. Semi-structured data storage
2. High-concurrency and low-latency point queries
3. Vertical compaction
4. Separation of hot and cold data
5. Pipeline execution engine tailored to multi-core CPUs
6. Nereids - a brand-new planner
This is a good viewpoint. You actually make me feel relieved. I used to worry about human jobs being taken by AI. Now that I think of it as something like cars, I can see human life will be much more convenient as long as we go through that transition.