101 karma · joined November 8, 2019
- I'm writing this service even though team X has built the same thing, because my team lead doesn't trust team X since the last time we depended on their service 3 years ago, they had a major downtime that screwed us up big time
- This service is using AWS Lambda simply because I think it's cool, despite the fact that the company has a dedicated team running k8s stack with argocd, argo rollouts, KEDA, etc for the entire company
- Service Y is written in this particular way because it's a service that is shared with another team that came from a company that was acquired, and they wouldn't use it unless we write it this particular way, and making the top execs happy is more important than dealing with a small tech debt (this is probably true)
But no one is going to write these in their RFC. But Sarah knows.
One example was an engineer wanted to build an API that accepts large CSV (GBs of credit reports) to extract some data and perform some aggregations. He was in the process of discussing with SREs on the best way to process the huge CSV file without using k8s stateful set, and the solution he was about to build was basically writing to S3 and having a worker asynchronously load and process the CSV in chunks, then finally writing the aggregation to db.
I stepped in and told him he was about to build a data warehouse. :P
- Read access through OLAP, not OLTP. You just need to setup a near real-time replication between your OLTP and OLAP.
- Write access through API, just like your application. You can add fancy things like approval layer, e.g. you agent cannot "ban_user(id)", but it can "request_to_ban_user(id)", and the action only happens once you approve it.
Based on my experience with Claude, it's pretty damn good at doing data analysis, if given the right curated data models. You still need to eyeball the generated SQL to make sure it makes sense.
> and how?
1. Replicate your Postgres into Snowflake/Databricks/ClickHouse/etc, or directly to Iceberg and hook it up to Snowflake/Databricks/ClickHouse/etc.
2. Give your agent read access to query it.
3. Build dimensional models (facts and dimensions tables) from the raw data. You can ask LLM for help here, Claude is pretty good at designing data models in my experience.
4. Start asking your agent questions about your data.
Keep steps 3-4 as a tight feedback loop. Every time your agent hallucinates or struggle to answer your questions, improve the model.
Side note: I'm currently building a platform that does all 3 (though you still need to do 2 yourself), you just need Postgres + 1 command to set it up: https://polynya.dev/
I used to play Nullpomino (an open source tetris game) for hours. It has a mode called marathon, where the goal is to play as long as possible. It's normal to see good players surviving for hours, even with increasing level. Here's an example of someone playing for 1h, 90% at level 20 (at level 20, pieces appear immediately at the bottom): https://www.youtube.com/watch?v=rHXxKywAQME
But from my experience, I'd argue 90% of use cases of Kafka don't need sub-100ms latency. In my previous company (fintech company), pretty much 99% use cases are for asynchronous processing that can tolerate seconds of latency. We just use Kafka because it's there, there's a team that operates it for everyone so the ugly parts are hidden from everyone else, and the cost is pretty much "subsidized" (the cost is attributed to the team that operates it).
This allows the non-engineers (and also engineers) to use Claude Desktop to do day-to-day operations (e.g. ban user X for fraud) and analytics (e.g. how much revenue we made past 7 days? Any fraud patterns?). The MCP helps to add audit, authorization, and approval layer (certain ops action like banning user will require approval).
CS7210 (DC) is the hardest class I took. We had to write a correct Paxos implementation, then use it to build a distributed sharded KV store. I remember having to spend 10+ hours on a single day during weekend. It was worth it though, learned a lot about distributed consensus, and how difficult it is to get right (there were test failures that were fixed after hours of debug, and the fix was literally changing the order of some code lol).