jbmsf

I am fairly ignorant if this space. Do any of the standard methods use multiple hash functions vs just one?

When I set up a product deployment system, I want:

1. IaC for the network, database, container definitions, and other cloud things. I use terraform here. 2. CI/CD for the application software, which ultimately builds a container image from version control, tags it in some well-know way, and tells the container scheduler to redeploy. I tend to use CircleCI, but any CI solution will do. 3. Configuration management for my container definitions, injecting environment variables for config/secrets into the definition or runtime. I tend to use Ansible or Terraform here.

For me, the main criteria is whether the product has a large number of components with configuration that needs to reuse common structures. If I'm just deploying one or two components (e.g. your typical early-stage monolith), Terraform is fine. Bt if you have many users contributing to many components and want to provide higher-level tools, I vastly prefer using Ansible; you just have more control.

That is, I believe in Ansible as a tool for applying configuration from source control to some other system; I would no longer use it to provision cloud resources.

I think it's economy of scale, not kool-aid.

I can run PostgreSQL myself but there's a ton that goes into running it with redundancy, backup, encryption, etc that takes deep expertise to well. I know from experience that it's easy to get database failover wrong. I could probably cobble something together, but it would be mediocre and wouldn't handle network partitions reliably. On the other hand, RDS is used at a scale well beyond what I could afford to build out and has benefitted from much more usage. Problems that are low probability for me are regular events for RDS and have that experience built in.

Ultimately, you are paying a cloud vendor for service value and operational experience. Some services aren't as good as others in both regards, but for the ones that are good, the overhead of doing it yourself is an exponent, not a fraction.

Strong agree.

Plus I've seen more than a few efforts at multi-cloud that resulted in a strong dependency on all clouds vs the ability to switch between them. So not only do you not get to use cloud-specific services, you don't really get any benefit in terms of decoupling.

I take your point, but I still don't quite agree.

There are obviously plenty of companies that are willing to couple themselves to a single cloud vendor (e.g. Netflix with AWS) and plenty of business continuity risks that companies don't find cost effective to pursue. Has anyone been as vocal about decoupling from CRM or ERB systems as they are with cloud?

My own view is that these kinds of infrastructure projects create as many risks as the solve and happen at least as much because engineers like to solve these kinds of problems than for any other reason.

I can't agree, at least not in general.

The optionality of being cloud agnostic comes with a huge cost, both because of all the pieces you have to build+operate and because of the functionality you have to exclude from your systems.

I am sure there are scales where you either have such a large engineering budget that you can ignore these costs or where decreasing your cloud spend is the only way to scale your business. But for the average company, I can't see how spending so much on infrastructure (and future optionality) pays off, especially when you could spend on product or marketing or anything else that has a more direct impact on your success.

It's always a trade-off though. You say you write most of your own software, but that's probably not true for, say your OS or programming language, or editors, or a million other things. Cloud software is the same; you might not be producing the most value if you spend your engineering hours (re)creating something you could buy.

In my own experience:

- AWS SNS and SQS are rock solid and provide excellent foundations for distributed systems. I know I would struggle to create the same level of reliability if I wrote my own publish-subscribe primitives and I've played enough with some of the open source alternatives to know they require operational costs that I don't want to pay.

- I use EC2 some of the time (e.g. when I need GPUs), but I prefer to use containers because they offer a superior solution for reproducible installation. I tend to use ECS because I don't want to take on the complexity of K8S and it offers me enough to have reliable, load-balanced services. ECS with Fargate is a great building block for many, run-of-the-mill services (e.g. no GPU, not crazy resource usages).

- Lambda is incredibly useful as glue between systems. I use Lambda to connect S3, SES, CloudWatch, and SQS to application code. I've also gone without Lambda on the SQS side and written my framework layers to dispatch messages to application code. This has advantages (e.g. finer-grain backoff control) but isn't worth it for smaller projects.

- Secrets manager is a nice foundational component. There are alternatives out there, but it integrates so well with ECS that I rarely consider them.

- RDS is terrific. In a past life, I spent time writing database failover logic and it was way too hard to get right consistently. I love not having to think about it. Plus encryption, backup, and monitoring are all batteries included.

- VPC networking is essential. I've seen too many setups that just use the default VPC and run an EC2 instance on a public IP. The horror.

- I've recently started to appreciate the value of Step Functions. When I write distributed systems, I tend to end up with a number of discrete components that each handle one part of a problem domain. This works, but creates understandability problems. I don't love writing Step Functions using a JSON grammar that isn't easy to test locally, but I find that the visibility they offer in terms of tracing a workflow is very nice.

- CloudFront isn't the best CDN, but it is often good enough. I tend to use it for frontend application hosting (along with S3, Route53, and ACM).

- CloudWatch is hard to avoid, though I rather dislike it. CloudWatch rules are useful for implementing cron-like triggers and detecting events in AWS systems, for example knowing whether EC2 failed to provision spot capacity.

- I have mixed feeling about DynamoDB as well. It offers a nice set of primitives and is often easier to starting use for small projects than something like RDS, but I rarely operate at the scales where it's a better solution than something like RDS PostgreSQL with all the terrific libraries and frameworks that work with it.

- At some scale, you want to segregate AWS resources across different accounts, usually with SSO and some level of automated provisioning. You can't escape IAM here and Control Tower is a pretty nice solution element as well.

I'm not sure if I'm up to 20 services yet, but it's probably close enough to answer your question. There are better and worse services out there, but you can get a lot of business value by making the right trade-offs, both because you get something that would be hard to build with the same level of reliability and security and because you can spend your time writing software that speaks more directly to product needs.

As for "having a fantastic time", YMMV. I am a huge fan of Terraform and tend to enjoy developing at that level. The solutions I've built provide building blocks for development teams who mostly don't have to think about the services.

Coming from a software generalist perspective, I found this phrase very familiar: "experienced hands, high coordination, and focused research objectives—are great for answering questions but not for asking them".

The best engineering teams are excellent at execution, but you need something more to choose the right problems.

Because context matters, everyone is going to have different opinions on some (or all) parts of this one.

From my experience, I've run into a few challenges with some of the claims

For 1: If engineers always operate their code, you will end up with an organization structure built around "these developers work on (and operate) this code." That might be what you want, but it's often not what I want. There are going to be some part of the code base that don't change very much and I'd rather not have a specific team perpetually taxed with operating it (instead of building new business value). Likewise, there are some engineers who are much more productive at building new things (and not just in the the crappy, "now it's your problem way"). I don't want those engineers to have a larger share of operations because they are more productive at development. Support-focused roles have value.

For 5: I have never seen a maintainable automated test suite. I'm sure they exist somewhere, just not in my experience, and I've never worked with anyone who knew how to ensure that developers create one. This means that, with purely-automated testing, your development costs can become dominated by "fixing the tests." My preference is to automate the most important things and to have have humans, with judgement, test behaviors based on test cases written in English.

I no longer have to implement Kahn's algorithm twice a year!

This is great! Thank you.

At my last company, the system that eventually became our Knowledge Graph started out as RDF. It seemed like a great fit, but we didn't really have much luck. There was an unnecessarily complicated process for schema definition and the SPARQL engine we chose (on top of PostgreSQL) wasn't ready for a production load (e.g. was not performant).

I really liked the concept and we borrowed the "triple" approach in our next iterations, but we really didn't get any benefit from using RDF.

Quite possible it was our fault.

Perfect. Thank you for the reference.

That's not really helpful. Let's assume you have a distributed system; you probably don't want all of your system components connecting directly to TimescaleDB. You also probably want to have some layer that implements queuing and handles back pressure if it can't insert into the database at the rate that events are coming in. You may want to batch insert data.

I'd assume that most anyone building a system like this at scale has to solve these problems; does everyone roll their own?

I'd love to hear more about how your data ingestion works. I'm thinking of implementing TimescaleDB myself, but in my initial read of the docs, the focus seemed to be managing the database, not getting data into the database...

One of the reasons duplication is used badly is that it is one of the easiest abstractions to recognize.

One of the ways I've seen DRY go horribly wrong involves reusable code units evolving into shared dependencies that often interdepend in complex ways. Unfortunately, the problems of such a system are observed much later than the original code duplication and fewer people have the experience to see it coming.

Always happy to learn about more good ways to use PostgreSQL!

You know you can prefix your ddg search with !g to use Google?

Useful for technical searches, where ddg is sadly, not good enough.

As always, it depends.

I've seen staged releases work quite well at multiple companies and fail miserably at some others.

That said, continuous deployment is the right choice in many/most contexts. I'd just prefer that people see it as a choice instead of assuming that it's the only way that can work.

I think you make some assumptions that are context-specific.

The decision to always be backwards compatible is just that, a decision. It comes with some really nice benefits around CI/CD, but also adds overhead for certain kinds of development, especially if data models are evolving rapidly. I've been in situations where we made the explicit decision not be backwards compatible due to business context that prioritized these evolutions.

Likewise, deploying to production continuously is not always an option. I've been in situations where customer relationships (contractual or otherwise) require that changes not be made continuously. Accidental outages or regressions are simply not an acceptable risk in some business contexts.

Unless I've missed something, it's worse than that: even if you do verify the token, it only lasts for ten minutes, so you are obligated to tie it to some other form of auth unless you want your users to re-authenticate every ten minutes.

However many developers can correctly follow the meager instructions to validate a token, the number who will properly implement a full auth system is even smaller...

Location: San Francisco, CA Remote: Yes Willing to relocate: No Technologies: See CV. (Python, Node, AWS) Résumé/CV: https://cv.jessemyers.com/ Email: jesse (dot) myers (at) gmail (dot) com

I've been developing professionally for 19+ years, primarily at small-to-medium sized startups. I switched gears this year to find better life-work balance, providing part-time consulting to several companies. I'm ready to switch back to working on a single thing, preferably still part time. I prioritize good people and flexibility over most other things.

There's a brief mention of the load-balancer (nginx) in front of the Go servers; I'm curious if there's anything interesting happening there. I'd imagine that if you lose a server, all of the clients will try to reconnect and traffic will be spread across the existing servers. That's all find and good, but presumably when you bring up a new server to replace the failed on, it'll be seriously underutilized. Is there some easy solution here in nginx-land?

I've used Alembic extensively for Python projects and sequelize somewhat less so for Node projects. It's not a huge sample space, but Alembic's design is much better IMO because it can detect and manage conflicts across merges.

I like the idea, but it feels a bit heavy handed outside of a very large team.

I think the first step here is to get away from the assumption that importing a module will have "interesting" side effects. This is not only a problem with Python...

I tend to create mini "dependency injection" frameworks that create a pattern for loading module code at some point well after import. This patterns tends to reduce to wrapping whatever code you have in the module in a function/closure instead of just running whenever.

Again, I like the idea of enforcing constraints with code, but I don't think it's a substitute for educating developers to avoid certain patterns and giving them infrastructure that makes the alternative easy.

Enough that I don't technically have to work, not so much that I won't keep working.

We had a bizarre experience. Our acquisition was motivated by the value of our contracts and came with a number of earn out goals to ensure that management kept these going for a year. All of the money for the acquisition was held in escrow for this time period, with payouts at the end of the year.

Our executive management was very savvy and negotiated that the acquiring company couldn't interfere with our operations for the first year in any way that could threaten our meeting the earn out goals.

The net effect was just weird. In many ways, nothing really changed for a year. In other ways, everything kind of ground to a halt because there was no motivation to do anything and no new management to steer.

I left at the end of the year.

Aren't you using a technology to solve a social problem?

I don't have the experience with Kubernetes to have an informed opinion, but I've solved plenty of social problems in software organizations. Those are usually approachable without making a long-term commitment to a technology that may or may not be the right choice.

I'm definitely guilty of using and promoting anemic models. I find that teams have a hard time deciding whether logic is service or domain or whatever and gravitate towards putting everything in one place if domain objects are allowed to be at all smart. On the other hand, people do well with the rule that logic belongs in procedural layers with clear names and some sort of maximum size/complexity.

I also find that these discussions matter less in microservices because the size of each service is so small that you usually don't need more than two layers: one for business logic and one for persistence.

I very much agree that microservices create their own challenges at service interfaces, but since these boundaries are usually expressed as http operations instead of object oriented function calls, I look for different solutions, especially various forms of system-wide introspection and testing.

How many companies put thought and effort into defining how they want communication to work?

I've seen Slack work when there is both intention to how channels are organized and leadership nudging people to follow these intensions.