Comment by asdkhadsj | Hacker News Reader

Comment by asdkhadsj | Hacker News Reader

Parent Full thread

Out of curiosity, any tips for companies small enough to not warrant an ops position, but also so "lean" (read: broke) that we can't too heavily rely on cloud solutions?

Ie, I advocate self hosting a lot in my ~8dev team (in a real, 10 year old small company). I advocate this, because the last company I was at was the traditional startup and we blew tons of money on cloud solutions. Ie, if we needed to aggregate logs, we threw everything at papertrail, rather than managing it ourselves. This has given me a cost aversion, so I theoretically prefer self hosting when sane. Mostly for everything but actual VMs.

With that said, a lot of self hosting leads to a lot of infra. I agree that hiring devs who also know ops is a good way to go (and would fit us well), but in this scenario, do you still think it's the better/best?

edit: to be clear, I advocate using AWS/etc. I don't advocate using dozens of SaaS options to solve our modest needs at non-modest prices. Many SaaS solutions were designed for scales much larger than ours, self hosting seems most apt.

Man, so many red-flags in what you posted.

So, full-disclosure: I come from Ops. These days I'm more an orchestrator/developer than a hardware/bios/cable monkey though. For context my last deploy was 2,500~ physical machines. So I'm not the kind of ops that the parent is talking about.

Now, if you don't have the money to do cloud, and you don't have the headcount/experience to do self-hosted properly; then you're _done_. You will accrue technical debt at an exponential rate. At my last job Ops on the production environment was done exclusively by developers.

When I arrived it was an absolute shit-show nightmare. Silent data corruption, weird dependencies on services which were not highly available, hand-built machines, next to "cattle" but which had multiple deployment scenarios (capistrano, chef, bash, PHP kludge) none of which were documented.

But the worst thing was definitely the silent data corruption.

I agree in part with the parent, Ops is a dirty word. In my current company there's a separate "Ops" entity which is a true horror show exactly as he describes. However having people trained on ops in your team is -absolutely essential- to doing devops correctly. Developers and Operations think differently. It's a gross generalisation and there are exceptions but by-in-large developers are focused on "how can I make this work" (even robustly), where operations are focused on "how can this break and what are the implications if it does" and those methodologies are intrinsically opposite to each other. A defensive programmer is a great thing but they are not only rare they are not a panacea.

--

Pick a side; do cloud or hire an ops.

Cloud is basically "ops as a service", giving you all the tools you need to get by efficiently without classically trained sysadmins. I really promise that you will not get very far without someone handling this side of things for you, whether it be the SRE's at google/AWS or someone you hire.

I added an edit, do you still feel this comment is accurate after my edit? Ie, I'm not advocating physical machines. The cost + time ratios don't add up imo. Using AWS/etc is simply better, I believe.

I'm talking about not using dozens of SaaS options for small tooling, think CI, log agg, metrics, git, etc. Much of these are trivial to self host when your traffic is small and predictable. Thoughts?

(Not the OP) If cloud equates in your mind to expensive, then you were probably using the cloud wrong. Using AWS (for example) as a data-center and self hosting on EC2 can be pretty expensive compared to just getting a few dedicated boxes. The point of cloud architecture with tools like terraform are meant to be able to have resources be torn down and scaled down when they are unnecessary. You can't scale down bare metal, and you can't scale it up either automatically. If you have a consistent workload and know exactly what you need on any given day/time and for any given number of clients and can predict what you need - bare metal is honestly not a bad choice.

Edit: Just realized I didn't answer your initial question in the post: A cloud architect, who comes from an engineering background, is more along the lines of what you would need versus just a pure ops person. My last company was a small company (startup mentality) and as chief architect I was able to design our cloud architecture as well as write code on our projects when my developers were struggling to meet deadlines.

So what the 8 devs really need is a cloud architect?

I think there's a miscommunication here, but I probably should have said SaaS rather than "cloud".

Ie, I don't advocate self managing hardware. That's a difficult problem, one more cheaply and easily solved by using AWS, GC, etc. Yet, there are dozens of SaaS options for so many little problem you could have, and they can bleed you dry. A balance is needed.

I mentioned in my original post that I advocated self hosting except for VMs. Ie, I promote usage of AWS/etc. I just don't think we can afford Papertrail, and the billion other SaaS pieces I used in my last company. Self hosting has been far more scalable and cheap, due to our modest needs.

In the most successful of the companies of the sort you've been describing that I've been around SaaS was kept pretty minimal. This was less a policy than a consequence of just having *nix skills or a desire to learn. Also many SaaSes have relatively serviceable self-hosted FOSS counterparts.

Log management in particular is really a shell script/self hosted app kind of task, just share the money saved on Papertrail with the team and I'm sure they'll get on board.

Reply on news.ycombinator.com