Hacker News Reader

Comment by tguvot | Hacker News Reader

those "operational" jobs exists today. just not in the small startups, that still didn't figure out, that despite running in the cloud you need networking and cybersecurity people

View on HN

As a startup CTO, I agree you need networking And cybersecurity. I am not sure if you need fulltime people specializing in this though.

I used to work for large companies with big on-prem footprint. Networking and security in that world is a different game and warrants dedicated people.

But for a startup with two services running in the cloud, with so many out of the box tools? (IDS, WAF, log based monitoring, SDN and all the configurability that comes with it). That can go a long way, without dedicated people.

>As a startup CTO, I agree you need networking And cybersecurity. I am not sure if you need fulltime people specializing in this though.

highly depends on function of the service and it's scale

>I used to work for large companies with big on-prem footprint. Networking and security in that world is a different game and warrants dedicated people.

>But for a startup with two services running in the cloud, with so many out of the box tools? (IDS, WAF, log based monitoring, SDN and all the configurability that comes with it). That can go a long way, without dedicated people.

maybe for network. but in my experience most of engineering (starting with junior developers and ending with vp/cto level) doesn't understand cybersecurity specifically or security in general . so even if there is tooling available, people don't understand when, how or most important why to use them.

That's fair. It's a red flag in general if VP/CTO doesn't have the basics of security in place anyway. My experience in my peer group is that they are all fairly knowledgeable about security, if not experts.

Most startups don't have the scale, there are exceptions of course.

i worked in variety of companies that varied from development of security products and were very security oriented to companies where security were driven by client needs (telecom industry for example), to "other places" where management grew with the company and it's case of "you can't teach old dog new tricks". there are always security departments, but unfortunately most of the time their thinking is "slap crowdstrike everywhere, it will solve all problems" and "here is sdlc that engineering must to follow" (lol)

Indeed, I've seen that latter behavior in large companies back in the days. A security department that refuses approvals to upgrade operating systems because it's too risky, a full-blown ops team that doesn't know how to do it without killing all services for days on, doesn't have recommendations on security patches, doesn't know if a CVE is actually exploitable in that setup or not - the list goes on.

i work now on fedramp certification (essentially leading scoping and solutioning) and interaction with security department is both funny,sad and scary af. i discovered that they developed risk assessment policy for system components (in commercial environment) whose purpose to drop down risk level of components in order to remove need for security patching for SOC. and crowdstrike in monitoring mode (nobody knew that it's in monitoring mode) because they afraid of enforcing mode. and that temporary access from/to production network is actually permanent because there is no flow in ticketing system to remove it .

The longer my career goes on, the less a lot of these cloud services make sense to me. I know why we have AWS deploy our production instances, but why are all of our staging and dev instances running on EC2? Can't we just buy an old PC and put it in the office and run dev builds on that? Sure it might be slow, but it's not meant to handle 100k QPS, it's just for QA

> Can't we just buy an old PC and put it in the office and run dev builds on that

It's cheaper to pay for those cloud resources than hiring 2-3 more people.

I think if you had a whole server rack, you might have to hire someone for that, but if you're just running one or two servers with containers to run your dev builds, you could get that set up on K8s and manage your deployments the way you would with AWS. I don't think it would require hiring new people, an existing devops guy should be able to do that

It's still additional operational overhead for marginal RoI.

If you're an organization that already has a SRE or Infra Eng hired, there's no point supporting a niche setup when every other team is using a prebuilt and troubleshooted environment.

Edit: Can't reply to OP atm, but he makes a good point. For a higher margins and less commodified segment like ML Compute, on-prem computing still makes financial sense.

That said, that's still managed by actual SWEs not a generic Sysadmin, due to intricacies of GPU Architecture and Model Development, and OP is extremely underskilled for this role.

I think if all you're running is web microservices, then it's probably not worth it. But my company builds machine learning models offline and I feel that if we bought a dedicated GPU machine (literally just build one and put it in the office, not a rack mounted thing), we would save money in the long run after building a custom training pipeline

Why would you have your QA environment unlike you prod environment? Even the infra setup should be as close as possible, so you can test infra changes in QA before promoting them to prod.

I can't see how it would make sense to deploy in two completely different environments that are not alike at all.

> I can't see how it would make sense to deploy in two completely different environments that are not alike at all.

Because you are not saving that much money by having multiple bespoke on-prem environments per team.

Troubleshooting, management, and security headaches grow.

It's just easier to have everyone use a pre-built and troubleshooted environment.

in higher management there is aversion to any spend on physical stuff. where i work, we as everywhere look to cut costs. one of ways allowed to save ~$1.5M on aws costs over 5 years, by one time investment of $150k into physical equipment. It fell through.

The other aspect it's that somebody needs to manage old PC. usually there needs to be a team managing this infra and it creates friction between developers and infra team. so devs run into cloud and try to manage everything on cloud themselves. but at some point arrives consolidation and new team dedicated to managing same infra in aws.

Sorry, this won't cut it anymore. You would loose all the fun instabilities when the cluster is upgraded again, the round trip times could not be optimized by distributing things further, the magic of all the different translation layers between you and the hardware would vanish. Worst of all is not all the money and headache you would save but you also would save time. Why would you do that?

Right, much better to deal with all of those issues in production only.

Reply on news.ycombinator.com