CircleCI Down
status.circleci.com
status.circleci.com
Sounds like they haven't got the first clue about what is causing it.
Catastrophes happen. If you deploy something that has a destructive migration you can't easily roll back without reverting to a backup and there's a problem that you've not seen in QA then you're in for a bad time. This is compounded if you also discover your backup process hasn't worked properly for a while. If that happens you're facing some serious downtime, and the dilemma of either trying to fix the problem, or trying to rollback to the last working backup.
There's a good reason why grumpy old devs like me insist on writing docs, having playbooks, testing everything including non-code stuff, and we still fear major deploys. I have scars from exactly those sorts of disasters.
Hopefully the devs at Circle get past this with as little stress as possible, and they learn from what went wrong.
So you might end up communicating something ("we think it comes from X, we're fixing it that way"), just to find yourself changing you mind a few minutes later.
Changing your message is usually not well perceived, even though that's actually normal during an investigation.
I would not like to be in charge of the communication. Finding the balance between saying too much or too little is tricky.
I just don't have a tremendous amount of confidence that they take their infrastructure seriously at this point.
Most 3rd party senders today already insist that you setup DKIM as part of your setup process and if that happens, you're going to pass a DMARC check. It's hard to setup for older companies with thousands of servers in their own data centers that are each individually sending email. Cloud native companies sending their email through a few 3rd parties like Sendgrid/Postmark or a newsletter tool are EASY to setup.
I'm mentioning this on a post about their infrastructure being down for 6 hours because yes, it's related. Email delivery for the primary domain is absolutely an IT, Engineering, Operations and Security problem, not a marketing problem. It goes directly to the application especially when one of the main facets of the application is to send emails about your repos and login credentials.
Blame shifting it to the marketing department does not hold up.
When multiple people are commenting on this post about just how frequently their outages are happening it shows a problem in the overall infrastructure mindset for it to continue. Maybe they know exactly what the problem is and somebody higher up is keeping them from fixing it in order to prioritize other things.
Either way, for company that's supposed to be providing a core devops function to have outages that frequently as well as making it dead simple to spoof email that looks like it's coming straight from them...it's not a good look.
I agree that reviewing is the first step, but not everyone needs to take further steps. And I highly doubt CircleCI is unique here. I think it's a massive leap to conclude "lack of confidence in taking their infrastructure seriously" from not knowing the reason why they haven't flipped the switch from none to reject or quarantine.
Technically sophisticated users know that email spoofing is already rampant and to watch for signs of it in their email client. I'm not saying it's not a good idea, but that flipping the switch is not that simple and comes with significant downsides in a company with many services and users.
IMO I think going to the next level with DMARC is usually more of a prioritization or cost-benefit analysis type decision than a competence once.
For an core devops tool, that's not okay.
We get all the customization I want and it scales just fine.
But I am also still annoyed that when CircleCI announced their templates, they did not offer you the ability to have private templates (or something along those lines, it left a bad taste in my mouth and we moved to Jenkins a couple months later)
These days GitHub Actions is awesome and full featured and does everything I want though without getting in the way. I doubt I'd use anything besides GHA unless I wanted to decouple from GitHub as a dependency.
Even then, I wouldn't be surprised if someone else hasn't already come up with an open source or local runner for the GHA yaml files as a stopgap.
- you are limited to 2 core / 7 GB instance, whereas CircleCI offers up to 16 cores / 64 GB (for example, building software usually scales proportionately to machine size, and this can in theory be up to 8x faster on CircleCI)
- no support for ARM instances
GHA also supports bringing your own self-hosted runners [1][2] where you install their agent, so you could, e.g., use an ARM instance on AWS with tons of cores. It looks like CircleCI offers this as well.
[1]: https://docs.github.com/en/actions/hosting-your-own-runners/...
..at least that's my experience and why I founded an extremely opinionated CI service for mobile apps years ago :)
But I can see that specific use case being a good one for an opinionated CI environment.
I largely work with Docker containers and Serverless. 2 areas that I don't see any advantages going outside of Jenkins (and see many downsides)
It is significantly cheaper and more efficient to pay a third party.
That said, there are a number of reasons to not use the Jenkins server in the office:
1) Someone on staff needs to maintain it.
2) A single hardware failure can cause significant downtime.
3) Your office internet service may have limited bandwidth and be a bottleneck for your build or artifact deployment
4) Having your server on-site may be considered a security risk.
I'm not saying that a server in the office is a bad idea, I'm just saying that each business needs to consider the advantages and disadvantages. I'm sure there are those who could get by just fine with a server in the office.
You don't want that server in some spot where someone can accidentally kick it and suddenly the server goes offline. You also don't want it in an unventilated closet where it'll overheat.
As for noise... in this day and age of open office spaces and chatty coworkers, you'll be lucky if the server noise is your biggest problem. Unless everyone is working from home, and now the problem is physically accessing the server if it goes down or needs other physical maintenance.
Forgive me if this is just "one of those problems" everyone has to deal with. I'm admittedly still pretty young and naive, and I've found myself in a mentorship/leadership adjacent role at a fast growing company where my relatively small team has a vastly deeper well of technical knowledge and aptitude than the rest of the organization. So I do wonder sometimes if I'm wrong about it, but my intuition, and it seems that of GP's too it seems, drives me to spring to this kind of solution (that we should be doing whatever it is ourselves because it's mission critical for what we do) often and it really gets to me that it's not so readily heard.
It's somewhat ironic because a big part of the point of HN in general is building and promoting these kind of progressive services, but the longer I'm in the industry and the more advanced my knowledge and responsibilities get, the more I keep finding that it's really hard to communicate the gaps in between the marketing claims these SaaS tools promote to higher-ups who don't see the incurred technical debt and wasted time working around their shortcomings. I've found this to be especially true when the issues are ones that one only bumps up against after rigorous use or problems that decision makers try to address by encouraging the addition of yet another third party tool to the mix.
I cited problems with hosting something on-site, but didn't list the problems of integrating with a service too.
When you integrate with a service you need a member of staff to maintain that integration too. Of course we hope that the integration is easier than maintaining a server but that is not always the case.
You also take on the risk of the SaaS being flaky, like we see with CircleCI.
The SaaS can always raise their prices or go out of business, requiring your company to switch off of that service.
Maybe the SaaS is a security risk. What happens to your company data and builds if/when they get breached?
There's no silver bullet, there's just a set of trade-offs. You get to decide which is best for your use-case.
I really like the way you worded this. I think in hindsight it's obvious but you conceptualized it in a way that I've found hard to communicate to others so far. I see now that this is probably because I've been so zeroed in on my own idiosyncratic technical issues with a given service when applicable rather than expressing it through the criteria the people I report to must consider when evaluating solutions brought to them, so thank you for that!
In both cases, have seen the "CI is down" issues crop up.
There seems to be a middle ground that I have yet to fully do, but have seen folks do and heard it works well:
Make it so folks can deploy their CI pipeline anywhere. In my case, I would create Docker containers that can run the CI pipeline, pipe the output/result to anywhere you want and have it be able to run anywhere.
The goal is that you could deploy it on a VPS somewhere, or on a Dell blade in your closet, or even run it on a dev's machine. The idea is that you want to reliably be able to run the CI steps anywhere without jank while following the full process the same as if it was running on your dedicated job runner solution. Redundancy, if you will.
I think the idea isn't necessarily "Jenkins in your company closet is better" and more of "how easily can you setup your CI pipeline to get your work done?"
Then they reworked their YAML format to enable pipelines (and make everything way more complicated, and break all our existing flows). Then they reworked their images to make them more efficient or something, but now there is almost never an image that has everything I need, so I have to build my own image in docker (with yet another CircleCI job). Now that Github Actions exists, we have been slowly migrating everything off CircleCI. Too bad they lost the simplicity that brought us there in the first place.
The problem CircleCI faces is that most hosted VCS now support CI/CD tools, as do most enterprise clouds. These will all have better integration with most peoples systems because they'll already be using the VCS or the public cloud (and if you're not using the public cloud you'd likely favor Jenkins / Concourse / etc over a cloud CI solution). So CircleCI's relevance is constantly being eaten at. The last thing they need it to damage their own reputation with these constant outages. I really hope CircleCI can turn around as it's great to see some competition but at this point in time I'm not feeling to optimistic about their long term future.
a) X gets large enough to matter and also has scale issues.
b) X gets acquired by Y and sends a "what a great journey we're so excited that X will have Y's resources" email and then inevitably becomes just another forgotten tool under Y
/s sorta
Not before Y cuts all development to X. It doesn't always happen, but if you see Embarcadero buying a piece of your tech stack, find an alternative, immediately.
Been on BK for about 6mo now and still loving it.
What do you mean by "these types of services"?
E.g. I've worked in a business who self-host their Gitlab instance, there was a non-negligible amount of work for backups/upgrades on top of troubleshooting performance issues once in a while, amongst other things in the same vein.
Granted, self-hosting comes with it's own share of problems too, no solution is a silver-bullet without any issues, but being able to "freeze" things to a stable mode helps to stabilize other processes.
Most people don't want any responsibility though so they do everything they can to push work off to a SaaS or cloud provider. That way when the SHTF they can browse reddit and point their boss to the status page.
Phew, those are a lot of assumptions. Many businesses choose cloud services cause they take the tradeoff between that cost or having someone at hand to maintain the infrastructure, depending on its complexity. I've worked with employers using many cloud providers for years now, the amount of times I would _legitimately_ be prevented from working at all by outages is extremely low.
The agency part, I do get. But come on.
However, I'd still argue that choosing when downtime can happen is important when you're pushing out larger changes to larger organizations. You don't want to be in the middle of a borked migration when your CI/CD service craps out and you can't rollback/push more updates or even restore backups as it was all automated via your CI/CD service.
GitLab CI #1 in my opinion.