What’s Next for Gitlab CI: Auto DevOps
about.gitlab.com
about.gitlab.com
There's no way this claim would stand up to any scrutiny or more data-gathering. It's just marketing fluff.
EDIT: More on what upset me about this claim.
First, we have no idea of what the composition of the 10,000 applications Bitrise surveyed are like. There could be a single company that uses Gitlab to test thousands of applications that completely invalidate this result.
Second, this insults my (and every other competent readers') intelligence. This is badly done marketing. And their audience is developers. So they know they're misleading us but I guess they don't care.
I've worked for Atlassian companies lately but none of their crap calls out to the Internet and tells the mothership what they're up to. Well I guess they do. But not in a way that other people can tell. Totally not an Atlassian shill (often I wish we had gitlab) but their stuff works self hosted real well.
There's no way a third party could know what the numbers are, because when you use software which doesn't sell you out, well, you're not sold out.
But now i have to say i'm afraid gitlab is getting bloated. Why not keep the core product as a seperate thing from things like CI? a simple plugin style system would be enough for it to not feel like bloat but feel like extra options.
If you don't want to use the CI, it shouldn't get in your way. We'd love to hear any specific suggestions you might have, since we're always looking to improve. You can open an issue about them in [3] if you want.
[1] - https://about.gitlab.com/handbook/product/i2p-demo/
Basically, they build the hooks but aren't dogfooding any of it, since their own integrations don't have to.
And speaking from the perspective of someone using GitLab with Jira and Jenkins, it just isn't the same. Is that necessarily because of problems with the interface or is it just those specific plugins? I don't actually know.
While CI is integrated, it does not force you to use it, and there are ways to integrate with external solutions.
While CI is part of the same product, you don't run the tests in the same machine, so you still need to provide the workers in additional ones or use the autoscalling mechanisms and have it bootstrap machines in the cloud as needed.
So what I mean here is that while it is part of the product, it's mostly the "frontend" and the "APIs", the heavyload part of running it is totally optional.
But there are definitely some customers who are sidelined by this approach. Atlassian is pretty committed to tools that talk to each other with documented APIs (Jira, Confluence, Bamboo, Bitbucket/Stash), and for large organizations, that's often a better fit.
There are many players doing "unix" application, and very few trying to build a suite. You need to pick your fights.
First, the automatic notification e-mails stopped working until I did another upgrade. A later update broke issue merging: merging one merge request closed all work requests (event the ones with WIP).
If you also consider the sudden removal of the backlog from the issue board a while ago (they added it back since), I'm now afraid to update our Gitlab instance.
This is very unfortunate because Gitlab is a very nice and useful tool but if QA doesn't improve, it will be very difficult to prevent a migration. These problems are show stoppers for us.
- gitlab (the core)
- gitlab-sidekiq (a work queue)
- gitlab-workhorse (provides the frontend)
- gitaly (a git wrapper that caches stuff)
- gitlab-shell (a shell spawned when doing your git clone)
Those are written in either go or ruby. Sometimes mixing the two in the same repository. In the main gitlab repo, there is also some unvendored js dependencies for the frontend, necessitating to jump through some more hoops.
Gitlab has a bunch of hardcoded paths to logfiles and config files. Some config files are toml, others are yaml. Different services need different configs, sometimes duplicating the config entries in a different format.
I love gitlab as a product. But as a sysadmin, it's one of the worst thing I've ever had to deploy.
[0]: Well, it's not pulled there, but it's at https://github.com/NixOS/nixpkgs/pull/27159 if you're interested.
It works now but I can't hack the source to make custom improvements, so it ends up not being as open source as I would have hoped. There's one particular issue in trying to push that results from being on a URL subpath (http://.../gitlab). I can't fix it myself because of that.
Don't get me wrong--it's a good program and has worked for the team I support. But click and deploy is not useful for people who don't have bare bones servers they can spin up and instead have to work with shared servers.
I think we use toml for the Runner since it is written in Go and should be deployed on another server. The rest is mostly in yml files although we trying to move as much as possible to the UI to make it more user friendly.
Couldn't you just parse YAML in Go, or TOML in Ruby? (Or why not give the option of multiple formats?)
That would be almost precisely against the whole idea of NixOS.
One of the benefits of the omnibus package, is that all configuration is done through a central gitlab.rb file. So the different configuration formats for the individual services is not noticeable to the administrator. Unfortunately, this doesn't necessarily translate to source installations.
We try and provide good documentation for building from source when our omnibus package or docker image isn't an option: https://docs.gitlab.com/ce/install/installation.html. Any feedback on issues you might have following this guide is welcome. I don't know that we can fix the different configuration formats, but we should be able to fix hardcoded paths. Any examples you could provide would be appreciated.
We do provide docker images for those on operating systems we do not support. https://docs.gitlab.com/omnibus/docker/. If you're just looking to run the latest version, this might be the easiest way for you to proceed.
For the omnibus, I'll try looking at how it works, see if it fits our needs better. But I suppose gitaly, which is written in go, still has its own configuration elsewhere, in which you have to replicate the storage list ? That's the sort of thing that I find sort of annoying. It's not a huge issue, but having to manage a bunch of configs, making sure they're all synchronized and up to date sucks as a user experience.
I do have to say, your upgrade guides are very nice and have been helpful when upgrading our instance :).
As far as docker and other things go, I considered it while fighting to get gitlab 9 to work on bare nixos. But the thing is, I manage all my services through nixos, having a central place for all my configuration. This[0] is my configuration (well, a very old one). Everything is in one place, and everything can cross-reference each-other. See my firewall for instance[1], that cross-references various other service's configuration to get their port.
With docker, this is much harder to achieve.
[0]: https://gist.github.com/roblabla/8d1555ceb202eddb1b77
[1]: https://gist.github.com/roblabla/8d1555ceb202eddb1b77#file-c...
So yes, Gitaly does come with it's own configuration file that needs to be managed. It does use the same paths as the main gitlab-rails application, so our cookbook has a method to convert the configuration of one to the other[1]
[0]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/fil...
[1]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/fil...
But when we integrated it the sum was more than the parts. The benefit of deep integration and not having to switch applications are becoming more apparent to us. It is hard to articulate why the same can't be done with plugins, the video in the article is maybe a good start.
Review Apps, changes metrics in the merge request, the container registry being aware of your permissions. All this can be done with plugins. But having it in one applications is so much easier to set up, upgrade, and use day to day.
Dont you mean, Gitlab,
"2/3 of people surveyed by Bitrise, a CI service you may have never heard of, which is free, are using Gitlab, which is also free"
I've been at companies where I've tried (and failed) to convince other developers/higher ups that something like automatically creating an instance of an app based on the feature branch (what Gitlab/others are calling a "review-app"), is the best way to test individual features with high velocity (of course, this requires one more integration level test in the staging environment to make sure everything works together well). I'm impressed/awed/grateful that Gitlab is pulling these features into their 100% open source solution.
The most common complaints against gitlab are:
- Gitlab's UI is bad-ish (this is what made me consider going to work there/contributing and helping fix that)
- Gitlab is slow
And I think they're dumb. If you think UI on your code hosting tool and slow file operations (on the order of minutes, not hours) to be what would slow their org down, instead of the fact that you haven't automated things that Gitlab wants to automate for free for you (like test deployments actual deployments, build steps), in one coherent place (no juggling github, jenkins, jira, whatever else) then I think you're wrong.
The 'review-app' pattern is interesting but doesn't scale for more complicated apps. You'd need to create a 'review cluster' for every branch which quickly becomes untenable. It's usually not lack of automation but cost driving down the number of non-prod environments.
I'm actually a big fan of Gitlab but they should probably focus on user experience and Git hosting and let the other sizable number of products/projects which already exist focus on deployment. Spinnaker, for example, is probably what I'd recommend companies adopt for CD if they're looking to go in that direction.
Also, companies I've worked at in the past (when sufficiently big) have had "playground" AWS accounts that had a fixed cost for daily all-you-can-use resources. I'd like to think that if you're big enough to think you need >2/3 supporting services to come up with your app, maybe you already have enough traction for that kind of cloud playground account?
I can only speak definitively for myself, but most apps I have built are generally 1 app instance, 1 DB and _maybe_ 1 service for caching. Thinking about how to make these easier to spin up (which includes making sure it's easy for me to make a close-to-production environment as quick as possible), has benefited me, despite initial pain.
Also, if cost is the angle, there are tens of things companies could do, but don't that could bring down cost. Outside of "have more performance-minded developers", "add more computers" is often the cheapest option a company has as far as increasing performance (hence the whole rise of horizontal scalability right?).
I agree that Gitlab should focus on user experience (I'm not sure what you were referring to by "Git hosting"), but I am happy that they've decided to add this CI/CD (and issue boards, and CI), in ONE place.
The move to a hamburger menu saw significant outcry, and was followed by one of the more popular tickets I've ever seen on GitLab asking for the ability to restore the menu pin feature. Every complaint and criticism was basically replied with "Nah, we're keeping it.", because the new hamburger menu had gone through "design testing".
If you're looking to improve UI/UX, the design team should look through those tickets and learn how to listen when so many users are speaking up against a redesign.
Rest assured, that we are hard at work at this (see for more information https://gitlab.com/gitlab-org/gitlab-ce/issues/32794). Even better, if I am correct, there will be a new navigation concept (first alpha iteration) released in 9.4! This will be hidden in a feature flag, which will be accessible from your personal settings.
In the coming months (and thus releases, every 22nd!) we will continue to improve upon this concept, to eventually enable it by default for everyone to enjoy.
We know that the shortcomings of the hamburger icon are suboptimal, but also know that reverting is not always the correct choice. With the new navigation concept we focus on getting a clear distinction between global and contextual navigation. We did proper UX testing to be able to clear out multiple bumps in the road/implementation and are pretty stoked about finally getting it in!
Again check for more information this issue https://gitlab.com/gitlab-org/gitlab-ce/issues/32794. The scope of revising navigation is pretty far reaching and has been on the minds of almost everyone on gitlab!
Note that colors, functionality and any specifics can change throughout the process of implementing over the coming months :)
Bad UI on your git hosting tool is very likely not what's holding back your org from delivering faster. It's more likely things like not automating your builds/pipelines/deployment.
Maybe you're at a company that's already doing this properly, you've got Github/Bitbucket Server/whatever set up, with Jenkins Integration, Spinnaker, Quay.io, and you're humming along. While I could make a case for why Gitlab might still benefit you in that environment, it's more likely that at that point, you don't even need it. For every org that is NOT there yet, I think Gitlab offers a benefit -- not thinking about how to get all those other things working.
Maybe where I am on this trade-off is just not the right side, but I see it like this: I prefer a tool that hosts code, automatically tests and deploys a testing instance of the application to host my code than a tool that hosts code but requires more setup to offer that kind of value, even if that second tool has better UI and is seconds or even minutes faster.
The best GUI is the one I don't even have to use -- and I think Gitlab is doing it's best to head towards that, by automate more of the development process for me -- I don't see Github doing this (though they're obviously not against it, they just don't seem to be innovating in this area)
The crowd that would benefit the most has a modern saas mindset.. simple, fast ui. less clutter..
We're working very hard on making it a pleasant and enjoyable experience. We're currently planning a navigation overhaul [1]. You can also check out a list of all current UX issues in [2]. And, any specific feedback is greatly appreciated, you can open an issue about anything at [3].
[1] - https://gitlab.com/gitlab-org/gitlab-ce/issues/32794
[2] - https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf...
I'm glad you can self-host it, and that they're open source, they've got that going. But if GitHub would ever open source their tool, it's over.
I think you're using too much hyperbole -- it's all relative, and if you think Gitlab's UI is the bottom of the pile, you haven't used enough software and truly seen what bad UI looks like. I'm not saying UI doesn't matter, I'm just saying what they have is workable (and can get better, and they've expressed that they care to improve it).
Email, IDE, Editors are not the same as your code hosting, testing, deployment tool, that's a false equivalency. Maybe I must be the only one that is OK waiting 20 seconds for a merge request to complete.
I don't think it is reasonable to hold the fact that they copied/mimiced Github's functionality or UI when they started out against them. Society couldn't move forward if people didn't recycle and improve ideas.
As far as the differences go, while what you said might have been true, it's not these days. Also, you haven't addressed the point that Gitlab offers a shorter path to Doing It Right (tm) (albeit a pretty subjective definition of "right") than Github does.
Why should hosting, testing, and deployment be handled by the same tool (as opposed to tools that handle each need individually)?
`cat file.txt | sort | uniq` is excellent, but I think a lot of the ergonomics of that kind of small/focused tool reusability is due to the super low friction of using pipe, and how well STDIN, STDOUT, and STDERR work behind the scenes. This kind of falls apart when you try and string together web services.
In my opinion, Gitlab's inclusion of so many parts of the process is good because it saves you from having to manage all the other tools, and it gives you answers for the complete process (where it's possible you could get painted into a corner depending on which focused tools you picked to do some pieces of the process, and have to write your own glue or go on a long search).
We also like to think that we bring a couple of novel features to the game. You can check out a list of all our features in [4]
[1] - https://gitlab.com/gitlab-org/gitlab-ce/issues/32794
[2] - https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf...
To the other commenters: Do you have any examples?
Also, I’m excited about auto deployment. The less tools one needs to get software shipped, the better.
On Github you can easily see what happens, comments are surrounded by a black border, the name of the comment author and the date are clearly visible and have a different background color from the comment text. Actions like closing and re-opening are distinctly different from comments and are color coded (green -> re-opened, red -> closed, purple -> merged). Emoticons are part of the comment box and don't visually drift into the next one.
Gitlab on the other hand separates comments by a thin (1px) light-grey line that is hardly visible on low contrast monitors like TN displays. Closing and re-opening is also displayed as a light-grey icon on a white background. The author's name has the same color as the comment text, with the date being, again, light-grey. Nothing keeps the contents of a comment together, it's all just text on a white background. The emoticons are just floating around the comments, with the same margin to the next comment as to the comment they are belonging to. So you have to know that emoticons are always below a comment.
Github's design is far, far superior to Gitlab's design because it allows skimming instead of concentrating on finding where each comment begins and ends.
[1] - https://gitlab.com/gitlab-org/gitlab-ce/issues/32794
[2] - https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf...
My biggest gripe is just that the issue elements (comments, actions, label changes, description edits) are looking too similar for a quick glance. The light-grey font on a white background makes it pretty hard. My suggestion would be to do it a bit like Gihub and to add a black border around stuff, take another background color for actions, and get rid of the grey font and replace it with black throughout.
Loading a page could load almost instantly but most of the time it did load after 2 or 3 seconds.
Gitlab does not work well when resources are scarce, it does require a bit of a beefy server to get fast and reactive, atleast in my experience.
I've found Gitea/Gogs to be a product more focused on running something like Gitlab with much less resources.
For personal and home use Gitea/Gogs is probably a better choice indeed. As you point out it runs with much fewer resources and as an added bonus it runs as a single go binary which simplifies hosting it tremendously. You could literally just download the binary and run "$ ./gitea web". Gitlab is another beast in that respect, especially if you do not opt for their Omnibus install.
The navigation is still a bit convoluted, but has gotten better. There are still some issues of nested navigation menus and difficult to find pages. As far as page speed goes it has gotten better and they are actively working on this.
As long as the rate of improvement is as good as it is I see no reason to switch to another solution. Especially seeing how the community edition is free and satisfies basic developer needs.
Gitlab is also the only code repository service which integrates CI jobs and statuses as well as a front-end for a self-hosted docker registry. Absolutely a killer feature!
If you'd asked me a year ago, I would've said some very harsh things at this point. But I just went to find examples of the issues I had a year ago (and we still have on our own internal GitLab instances), and it looks like most of them have been resolved. There's still weird duplication (why is there still a "Repository" tab when you can see the repo on the front page) but that's all just small warts that don't matter as much.
I think GitLab still suffers from a weird workflow, but it's much better now than it was in the past.
> Gitlab is slow.
The reason why this is an issue is not the fault of the technology imo, but rather a fault of the marketing and community. GitLab is not meant to usurp https://github.com. It's meant to be a viable alternative to GitHub EE.
Unfortunately, there appears to be a split-brain in the community. GitLab themselves are saying that https://gitlab.com has scaling issues that they are working on resolving, but that self-hosting will not have these issues unless you're planning on running at the same scale as them. But the community appears to want more people to swap away from https://github.com to https://gitlab.com (that's the impression I get).
By doing that, you've now said "any comparisons between https://github.com and https://gitlab.com are entirely valid because we're advertising them in the same capacity". That's selling yourselves short.
If GitLab had decided to go for a federated model, then this sort of marketing problem wouldn't be nearly as much of an issue (you would run into the Mastodon effect, where everyone joins the "official server" -- but if you made it possible to migrate the users, permissions, issues and state of the repo then it wouldn't be a long-term concern). They didn't, and that's fine, but that means that marketing https://gitlab.com as a replacement for https://github.com is going to just cause problems.
I just have bare repos on servers I can ssh into. End of problem.
I guess there are probably no surveys of this though...
I didn't even realise Gitlab doesn't have auto-deploy yet.. We set up a .gitlab-ci.yml script that deploys the code.
For organizations it makes sense to have something more featureful.
What other features do Gitlab, Github, or Bitbucket have that wouldn't be covered by that type of setup?
nb: I dont see the point of the graph about numbers of questions on SO. A high level of questions = A lot of problem, no :)?
We use it in combination with BugsEverywhere for issue tracking, with the added advantage that bugs are tracked per-branch. You can work offline without issues.
The major complaint with this system is the disconnect for users to report issues. BE includes a web frontend, but it's not great.
We use mantis as a general tracker, but still keep detailed bug tracking in the VCS with BE.
If I remember correctly gitlab initially started as a frontend for gitolite.
Gitlab provided a datasource. You've got a better one showing your counterpoint, I take it?
[1]: http://blog.bitrise.io/2017/01/27/state-of-app-development-i...
It allows you to organize your repos, and do things like nesting them in categories.
Why Github won't add this feature is beyond my understanding, and I really appreciate Gitlab stepping up and adding this into their platform.
They also have a great system for migrating your repos over from Github.
If you set up a runner in a VM and want to upload artifacts, for some reason Gitlab CI requires you to install gitlab-ci-multi-runner on the VM guest (even though it is the host that is actually the runner). But when you do that, you have to actually go through the entire build process before seeing a non-terminal error that gitlab "couldn't find gitlab-runner". So now you have to go in on the VM guest, manually create a symlink from gitlab-ci-multi-runner to gitlab-runner, take a new snapshot of the VM state, and artifact upload will now work.
I say "non-terminal" above because a failure to upload the artifact doesn't fail the build. So you can be looking at all checkmarks for everything in your pipeline and have no idea for which of those jobs you actually got artifacts.
Also, every once in awhile I'll get failed builds where there is nothing at all in the build log. Typically, clicking "Retry" will successfully start a new build and complete successfully.
Also, there is no way to specify to gitlab to keep only the last N artifacts.
On the positive front, I finally figured out a way to set up an ssh server from within the msys2 environment in Windows, which requires no development whatsoever on the gitlab-ci infrastructure. That means gitlab-ci will ssh into msys2 on the runner, giving me a POSIX-y build environment on all my platforms. So that's nice.
Sorry if this sounds overly negative and like a poor substitute for actually filing bugs. But the last time I worked on fixing a bug it took months for the maintainer just to click a button. This seems to be the only place I get timely response.
Talking about artifacts retention policy, we actually support defining an expiration date and we're also considering to add a maximum number in the future. You can find discussions about this topic in https://gitlab.com/gitlab-org/gitlab-ce/issues/23777 and https://gitlab.com/gitlab-org/gitlab-ce/issues/19208: in these issues there are also problems and consideration about having this feature implemented.
Feel free to jump in and push your ideas!
I was blown away by how complete, fully featured, and easy to use it is for a free product.