dnsmichi

GitLab team member here.

You can make suggestions to move features between tiers, i.e. moving a paid feature to the free tier, by opening an issue following https://about.gitlab.com/company/pricing/#changing-tiers-and...

GitLab team member here.

> it would be nice if there was an option for a group-level Readme on Gitlab.

Thanks for the idea. The proposal for Group READMEs is discussed in this thread: https://gitlab.com/gitlab-org/gitlab/-/issues/15041#note_102... Suggest adding your preferences for the proposed solutions, thanks.

Thanks, great idea. Added a comment into the feature proposal in https://gitlab.com/gitlab-org/gitlab/-/issues/273250#note_11... - suggest upvoting and/or subscribing to notifications.

Today I learned, thanks a lot!

Created a MR for the Developer Evangelism Hacker News handbook to add this formatting tip, and some more https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...

Thanks for sharing your ideas, appreciate the thoughts to learn new perspectives :-)

And also thanks for contributing, every contribution counts and helps :-) Maybe see you at a future hackathon :) https://about.gitlab.com/community/contribute/

True, thanks. Some workflows can require larger merge requests, having the platforms and tools that enable smaller iterations help reduce (or eliminate) them though.

You are more brave than I am - I have it explicitly disabled :-) https://gitlab.com/dnsmichi/dotfiles/-/blob/main/.gitconfig

When I pull and it runs into a merge conflict, I want to see that error/log on the CLI. Reason: Sometimes the automated pull-rebase takes a very long time to resolve conflicts after each step. I prefer to first run

  git fetch

  git diff branchname origin/branchname

and then decide my strategy :)

Bit off-topic but since we share .gitconfig tips - I upgraded to a recent Git version and enabled the "git push" option to setup tracking automatically. No more "git push -u origin branchname" actions. https://gitlab.com/dnsmichi/dotfiles/-/blob/main/.gitconfig#...

  [push]
   autoSetupRemote = true

After reading this comment about "git push --force-with-lease" in https://stackoverflow.com/a/53011907/1821348 my workflows with "git fetch" first would not work with "git push --force-with-lease", behaving as if "git push --force" was run. Theoretically - I need to do some testing here.

Thanks for the pointer into learning something new :)

GitLab team member here, putting my personal hat on - from my experience in using different Git workflows since 2009, a smaller clean unit of work can help with debugging and troubleshooting. It also provides a way to new team members and contributors to understand the thought process and ideation to implement a new architecture, apply performance fixes, add documentation, work with tests, additional fixes, until its final release. Most of this can be tracked within a MR/PR and the history of code reviews, etc. - even after the merge and squash and Git branch delete, not trying to argue with this functionality. :)

From the Git CLI, without any reference to Git* platforms, it is not so obvious when searching for a commit that introduced a bug, e.g. using "git bisect" for binary search. Reading a 10,000 lines git diff can be harder than a smaller commit that also explains the reasoning in the commit message. Speaking from own experience and programming mistakes in a small team, focussing on clean commits and a good history tremendously helped in stressful debug situations. Until you hit a compiler regression bug, but that's a different story then ;)

I'm personally still very fast on the Git CLI, but I also know that there are a variety of CLI and UI tools out there that can help with analysing large Git commits. Potentially in the future also AI assisted that tell us which change a diff caused a performance regression in a release 5 months later. Or we don't need it at all because Observability driven development enabled to see these problems before merging and code reviews, e.g. the memory leak but only when DNS fails. True story from ~2016, more in my KubeCon EU talk at https://www.youtube.com/watch?v=BkREMg8adaI and project at https://gitlab.com/everyonecancontribute/observability/cpp-d...

Depending on the workflow and collaboration in branches in teams, amending may not be ideal. Amending changes to an existing commit will generate a new sha checksum, thus resulting in a diverging history if the branch is often pushed to the remote Git server. That's one of the rare cases where "git push --force" makes sense but also requires awareness for other collaborators on the branch - they need to run "git fetch && git reset --hard origin/branchname" every time, and best "git stash" to temporarily remove any local changes, with running "git stash pop" afterwards.

One reason to regularly push commits can be to trigger CI/CD, security scanning and dev/staging deployments or review apps.

GitLab team member here - before joining GitLab in 2020, I was a Git/GitLab trainer. https://www.netways.de/en/blog/2018/05/24/releasing-our-git-...

GitLab team member here.

You can use `git rebase` to perform the so-called "squash" action, that squashes multiple commits at once.

For example, the latest commits in the branch look like this:

123 WIP1

456 WIP2

789 WIP3

012 FeatureA

WIP1,2,3 as commits should be merged into a single Git commit, and be put on top of the FeatureA commit - so to speak, use the FeatureA commit as a new base.

Git uses the rebase command to do exactly that - you'll perform an interactive rebase onto FeatureA commit. The interactive rebase allows you to define specific actions.

"pick" keeps the commit. In order to perform a squash of the WIP1,2,3 commits, you'll keep the first WIP1 commit, and tell Git to continuously squash the next commits WIP2 and WIP3. Each operation is done at once.

pick 123 WIP1

squash 456 WIP2

squash 789 WIP3

pick 012 FeatureA

This results in a new history and commit - squashing changes the commit sha checksum, with new content in the Git commit, as well as a new base commit.

567 Squashed

012 FeatureA

"squash" in an interactive rebase keeps the individual commit messages for each commit, and at the end, the editor for commits allows you to edit the final messages. This can be helpful to review the squash action, and for example amend or abort it.

If you plan to not keep the individual commit messages, "fixup" throws them away and can be used as alternative action.

"git rebase -i" has more options - you can also stop at a specific commit, and amend it, e.g. when a "git add" was missing a file earlier.

Note: Any change to a commit within the Git history forces all later commits to change too, as the linked commit base changes too, thus regenerating the sha checksum. This entirely rewrites Git history trees, and can be very invasive - if you intend to keep specific commit IDs (for release tags for example), ensure that the workflows to not allow rebasing on certain branches. One possible workflow is to keep the main default branch protected, disallowing to rebase and push a changed history, add git tags there, and only rebase in a Merge Requests branch prior to review/approve/merge cycles.

Moving from Git commands to GitLab - GitLab also offers a Merge Request option to squash commits automatically when the MR is accepted, so that all commits in the MR branch are squashed, and you don't need to do it manually. https://docs.gitlab.com/ee/user/project/merge_requests/squas...

Last but not least - the quick action /rebase in a MR comment allows to trigger a rebase via the UI too, thus not requiring client side clone/fetch/rebase. More tips in https://about.gitlab.com/blog/2021/02/18/improve-your-gitlab...

GitLab team member here.

Thanks for sharing, I have posted the project in the forum topic collecting artifact cleanup resources [0]

If you need a way to generate lots of artifacts for testing your script, you might find this small CI config helpful. [1] It uses matrix builds, and dd to generate blocks of artifact data, with no extra dependencies, and can be included as a remote include in your test projects.

[0] https://forum.gitlab.com/t/remove-all-artifact-no-expire-opt...

[1] https://gitlab.com/gitlab-de/job-artifact-generator

> The instructions for removing the most recent artifacts also just never works (https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#k...)

What's the error / job logs you are seeing with having the project settings disabled for keeping the latest artifacts? The async operation to delete artifacts can take a while. Suggest to continue on the GitLab community forum: https://forum.gitlab.com/ where more folks can help. If you continue seeing the problem, also suggest to create an issue as a bug report. Thanks! https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_t...

Another thought: You could clean the artifacts via the GitLab API, if that helps. An example script is available in https://gitlab.com/gitlab-de/gitlab-storage-analyzer also listed in the FAQ https://about.gitlab.com/pricing/faq-efficient-free-tier/#ma...

> What does this cover?

https://about.gitlab.com/pricing/faq-efficient-free-tier/#ma... provides a list of what counts towards the storage limits, and also ways to manage, reduce and cleanup storage.

> Does it cover the size of a git repo? Build artifacts, both temporary and exported to packages? And what about Docker images?

Yes, they all are storage types that add to the overall storage consumption.

> Docker images concern me the most. It's terribly easy to build and use Docker images that are >100MB, and 50 of those easily go beyond 5GB. Two projects with nightly builds easily go beyond that in a month even when doing nothing at all.

You can define container cleanup policies within the registry, matching on age, tags, etc. https://docs.gitlab.com/ee/user/packages/container_registry/...

GitLab team member here. It seems related to the problem that you cannot open an empty PR https://stackoverflow.com/questions/46577500/why-cant-i-crea... and force overriding an old branch base is a workaround to land at an empty diff. Another workaround could be git commit —-allow-empty, making the GitHub UI think that no changes between branches is equal to the merged state.

The behavior is not reproducible in GitLab, where empty merge requests can be created, and remain in the open state, also with zero changes at some later point. Added in GitLab 9.2 as part of the issue to new branch and wip/draft merge request flow: https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28558

GitLab team member here, thanks for sharing!

You can also set Git push options understood by the GitLab server to create merge requests [0] on the CLI.

Sid's dotfiles provide an example in [1]. The workflow is 1) push 2) create merge request 3) set target (master/main) 4) merge when the pipeline succeeds.

alias mwps='git push -u origin -o merge_request.create -o merge_request.target=main -o merge_request.merge_when_pipeline_succeeds' # mwps NAME_OF_BRANCH

There are more push options, such as setting the MR as draft, add labels, milestones, assignees, etc. My personal favorite: Remove the source branch when the MR is merged. That's a project setting too, but sometimes not set. Using the push options, you can force this behavior and avoid stale Git branches.

glab as CLI tool provides a similar functionality to create an MR. Its development has been moved to this project [2]

[0] https://docs.gitlab.com/ee/user/project/push_options.html#pu...

[1] https://gitlab.com/sytses/dotfiles/-/blob/master/git/aliases...

[2] https://gitlab.com/gitlab-org/cli

Thanks for your feedback. I’d suggest starting the analysis to identify the type of most storage being consumed. Maybe CI/CD job artifacts are kept forever and need an expiration configuration [0], or container images in the registry would need cleanup policies [1]. The documentation linked in the FAQ provides more analysis and guides to help. [2]

If you need additional help with analyzing the storage usage, please contact the GitLab support team. You can also post cleanup questions on our community forum. [3]

[0] https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#w...

[1] https://docs.gitlab.com/ee/user/packages/container_registry/...

[2] https://about.gitlab.com/pricing/faq-paid-storage-transfer/#...

[3] https://forum.gitlab.com/c/gitlab-ci-cd/23

Thanks for sharing for your feedback. I have added more insights into caching and checkout strategies to reduce traffic and speed up job execution in GitLab Runners in this comment: https://news.ycombinator.com/item?id=32408960

Thanks for the suggestion of adding a Git cache for a GitLab Runner. A similar mechanism already exists that allows control of the Git strategy with fetch on a local working copy which is faster, and clone if not yet existing. All benefits and limitations are documented [0].

To help prevent unneeded traffic when a new pipeline job is executed, GitLab Runner uses a shallow Git clone by default on GitLab.com SaaS that only pulls a limited set of Git commits from the current head instead of a full clone. [1]

There is a feature proposal [2] to add support for partial clone and sparse-checkout strategies that have been added in more recent Git versions. Recommend commenting/subscribing.

Please note that for GitLab.com SaaS Shared Runners, the traffic limits do not apply with mostly internal cloud traffic. [3]

[0] https://docs.gitlab.com/ee/ci/runners/configure_runners.html...

[1] https://docs.gitlab.com/ee/ci/large_repositories/#shallow-cl...

[2] https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26631

[3] https://about.gitlab.com/pricing/#what-counts-towards-my-tra...

> erhaps now is a good time to recommend the ever-popular BFG to anyone unaware: https://rtyley.github.io/bfg-repo-cleaner/

Thanks for sharing this tool to help cleanup the Git history. Please be aware that it will rewrite the history, which could be very impactful to existing branches, merge requests and local clones. A similar approach is described in the documentation: https://docs.gitlab.com/ee/user/project/repository/reducing_...

GitLab offers ways to manage retention policies for your container registry. Please see the docs for guidance: https://docs.gitlab.com/ee/user/packages/container_registry/...

> And, I'm not sure if GL is counting the whole history, or the current? And are they pruning old after a certain date?

GitLab stores the Git history with all data revisions as commits. To reduce the repository storage [0], older Git commits can be pruned but this would rewrite the Git history as an invasive change [1]. To store larger files, it is recommended to use Git LFS [2]. Alternatively, you can use the generic package registry to store data. [3]

[0] https://docs.gitlab.com/ee/user/usage_quotas.html#manage-you...

[1] https://docs.gitlab.com/ee/user/project/repository/reducing_...

[2] https://docs.gitlab.com/ee/topics/git/lfs/

[3] https://docs.gitlab.com/ee/user/packages/generic_packages/

GitLab team member here.

Maybe you can re-use the script I shared in this comment [0] to query the GitLab REST API to generate an index. I'm a fan of using the API programmatically, and tend to avoid git checkout/grep/etc. inside CI/CD pipelines (works too, everyone is free to choose their way).

Yet again, I love API challenges, so I've created a new script which parses a defined markdown footer for Tags and Due date, and generates an ordered index by due date. I did not know which format you are using, so I made up my own, see the MR description [1] and docs [2].

The script lives in [3] and is a mix of fetching files, parsing content with regex, and generating the index + creating a commit to upload automatically.

A demo overview is shown in [4] with the generated index.md, ordered by due date and linking the files by parsed heading title, file paths, and tags.

Hope it helps, feel free to repurpose, or ping me for questions on the GitLab community forum [5]. My Python code is a little rusty, I am slowly adopting all the 3.x design patterns after many years with 2.x.

I might follow your idea with notes and custom footer parsing. That's a really nifty idea, and helps solve my own chaos :-)

[0] https://news.ycombinator.com/item?id=32155848

[1] https://gitlab.com/dnsmichi/api-playground/-/merge_requests/...

[2] https://gitlab.com/dnsmichi/api-playground/-/tree/main/pytho...

[3] https://gitlab.com/dnsmichi/api-playground/-/tree/main/pytho...

[4] https://gitlab.com/dnsmichi/api-playground/-/tree/main/demo/...

[5] https://forum.gitlab.com/u/dnsmichi/summary

Thanks for flagging. opsindev.news is hosted on GitLab Pages using Let's Encrypt. Maybe the TLS ciphers or versions do not match. Which error message do you see, browser/OS and location may help (if you want to share)

If the MkDocs website does not work, suggest the following workarounds: Newsletter issues in [0] as markdown files, example [1] or the newsletter archive on Buttondown [2].

I'm using Buttondown to send the newsletter, MkDocs serves as web-searchable archive. Kudos to Michael Hausenblas here for the idea, he publishes the o11y.news newsletter.

[0] https://gitlab.com/dnsmichi/opsindev.news/-/tree/main/docs/a...

[1] https://gitlab.com/dnsmichi/opsindev.news/-/blob/main/docs/a...

[2] https://buttondown.email/opsindev.news/

GitLab team member here, thanks for sharing your experience!

The UX with editing the wiki, using WYSIWYG content editor [0] has been improved in recent releases. [1] I peeked into 15.2 (self-managed release coming on Friday, July 22) which brings rendering Mermaid, PlantUML, Kroki diagrams previews. [2]

Maybe the wiki can be worth to revisit in the future, there are more features underway, suggest diving into this epic [3]

> Issues have the same markdown support, we can use ctrl+v to insert screenshots, we can add tags to the issues, we can assign people to relevant issues, we can comment with new things to do/add, and we can use the search bar to find relevant content. All our "knowledge doc" is stored insides the issues of an empty project named "doc".

Great to see that you have found a workflow, and use issues to keep things documented and organized. Maybe a suggestion for creating an entry point into the dcc project: Use GitLab CI/CD to read the issues from the REST API, group by label, and render a Markdown README.md file which gets pushed to the repository automatically. That way the "index" is generated and provides greater visibility into the documentation issues.

I like API challenges, so I've hacked [4] a small script [5] using python-gitlab to better illustrate what I mean :-) Uses issues [6] with labels to generate an index.md [7] (can also be README.md for example).

Feel free to repurpose for your own needs, if that helps. The missing step is to automate it using GitLab CI/CD Schedules, but that's documented.

[0] https://docs.gitlab.com/ee/user/project/wiki/#content-editor

[1] https://about.gitlab.com/releases/2022/05/22/gitlab-15-0-rel...

[2] https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86701

[3] https://gitlab.com/groups/gitlab-org/-/epics/5401

[4] https://gitlab.com/dnsmichi/api-playground/-/merge_requests/...

[5] https://gitlab.com/dnsmichi/api-playground/-/tree/main/pytho...

[6] https://gitlab.com/dnsmichi/api-playground/-/issues

[7] https://gitlab.com/dnsmichi/api-playground/-/blob/main/index...

GitLab team member here, thanks for sharing!

I'm using the Web IDE to take notes in most of my projects, work and personal, and publish the notes with MkDocs and GitLab Pages to a searchable frontend/domain when needed. Editing also happens in Gitpod with live preview in the browser.

You can find all resources for o11y.love [0] and opsindev.news [1] in the GitLab projects, including .gitpod.yml configuration, mkdocs.yml setup, .gitlab-ci.yml deployments.

I have been writing lots of documentation in my past OSS projects, so I am used to Markdown as markup language, taking notes very fast. Learning Markdown requires some practice, and can be helped within Gitpod and the VS Code extensions, if the default preview is not sufficient. [2] [3] You can also sync the notes repository offline into VS Code as desktop IDE for example.

Using Obsidian.md to take notes and publish with GitLab pages [4] looks promising too; I have not tried it yet.

[0] https://gitlab.com/everyonecancontribute/observability/o11y....

[1] https://gitlab.com/dnsmichi/opsindev.news

[2] https://www.gitpod.io/docs/ides-and-editors/vscode-extension...

[3] https://code.visualstudio.com/docs/languages/markdown

[4] https://about.gitlab.com/blog/2022/03/15/publishing-obsidian...

Wonderful, thanks for letting us know, appreciate your collaboration and patience in resolving the problem. :)

Thanks for opening the ticket already :-)

I have talked with the fulfillment team at GitLab [0], and there was a problem identified with workers not syncing the purchased CI minutes after the DB migration. Corrections have been made, and potentially you should see the synced CI minutes soon. Support teams have access and can check with you, I do not have access.

Setting up your own runners is a sensible approach, and allows you to scale them for your own needs too. Maybe this workshop about pipeline efficiency can help with more ideas and insights [1].

[0] https://about.gitlab.com/handbook/engineering/development/fu...

[1] https://about.gitlab.com/handbook/marketing/community-relati...

GitLab team member here.

Thanks for asking - I have forwarded your question; sharing a summary what I learned below. Note that I am not a Rails engineer; for deeper questions I'd suggest commenting into the linked issues and tag engineers directly :)

Rails 6+ multiple database support is being used by GitLab. The major difference is that a hand-rolled DB load balancer is currently used. [0] tracks the effort to change the load balancer to the native Rails implementation for connection handling.

This comment summarizes [1] how things worked, and how Rails' support of multiple databases can help. It also provides flow diagrams and code snippets for better understanding.

Side note: The multiple database feature effort is also related to bringing Clickhouse as a datastore for Error Tracking [2] and more Observability data [3].

[0] https://gitlab.com/gitlab-org/gitlab/-/issues/296870

[1] https://gitlab.com/gitlab-org/gitlab/-/issues/296870#note_88...

[2] https://gitlab.com/groups/gitlab-org/-/epics/7772

[3] https://about.gitlab.com/direction/monitor/observability/#un...

GitLab team member here.

The comment [0] provides more insights into the planning and downtime requirements. The epic itself may be helpful too, it is linked from the blog post.

[0] https://gitlab.com/groups/gitlab-org/-/epics/7791#note_94102...