Hacker News Reader

Comment by delta_p_delta_x | Hacker News Reader

> without needing to create a CMakeLists.txt, adding a submodule for every dependency you use, including your dependency's cmake file, blah blah (xmake has its own xrepo tool which allows for dependency management in a MUCH easier way)

You make it sound worse than it really is today. Using submodules for everything is the pre-vcpkg, pre-FetchContent, pre-ExternalProject way of including dependencies—more than half a decade out of date, and arguably more appropriate for GNU Autotools.

With CMake and vcpkg, it's not that much harder: add vcpkg as a Git submodule, add a vcpkg.json, and use its CMake toolchain to bootstrap and install packages with find_package(), done. Said vcpkg.json can be as minimal as (taken from my own projects):

  {
    dependencies: [
      'spdlog',
      'vulkan-sdk-components', 
      'libpng'
    ]
  }

View on HN

Every time I try to fully migrate from submodules I hit the following roadblocks:

- vcpkg: dependencies that have custom patches only relevant for my software or where maintainers apply patches once every six months, dependencies where the author doesn't do versioning and the correct version to use is git HEAD

- vcpkg: not sure how I can pass specific flags to dependencies. For instance I need to build LLVM with specific CMake flags.

- CMake FetchContent : how do you handle dependencies that are other repos from your organization which may definitely get patches as part of the development of the software? With FetchContent all those go into build directories and aren't treated as source, I would like to be able to tell CMake "for this build, use source folder /foo for dependency "foo" instead of cloning its 300MB repo again

- How do you handle dependencies that takes ages to build. My software uses Qt, llvm, libclang, ffmpeg and a fair amount of other things. When I tried with vcpkg, the experience building for a new contributor took something like three hours on an average laptop and required dozens of gigabytes of space (software build itself is ~5 minutes with the current precompiled SDK I ship). The space thing is critical, I often get students, interns, OSS contributors etc which definitely cannot afford 30GB of free space on cheap laptops with 256G SSDs

> - vcpkg: dependencies that have custom patches only relevant for my software or where maintainers apply patches once every six months

Yeah, this is a pain point; maintainers need to be more on-the-ball about updating and maintaining packages and package versioning. Maintainers for large Linux package repos (e.g. apt, yum, pacman) can do it, I don't see why vcpkg maintainers can't.

> vcpkg: not sure how I can pass specific flags to dependencies. For instance I need to build LLVM with specific CMake flags.

Use overlay ports[1] and edit the portfile.cmake to pass in additional variables[2]. If you want this to be really configurable every time, then use `VCPKG_ENV_PASSTHROUGH`[3] in a custom triplet[4]. This Stack Overflow answer[5] (full disclosure: I wrote it) explains why you have to do this.

> I would like to be able to tell CMake "for this build, use source folder /foo for dependency "foo"

Use `FetchContent_Declare` with `SOURCE_DIR`[6; this leads to `ExternalProject`, but everything you can use there you can also use with `FetchContent`]. e.g.

  FetchContent_Declare(fmt SOURCE_DIR "${CMAKE_CURRENT_LIST_DIR}/../thirdparty/fmt/")
  FetchContent_MakeAvailable(fmt)

> How do you handle dependencies that takes ages to build.

Link vcpkg's asset and binary caches to a network drive that all developer terminals can access straightforwardly. S3, Azure, GCP, GitHub, local/network filesystems are all supported[7][8].

[1]: https://learn.microsoft.com/en-gb/vcpkg/concepts/overlay-por...

[2]: https://learn.microsoft.com/en-gb/vcpkg/get_started/get-star...

[3]: https://learn.microsoft.com/en-gb/vcpkg/users/triplets#vcpkg...

[4]: https://learn.microsoft.com/en-gb/vcpkg/concepts/triplets

[5]: https://stackoverflow.com/a/77954891/1654223

[6]: https://cmake.org/cmake/help/latest/module/ExternalProject.h...

[7]: https://learn.microsoft.com/en-gb/vcpkg/users/assetcaching

[8]: https://learn.microsoft.com/en-gb/vcpkg/consume/binary-cachi...

I mean, you say "Linux distributions can do it" but obviously you cannot use distro packages for any serious deployment where your users expect your app to look and feel exactly the same whether they're on Ubuntu or Fedora or Arch (or macOS or Windows). Or if you have patches, e.g. I have a couple patches to Qt.

> Use `FetchContent_Declare` with `SOURCE_DIR`

I know about this but it doesn't solve my problem: I want people who develop to be able to point to the source dir elsewhere, but I want e.g. CI and people who just grab it from github to have it point to some commit in usual FetchContent fashion. And I don't want to have to pass CMake flags for each "co-developed" dependency, that would be a very bad developer experience too. Every cmake flag I add is another couple support tickets from junior developers mistyping it in my experience.

> a network drive that all developer terminals can access straightforwardly

Not viable for OSS projects developed in the wild

> And I don't want to have to pass CMake flags for each "co-developed" dependency, that would be a very bad developer experience too. Every cmake flag I add is another couple support tickets from junior developers mistyping it in my experience.

Then set up your CMake logic to use FetchContent_Declare based on the boolean value of an option, set up CMake presets[1] with hard-coded values for each CMake command-line option, and ask your developers/set CI to choose between these presets.

You mentioned a precompiled SDK; you can still use vcpkg and overlay ports to redirect to these prebuilt binaries[2]. Replace 'system package manager' with your own paths.

Or, forgo vcpkg altogether if you think the long initial bring-up is a big problem, and just use CMake and your pre-compiled SDK to expose packages, libraries, and headers that your consuming source needs. Not everything is a vcpkg-shaped nail that needs a vcpkg hammer.

[1]: https://cmake.org/cmake/help/latest/manual/cmake-presets.7.h...

[2]: https://devblogs.microsoft.com/cppblog/using-system-package-...

Cmake + submodules seems to be the most reliable option. I always hit issues with vcpkg.

> more than half a decade out of date

Yes... but as great as vcpkg is, it's still not ubiquitous enough that everything is on it, in the same way that everything is available for Rust via Cargo, or for Python via pip, or for Java/typescript via NPM.

So submodules are still used quite a lot.

> but as great as vcpkg is, it's still not ubiquitous enough that everything is on it

Completely fair assessment; I wanted to add MIT's krb5 and realised it wasn't on vcpkg.

> So submodules are still used quite a lot

And this is the wrong solution. The correct one would be to put in the leg-work and write a new portfile[1] and submit it as a PR[2].

[1]: https://learn.microsoft.com/en-gb/vcpkg/get_started/get-star...

[2]: https://learn.microsoft.com/en-gb/vcpkg/get_started/get-star...

> And this is the wrong solution. The correct one ...

This is not very realistic. I don't want to become a package maintainer of somebody else's library.

But I 100% agree that submodules are not good. I'm hoping Pijul will handle that sort of thing better but I haven't tried it.

> This is not very realistic. I don't want to become a package maintainer of somebody else's library.

Fair enough; then maybe ask the library developer nicely if they could add vcpkg support. Many C++ libraries de-facto support CMake and vcpkg because these have reached critical mass over the past half-decade or so.

I'd say submodules are worse in every metric—you still have to maintain someone else's library (imagine a CVE patch comes through, for instance); submodules themselves are so fragile that they can break your own repo, requiring a full delete + re-clone, and it leads to 'I have this code, I can now make changes to it' which makes updating even harder.

> you still have to maintain someone else's library (imagine a CVE patch comes through, for instance)

I wouldn't call `git pull` maintaining a library.

> it leads to 'I have this code, I can now make changes to it

I would say this is a big advantage! Probably one of the few areas where submodules are better than e.g. crates/pypi. It's great for fixing bugs for example. You can fix bugs in Rust crates you use, by using a special override in Cargo.toml, and I assume pip/NPM support that somehow took, but it is more of a pain.

You're absolutely right that submodules are fragile. But they work well enough that people still use them.

> requiring a full delete + re-clone

I've got pretty close to that but actually I've been able to get out of every submodules breakage that I got into. Did require some magic commands I found in a mailing list somewhere though which isn't fun.

I've had zero problems with stability though if you avoid these two features:

* Worktrees, which sucks because worktrees are great

* git checkout --recurse-submodules (or submodule.recurse). This is just fundamentally broken and has been forever.

There are very good reasons for closed source projects in fact not to use public repositories but the very least org-internal forks/mirrors. In that case dependency management referring fixed public repositories is no-go. If there are better solutions than submodules for these I would love to know. Fetch-content might do the trick? (Specifically, a complex project composed only of private git repositories).

> If there are better solutions than submodules for these I would love to know.

https://learn.microsoft.com/en-gb/vcpkg/concepts/registries and

In essence:

- Publish your local/private mirror of the 3rd-party dep

- Edit the portfile for each dependency to point to your private repo, usually in `vcpkg_from_git`, `vcpkg_from_github`, `vcpkg_from_gitlab`, or `vcpkg_from_bitbucket`

- Publish these portfiles to your private vcpkg registry

- Set the registry in vcpkg-configuration.json (or the `vcpkg-configuration` field in vcpkg.json) to your private registry

Done.

Thank you!

I prefer ExternalProject_Add because that way I have full control of the build process. Adding even more abstractions leads to issues, edge cases and the like.

There's also https://github.com/cpm-cmake/CPM.cmake which is mostly a wrapper around FetchContent with good caching. I used vcpkg for https://github.com/zig-for/snfm - it made builds easy, but I'm on the fence if I'd use it for anything that is single platform. The thing that made it powerful was really good support for cross platform builds, but I could probably do the same thing with FetchContent and a few toolchain files.

Can I write a vcpkg.json that just refers to a git repository, tgz download link or similar decentralized identifier? (eg can I also use deps that have no special vcpkg support?)

You need to create an overlay port[1] for each dependency that isn't on vcpkg. So you need, at minimum, a `portfile.cmake` and a `vcpkg.json`[2] for each new dependency.

> just refers to a git repository, tgz download link or similar decentralized identifier

In the `portfile.cmake`, use the following functions correspondingly:

- vcpkg_from_git: https://learn.microsoft.com/en-gb/vcpkg/maintainers/function...

- vcpkg_download_distfile: https://learn.microsoft.com/en-gb/vcpkg/maintainers/function...

Each documentation link has a bunch of live examples near the bottom of the page.

[1]: https://learn.microsoft.com/en-gb/vcpkg/concepts/overlay-por...

[2]: https://learn.microsoft.com/en-us/vcpkg/get_started/get-star...

Reply on news.ycombinator.com