Microsoft Ubuntu repositories are broken because of space issues | Hacker News Reader

Microsoft Ubuntu repositories are broken because of space issues

github.com

191 pointsNiekvdMaas64 comments

Microsoft's dpkg packages themselves do some fairly weird/annoying stuff in their maintainer scripts.

e.g., azuredatastudio contains code that manages a /usr/local/bin/code symlink...

... and it contains code that converts a PGP key in armored form to binary form and dumps it in /etc/apt/trusted.gpg.d, even if the sysadmin already took the time to verify the PGP key and put it into their own .asc file in the same directory (ok, I guess at least they aren't force-appending their key to the old/deprecated/monolithic /etc/apt/trusted.gpg file like many others)...

... oh and in doing so they dump out microsoft.gpg to the current directory, whatever that may be... they should at least be using mktemp!

... and they are doing other things that they should be relying on debhelper to do for them, e.g. installing shared-mime-info-spec files manually rather than with dh_installmime; installing desktop-entry-spec files manually rather than relying on the triggers that already handle installation of such files...

As for teams, it has its own oddball way of monkeying with /etc/apt, and one other weirdness: it explicitly changes /usr/share/teams/chrome-sandbox to be setuid. If that file is supposed to be setuid then ship it as such... shipping it non-setuid and then modifing it in a maintainer script sets off alarm bells and breaks dpkg-statoverride...

I have zero trust on how they handle this, judging by your post it is justified.

This is why I decided to bite the bullet and go flatpak for the likes of teams, slack and other proprietary software.

> This is why I decided to bite the bullet and go flatpak for the likes of teams, slack and other proprietary software.

I always recommend this now if it is an option. Even if Microsoft does nothing wrong, it is way too easy for me to make an error and somehow enable Microsoft antivirus stuff on my Fedora machine which I still don't know how I managed to do.

Flatpak is best iff the vendor is producing the packages. I have found all too often than I'm stuck on an outdated version of a orphaned, unofficial pacakge...

Still I wonder if you'd author all non-proprietary packages how many of these violations you'd find. Especially in smaller projects. Don't have any examples ready since I also try to avoid anything but established software, but I'm certain in the past I've had issues like mentioned above, even persisting after removal of the package.

>explicitly changes /usr/share/teams/chrome-sandbox to be setuid

These kinds of hacks are pretty consistant with my few experiences with Microsoft (and other vendor's) closed software on Linux. It's all best avoided for community maintained solutions.

/etc/apt/trusted.gpg.d is being deprecated also in favour of signed-by=/usr/share/keyrings/keyfile.gpg

https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-...

Thanks for reminding me. I really wish apt-secure(8) and apt-key(8) would mention this!

The packages.microsoft.com repositories are regularly broken, which results in perennial build failures if you use GitHub Actions which includes it by default.

GitHub's response to "your default configuration is broken" was basically "welp, NOMFP":

https://github.com/actions/virtual-environments/issues/3410

Is there a way to easily change the default and if so what is a good mirror?

I have no need for MS packages, and so I simply

  for broken_apt_repository in `grep -lr microsoft /etc/apt/sources.list.d/`; do sudo rm $broken_apt_repository; done

before I do anything else. Others are not so lucky.

Uh somewhat off topic but is NOMFP a typo of NMFP (="not my fine problem")?

Hadn't seen it before and big G suggests NMFP is way more common, with no hit that I saw explaining the 'O'.

Is this another typo? NFMP = Not Fine My Problem?

That said, I’d also like to know.

I’d have guessed „none of my fucking problems“

Typo yes, that's what I get from HN:ing on my phone. Thanks, and fixed.

https://www.google.com/search?client=firefox-b-d&q=NOMFP

(It's a quote from the UK political satire series "The Thick Of It", in which the character Malcolm Tucker remarks, "NOMFP. N-O-M-F-P. Not My Fucking Problem. I quite like that. Did you like that? I'll use that quite a lot today.")

Aha, that was a couple of fathoms below my radar. Thanks!

At first glance I thought "Not Our MFing Problem." Thanks.

This seems quite plausible- moreso than using the first two letters of "Not."

I thought it was "not our motherfucking problem".

Not sure why but thought it was: not on my f*cking premises

You don't need to self-censor, we're all adults, we know what you mean, the US government won't deplatform you for typing an u.

Maybe “Not on my fucking plate”?

Oh, had this several times and removed MS repos in some projects. Maybe it worth to create the extension.

The packages.microsoft.com repositories are by far THE SLOWEST for `apt` updates. I wonder if there's a way to pick a faster mirror.

If you are on Ubuntu you could install from Canonical‘s snap repo hosted by Canonical and not MS.

I don't use snaps though...

Not only Ubuntu. And not for the first time [0], [1], [3] (Stack Overflow). Would be surprised, however, if the servers are in space instead of under water [2].

[0] https://social.msdn.microsoft.com/Forums/en-US/5d090bb8-6c22...

[1] https://github.com/MicrosoftDocs/windowsserverdocs/issues/50...

[2] https://news.microsoft.com/innovation-stories/project-natick...

[3] https://stackoverflow.com/questions/59624773/failed-to-fetch...

Just to clarify the title, Space in this context is the 'Biggest Key on your Keyboard' kind not the 'On Disk' kind.

It looks like a parsing issue where something in MS's automated process is replacing various parts of filenames with spaces. Looks like / and _ at a glance, but there could be more.

Just trying to prevent this type of comment, which I can only assume came from HN as nowhere in the Github issue does anyone mention disk space. -- https://github.com/dotnet/core/issues/6381#issuecomment-8631...

"nowhere in the Github issue does anyone mention disk space"

What is mentioned is ambiguous. So while disk space isn't specifically mentioned, neither is spaces in the context of filenames.

Here's what people are reading in that issue:

"Update: our infra team is still working to resolve this issue. They ran into some space issues but this issue should be resolved quickly. Unfortunately, I do not have an ETA yet."

There isn't enough context to know what it means.

"escaping space characters" would be clear.

I thought there was some diskspace issue

None commented yet on the suggested issue? They ran out of space? Really? Azure Monitoring, Azure Sentinel - surely they have the simplest metric: used vs remaining space.

Or did they hit a limit in the blob/file storage. Still: no monitoring?

You may have noticed I have so many questions on Microsoft Ops.

rm -rf --no-preserve-root is a space issue, too.

Reminds me of an F1 story. A car was able to finish the race but stopped before reaching the pit lane. The team communicated it as being a fuel pick up problem. The underlying cause was that there was no fuel left to be picked up.

> because of space issues

yeah, spaces in filenames. A well-known yet insidious problem.

Is there any mount option to change raw spaces in the exposed filesystem with an unicode non-breaking space, or something?

> yeah, spaces in filenames. A well-known yet insidious problem.

It'll be fixed as soon as they finish their Backslash Conversion Project!

Wanted to update my Teams on Ubuntu 20 in the morning (after coming to office after months working from home) and had the same issue. Removed the package and installed it from snap. Works like charm so far.

Of all the social issues in the world, apt-get and wget 404ing for my version of linux are the only things that trigger me

it's not just ubuntu, it's basically all of them. basically my debian docker build failed because of that.

Same. More or less all of my company's pipelines are currently broken

What SLA are Ubuntu mirrors held to? This doesn't seem great, but won't a quick run of mirrorselect remove the offending repos and let you get on with life?

For your home setup: sure, it should be resolved soon, not too big of an issue.

What's much more annoying is that this breaks CI/Docker builds. Right now you can't deploy new container images that depend on a MS repository. I hope it's resolved soon.

We had the same problems already in the 90s and resolved them by caching, it's unthinkable that the whole build is broken by a problem with an external dependence - if you assume everything will be 100% available you basically make sure a problem will happen.

https://wiki.debian.org/AptCacherNg still exists and still works fine.

Aptly [0] is also really nice for maintaining mirrors and one's own Apt repositories though it is unmaintained.

[0]: https://github.com/aptly-dev/aptly

Not caching... mirroring or vendoring.

I wrote "hashcache" [0] [1] for this for my Linux distribution (which is the build from source variety) to avoid issues with upstream going away, or worse changing the contents of a file.

Many different caches can be used, in case one is down as well.

Simple example download script at [2].

[0] http://hashcache.rkeene.org/ [1] https://chiselapp.com/user/rkeene/repository/hashcache/ [2] https://kitcreator.rkeene.org/fossil/artifact/d546c4b48dfce8...

As the other comment says: cache it yourself. Not only will you save time downloading it, avoid sudden version changes, your build environment should really not depend on pulling things out of your control every single time

The modern argument would be that you're more likely to have problems with your own cache than someone else, and you should just use caching as a service.

The modern argument is most likely made by people who sell caching as a service and have an interest in you blowing $500/month on a glorified FTP or a script to setup Artifactory.

But this is Microsoft's cache, so it is already essentially on the same network, and is free.

It's a lot easier to remove yourself from the critical path if your cache has problems than it is to add the caching after the fact when a third party is having problems, though.

Exactly this!

You plan for failure. What happens if your cache fails? You fall back to Microsoft. But what happens if Microsoft fails and you don't have a cache? Well you build a cache in advance and default to that.

Caches can also improve build time, allow for security tools to scan, and other such benefits too. So there are other benefits outside of creating a redundancy.

We do local caching, and it’s caused more outages, downtime, headaches and lost productivity than I can count. Although the hardware requirement is minimal, all that other stuff makes it expensive. So run a risk calculation for your own use case.

well we cached the final docker images ;-) unfortunatly if you need to add another dependency for our build image were done. (well we could pull the old one and add the dependency and repush this image)

also there are cache misses :-(

The “mirrors” in this case are Microsoft run servers specifically for Microsoft made packages for Ubuntu. Rather than the actual Ubuntu mirrors for all Ubuntu packages.

Those actually work pretty well from my experience and are usually faster than the default archive in docker containers.

I've also found the ':' character in filenames also breaks Linux under Windows :(

its probably a silly question, but why are people using microsoft to fetch debian packages?

I'd assume they host some of their own packages plus it's probably the default for the Azure space. (just an assumption)

Microsoft repos were silently added to Raspberry Pi OS earlier this year, ostensibly "in case anyone wants to use VSCode" and definitely not just as a super shitty way of adding telemetry to every Raspberry Pi OS install without asking the user if they were cool with this.

Microsoft's packages of its own software for Debian.

For example, my team's services run on Debian, but are .NET Core-based, so our dev boxes & CI machines rely on Microsoft deb repos.

had this last night :( thanks for posting

Reply on news.ycombinator.com