unqueued

> Most large brands throw the UI design patterns of the platforms out the window in favor of their own UI guidelines.

I really wish they would not do that.

The best thing you can do to make your app trustworthy and friendly is to adhere to the host operating system UI guidelines and expectations.

Nobody wants your unique take on the checkbox or textarea please.

What problem are you trying to solve exactly?

I think submodules are good enough for what they do and there's room in Git's model for forward development and it's also flexible enough that third party tools can piggyback on it. Like Git in general it's a bit of a pain till you practice with it more.

Git can do some amazing stuff, I use it for data as well as code.

This isn't really like subtree it's like submodules, but for specific files. This could really be useful, you may want to keep track of a specific file in a repo but not need it as a submodule.

There's a whole umbrella of similar-ish utilities. The one that I think does almost the same thing is: https://github.com/mullvad/git-carbon

Submodules aren't perfect and can be really annoying. I tried to avoid them at first.

But if you work with larger projects I think you'll see how they are useful.

For example, some projects might commit their node_modules or composer vendor folder as submodules so that they have an exact snapshot of their dependencies that can be quickly cloned in place and referenced without mixing it with the parent repos source.

Submodules are certainly not the only way to do things with git. Sometimes monorepos or something else makes more sense.

But if you care about maintainability then what are you supposed to do when the majority of your commits are unrelated to your project because you're using git subtree or something similar? Or the majority of your files are also not directly relevant to your project but need to be there?

What about stuff like optional tests or maintenance scripts or even binary dependencies that you want to have closely associated with your project but don't want to clone every time?

My thinking is that the porcelain is incomplete but very open. The way submodules are represented can be added on to.

And the plumbing is consistent enough that you easily script around it.

You should check out the Datalad project to see some really clever usage of submodules.

And the url key can be annoying; I usually try to use relative paths in my .gitmodules folder when possible. You're actually free to use remote helpers and relative paths to get around hard a github url into it.

You can also embed and read arbitrary keys from your .gitmodules file with git-config -f which is really useful. You don't even have to use a .gitmodules file, in theory.

FYI A submodule doesn't have to use a gitfile and a corresponding `$GIT_DIR/modules/<name>` and there are good reasons not to. As long as the submodule has a '.git' it can be a symlink, or regular .git directory for a self contained embedded repo. You can still use the same gitlink in the parent repo representing it's commit id and git will still manage it.

If I have a 20GB submodule I'd usually just do a `git clone <url> <path> && git submodule add <path>` and it will be treated the same by git. But now I can just delete it and it's purged. And it is more portable less fragile in some ways because it's not de-referencing a gitfile. I prefer my repos to be more bottom-heavy and to not clog my modules folder.

I have a rough script that's the inverse of `git submodule absorbgitdirs` but it's a bit fragile.

It would be cool if there was some plumbing to expose this a bit more.

I appreciate that it uses git-notes and that is makes backup branches in it's own namespace. I wish it were a bit lighter though.

Haven't seen mentioned, but you might want to check out https://github.com/mystor/git-revise

The few times I've wanted to completely rearrange a repo, It's been enough just make a backup branch and then dump the log from there and pull what I want.

I have script somewhere I think it's called git-cherry-replace, which soft resets a commit id and copies the commit message and optionally populates GIT_COMMITTER_DATE and whatnot.

What I described is probably simple enough that an LLM can implement it.

But unless you're trying to retroactively reconstruct something to preserve the exact meaning, you should consider whether a rebase even makes sense.

I used bluez and bash to lock xscreensaver using some very minimal bash. It wasn't my idea I believe people on the Gentoo forums were doing it.

But you can just loop over something like `hcitool rssi "$MAC"` and project it somewhere, there's a variety of ways.

I like using dunstify with the -p option to persist on screen.

I think it's really impressive what these agents can do, but you should also consider whether you're asking it to burn tokens reinventing the wheel for you, or making a pretty wrapper around a wrapper.

> The alternative is that people complain "netflix doesn't work on firefox", switch to chrome instead, which is even worse.

So what if users complain? How is it better for Firefox to do something bad just because Google is doing something bad?

Firefox is supposed to provide an alternative to what's out there. Firefox also didn't support some popular proprietary Internet Explorer features, and they never attempted to. For a time, much of MySpace didn't work as well in Firefox. But I'm glad that Firefox didn't cave, even if some users complained that they couldn't make the scrollbars neon green or make music autoplay.

Not letting Microsoft or Google dictate how they implement a web browser worked out really well for them. Chasing proprietary platforms has ruined them.

At the time EME was adopted Firefox was much more popular, I think 20% back in 2012. Video platforms were using Silverlight for DRM. There's a good chance that EME would not have gotten off the ground if Mozilla didn't embrace it, or at least not as quickly.

Mozilla should have taken a stand and refused to support EME when they had the chance. They would be better off than they are now. And there's a good chance Netflix would not have thrown away a double digit percentage of subscribers.

Instead they embraced DRM and now they have nothing.

I wish more were done to push back against this consolidation of power by these platforms.

We're like frogs that have been stoking the fuel of our own pot.

It used to be that DRM was considered to be in conflict with the browser, because it was not acting on behalf of the user. If you must have DRM, then it is on the platform to shoehorn it in through an external plugin, like Silverlight.

When Firefox adopted EME extensions, I knew it was the beginning of the end; they were rolling out the red carpet for DRM. If we make DRM a switch that can simply be thrown, then it will become the norm, not the exception. And there have been proposals for years to DRM fonts and other absurdities. If a company insists on using DRM, then they should have to shoulder the burden of doing something that a browser was never support to support.

The nightmare that we're racing toward is you will only be permitted to cache a trickle of video at a time and your TPM attestation hardware must include a token in every HTTP request. Your browser will just be a software cablebox.

They aren't happy about URLs either and would love to require that if you want to share a reference to something, you have to do it on their terms, like generating a url in their app with a hash that expires and limited in how many times it can be viewed. I'm sure influencers will still have the privilege of unlimited sharing.

They've been slowly rolling this infrastructure out for the last decade. These are not isolated inconveniences, these are coffin nails.

I think symlinks are really underappreciated for organizing files.

One cool thing you can do is to treat directories as tags when you symlink into them, since a file can 'belong' to multiple directories.

If you're using Windows, you can just use WSL and then you don't need administrative access as the OP's program does.

Here are some simple tools to let you use symlinks as tags:

https://github.com/perses76/refmat-symlink

https://mikeknowl.es/projects/ztag/

https://github.com/michaelknowles/ztag

But if anyone is curious about ways to better organize their files in general, I would check the writings of Karl Voit:

https://archive.baty.net/2018/using-karl-voits-file-naming-s...

https://karl-voit.at/2022/01/29/How-to-Use-Tags/

Simple and portable self describing filenames has served me really well for a long time across many platforms.

This script has done a lot of lifting for me and it will work with WSL with little effort:

https://github.com/mdom/squaretag

If you are iterating over a lot of files, a read while loop can be a major bottleneck. As long as you use the null options from find and pipe into xargs, you should be safe with any filename.

I've found it can reduce minutes down to seconds for large operations.

If you have to process a large number of files, you can let xargs minimize the number of times a program is run, instead of running it once per file.

Something like:

  # Set the setgid bit for owner and group of all folders
  find . -type d -print0 | xargs -0 chmod g+s

  # Make the targets of symlinks immutable
  find . -type l -print 0 | xargs -0 readlink -z | xargs -0 chattr +i

Way faster. But there are lots of caveats. Make sure your programs support it. Maybe read the xargs man page.

There is a pretty good syntax for dealing with nasty filenames, if you must: ANSI-C quoting[1].

If you have to output in a shellscript in this format, use printf %q

from man printf:

       %q     ARGUMENT is printed in a format that can be reused as shell input, escaping non-printable
              characters with the proposed POSIX $'' syntax.

It is just $'<nasty ansi-c escaped chars>'

$ touch $'\nHello\tWorld\n' $ ls

One thing I do like about a filesystem that fully supports POSIX filenames is that at the end of the day a filesystem is supposed to represent data. I think it is totally sensible to exclude certain characters, but that it should be done higher up in the stack if possible. Or have a flag that is set at mount time. Perhaps even by subvolume/dataset.

One thing I haven't seen mentioned is that POSIX filenames are so permissive that they allow you to have bytes as filenames that are invalid UTF-8. That's why the popular ncdu[2] program does NOT use json as it's file format, although most think it does. It's actually json but with raw POSIX bytes in filename fields, which is outside of the official json spec. That does not stop folks from using json tools to parse ncdu output though.

Another standard that is also very permissive with filenames is git. When I started exploring new ways to encode data into a git repo, it was only natural that I encountered issues with limitations of filesystems that I would check out in.

Try cloning this repo, and see if you are able to check it out: https://github.com/benibela/nasty-files

It is amazing how many things it breaks.

If you are writing software that deals with git filenames or POSIX filenames (that includes things like parsing a zip file footer), you can not rely on your standard json encoding function, because the input may contain invalid utf-8. So you may need to do extra encoding/filtering.

[1]: https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoti...

[2]: https://dev.yorhel.nl/ncdu/jsonfmt

Both Finder and explorer.exe will mung mtimes from casual browsing, usually due to metadata updates. It is a common enough problem that there are utilities to bind mtimes to files.

IMO mtime the de-facto file timetsamp metadata, since it is most widely supported. The other file timestamp metadata I found were not as useful or portable.

And people use git for storing more than just source code.

I personally find it very useful to be able to clone my podcast index and have accurate timestamps not just from the commit, but from the file itself.

Timestamps can be important for a collections of files, but they are fragile and often an afterthought.

Sometimes search indexing and build tools get it wrong, so if you know what you're doing, then tools like this allow you to provide an override.

Just browsing an old photo collection without making any changes can cause a 15 year old file to show up as a "new file", even though it's contents have not changed. This is especially true with media files that the os might take it upon itself to regenerate metadata for.

Or you may wish to store an accurate snapshot of the timestamps of cached or intermediate build artifacts.

Thanks for sharing, I love stuff like this. I get frustrated at mtimes being munged. MacOS finder is terrible about this.

If anyone is interested, I wrote a more generalized tool for storing metadata called storetouch[1] which takes a snapshot of file's mtimes, and stores them in a nicely formatted file for convenient restoration. You can run it's output by itself with bash, or easily parse it's output.

I found that the commit or author date's did not always make sense to represent the true age of a file. Especially for non-source-code documents. For example, a repo of scanned documents. Or a git-annex repo of podcasts going back 20 years.

Instead, I treat the commit date as the commit date, and I have a versioned sidecar that stores the actual original mtimes of the files.

It can also handle all valid POSIX filenames (a surprising amount of utilities don't, including tools like git-restore-mtime, last time I checked). If you're already using a tool like git-annex or datalad though, they have the ability to store metadata for each file as well.

I would also suggest checking out git-store-meta[2] which was the original inspiration. It stores more than just mtime, but it is more closely tied to git.

[1]: https://github.com/unqueued/storetouch

[2]: https://github.com/danny0838/git-store-meta

You can already use it with git-annex to store binaries using the git-annex-remote-git-aafs[1] special remote.

Although I would be careful and make sure you understand what it is doing to your branch namespace. Even though in the worst case it would not save any space over directly committing binaries, they are in orphan branches that can be pruned without rewriting history.

But even so, you can just use any number of git-annex special remotes to bypass using git for sharing files.

They may eventually add first-party support for git-annex. But nothing is stopping you from using it now.

[1]: https://github.com/domo141/git-annex-remote-git-aafs

I think "confabulation" is a way more accurate term, I wish it had stuck instead of "hallucination".

A hallucination is a problem with input. Confabulation is false output.

Confabulation is when a person mistakenly recalls details and tries to "fill in the blanks", without realizing what they are saying is untrue.

That's what's great about btrfs. I started using it that way too. It was nice to be able to take instant snapshots of my fs. I just amended my existing backup scripts to act on temporary snapshots instead. Then, I started dividing things up into subvolumes.

I suggest moving your filesystem root to its own subvolume (the convention is to call it @root), then optionally mounting the actual filesystem root as like /mnt/fsroot, and mounting it by passing subvol=/. That is where I keep my snapshots. You can still have /home be nested in @root. I have subvolumes for /var/cache, /var/lib/docker, etc. And more fragmented qemu images just go on a NoCOW subvolume with +C set, and I back up normally.

I also highly suggest reflinks. Once you start using them you will use them all the time.

I don't know if zfs supports reflinks yet, at the time it didn't. But combining reflinks with subvolumes solves a lot of problems. Since btrfs doesn't support recursive snapshotting, you can just do cp --reflink=always -rp @root /path/to/new-subvolume, and it will combine the contents of the subvolumes without consuming any more space. And of course, btrfs-send will (usually) not duplicate reflink data.

I will probably give bcachefs on one of my machines.

I started using btrfs back when it was way less stable, but that instability was more than offset by the other features.

The last time I lost a btrfs filesystem was four years ago when it was less stable, but it did not matter because it only amounted to about an hour of logs lost, and I reconstructed it in minutes. So btrfs is a must on any embedded devices. It frees me to use cheaper media, because I can have two copies of all data and metadata, and compression, and fine grained external snapshots.

Zfs is great for making big complex arrays and if you plan out how your are storing your data ahead of time. But it has a steeper learning curve and is more seperate from the underlying OS.

Btrfs fills this perfect niche for me of being a drop in replacement for ext4, and being really flexible and offering some of the nicer features of zfs. (differential snapshots, checksumming, compression)

So I'm excited to try bachefs as an alternative to btrfs.

This is an example of how btrfs is poorly handled by many userland tools by default. You should never ever need to have thousands of snapshots.

I use btrbk to manage my snapshots. It keeps my snapshots pruned to 300, pruning intermediate snapshots, so I can still easily step back 52 weeks, or I can purge them if I want to improve performance and reclaim space. While at the same time, every single snapshot is sent to two compressed encrypted backup images over ssh. The diff algorithm is pretty good, but for some types of files that fragment frequently, it is best to put them on their own subvolumes. My places.sqlite files from my browser testing profiles were adding tens or hundreds of MiB to snapshots, so they went in a seperate subvolume. I don't think the zfs would perform any better.

I don't blame people for having bad experiences with btrfs, it is very inconsistently documented.

I think it is especially annoying how many userland tools keep every single snapshot as a subvolume of the root, so that you have tens of millions of read-only files stored in /.snapshots. Awful.

I actually have something I've been meaning to put on github. I wrote it over a decade ago and I need to dockerize it. I only made it in Vagrant.

But it can do something iVentoy can not.

https://www.iventoy.com/en/faq.html:

   The computer which run iVentoy must be in the same LAN with them.
   Besides, there must no other DHCP servers in the same LAN.

It can coexist with an existing DHCP server. That means you can just start it up on your own network without any modifications, and start netbooting machines.

It turns out that multiple DHCP servers can co-exist on the same network segment. So I just have a second dhcp server solely for a pxe bootstrapper. At the time, at least the initial download must be tftp, but I used a stripped down ipxe stub that took advantage of the universal undi network drivers, and then loaded the rest (usually syslinux) over a fast HTTPS connection.

From there, iSCSI was no problem, and I could also use tools like plop, memdisk, or grub4dos, as well as Windows 7/8/10 with wimboot.

I remember for some distros like Knoppix, we would use an NBI driver to just mount the iso directly over the network, which could be far simpler and more performant than NFS.

EDIT: To be honest, it is simple enough that it doesn't have to use Docker. It is mostly dnsmasq and PHP.

My biggest mistake after being laid off was to not be discerning enough about my next job. You need to be willing to say no to a crummy job offer, even as the bills pile up.

John, at around that time you mentioned, you gifted me with my first shell account on one of your coffeshop servers, Reva. I just wanted to say THANK YOU. Up until then, I had read books about unix from the library, and had failed to install Mandrake on my laptop, but being able to access such a well configured NetBSD environment was really helpful and set me on a good path.

I had never been truant from school before up to that point, but sometimes life rewards you for taking risks.

I was hoping that using hardware keys would eliminate some of the security hoops that we have to jump through. And it does seem to help. But the whole reason that I have a key is so I do not have to supply my phone number, and I have a more trusted way of proving my identity, even if I am connecting from an unusual location.

This makes me worried, because I am pretty sure Google is going to start removing keys based on attestation certificates.

I believe that this is much more about rate limiting than about security for the end users.

No, that is not correct, git-annex uses a variety of special remotes[2], some of which support deduplication. Mentioned in another comment[1]

When you have checked something out and fetched it, then it consumes space on disk, but that is true with git-lfs, and most other tools like it. It does NOT consume any space in any git object files.

I regularly use a git-annex repo that contains about 60G of files, which I can use with github or any git host, and uses about 6G in its annex, and 1M in the actual git repo itself. I chain git-annex to an internal .bup repo, so I can keep track of the location, and benefit from dedup.

I honestly have not found anything that comes close to the versatility of git-annex.

[1]: https://news.ycombinator.com/item?id=33976418

[2]: https://git-annex.branchable.com/special_remotes/

I have a 1.96 TB git repo: https://github.com/unqueued/repo.macintoshgarden.org-fileset (It is a mirror of a Macintosh abandoneware site)

  git annex info .

Of course, it uses pointer files for the binary blobs that are not going to change much anyway.

And the datalad project has neuro imaging repos that are tens of TB in size.

Consider whether you actually need to track differences in all of your files. Honestly git-annex is one of the most powerful tools I have ever used. You can use git for tracking changes in text, but use a different system for tracking binaries.

I love how satisfying it is to be able to store the index for hundreds of gigs of files on a floppy disk if I wanted.

For bigger annex repos with lots of pointer files, I just disable the git-annex smudge filters. Consider whether smudge filters are requirement, or a convenience. The smudge filter interface does not scale that well at all.

I combine git-annex with the bup special remote[1], which lets me still externalize big files, while benefiting from block level deduplication. Or depending on your needs, you can just use a tool like bup[2] or borg directly. Bup actually uses the git pack file format and git metadata.

I actually wrote a script which I'm happy to share, that makes this much easier, and even lets you mount your bup repo over .git/annex/objects for direct access.

[1]: https://git-annex.branchable.com/walkthrough/using_bup/

[2]: https://github.com/bup/bup