> I am surprised that mounting worked without error but I guess the device is still active via losetup.
Exactly. `rm` doesn't actually delete the file contents while the file is still open, it just unlinks it from the filesystem tree. So your loopback-mounted disk is still there and all its contents are still available through /dev/loopX.
> I'm assuming this would be similar to an actual disk failure though, if the device weren't there maybe btrfs will complain and ask to be mounted with the `-o degraded` flag.
If the /dev/loopX device wasn't there then it would be similar to a complete disk failure, yes.
> In this test about 26% of data is still fully readable
It's true that only 26% of data is still fully readable if you account only for files that are fully intact. But also note that about 78% of files were still completely intact.
This is not clear from your comment, but I'm assuming that you are using 4 devices for the btrfs pool as well?
In this scenario, with such a disk configuration and subsequent disk failure you would expect to lose about 25% of files, while the remaining 75% would be intact (especially if the files are small enough)...
But actually, in reality things can be quite better or quite worse, depending on a few factors.
For example:
1. If the free space was fragmented. In such a case, a significant percentage of files might actually be allocated on more than one disk, so you'd lose more files than expected if a single disk fails. Although I can see that on your latter experiment, you've defragged the btrfs filesystem beforehand, so perhaps this is not the main issue.
2. Depending on how btrfs allocates data, if the files are not completely filling all of the disks then they can be heavily skewed towards a subset of the disks.
For example, imagine that each of your disks are 1 TB-sized and your files total less than 1 TB.
In this case, all of your files could be allocated on the first disk only, so losing this disk could lead to losing 100% of your data.
Or for example, if your files are less than 2 TB, they might all be allocated on the first 2 disks only, so losing one of these disks would lead to losing a lot more files than you'd expect if files were evenly distributed across all disks.
But on the other hand, if you'd lose one of the other disks, you might not lose any data whatsoever.
3. Depending on how large files are and how much free space there is on each disk, btrfs might be forced to (or might choose to) span a file across more than 1 disk even on the 'single' profile, even if free space was not fragmented.
4. But of course, more generally, how many files you would lose basically depends on how btrfs allocates disk space across the disks for each file.
These disk space allocation algorithms can be quite more complex than you'd expect from a naive allocator, mostly due to performance reasons.
Unfortunately, I know exactly nothing about how btrfs allocates data, so I can't give you more insight than this, sorry!