How big of a NAS do you have?
That said, I have 24x16TB hard drives in ZFS RAID. It's three separate RAIDZ2's, so the amount of space on there is 18x16TB, so about 288TB. It's a fairly expensive amount of waste, but it's nice to be able to lose up to six disks at a time without having to worry. I could fairly easily get another 8 drives in there if I really needed it, but thus far my total consumption is only about 50TB in total, and I delete actually stuff when I'm done with it.
that is a lot.
maybe it (and this comment) will be subsumed in the next 10 years, but for now it is spacious and amazing.
One advantage, though, is that when you have a "nearly-unlimited" place to dump data, it kind of changes how you think about problems. If I want to do some experiment with video encoding, there's really nothing stopping me from encoding a video into 100 different configurations so I can directly compare stuff. I don't have to be sparing, and there at least won't be any surprise bills from Amazon or Google later.
They're spinning drives so I don't really worry about the write-cycle stuff.
I should point out, part of the reason I do this is as a learning tool. I feel like you learn a lot of neat tricks when you build and run this stuff on your own, and I think that there's really no negatives in the software engineering world to being proficient with the command line and server administration.
Specifically:
- What hardware caters to 32x disks (24+8)? (I'm picturing enterprise gear)
- What software are you using to coordinate it? TrueNAS?
It's a fairly typical 1U HP Proliant server, which counts as "enterprise gear". I don't remember the model but it's somewhat unremarkable. It only has like six bays in the front, and I use one for a NixOS install, and two of them with SSDs for read and write cache. It has an old 10 gigabit ethernet card for connectivity. The server I bought used for around $150, the ethernet card I got for about $35 used on ebay.
The actual disks live in 8 bay external chassis, and connected via USB3.0. This may sound horrifying, but USB3 is rated for like 5 gigabits, way more than you can easily get from a home RAID with spinning disks. I have a PCIe 3.0 x16 USB3 card in there, with four plugs, each with a dedicated controller, meaning that in theory each port could get 5 gigabits. The reason I could fairly easily get eight more in there is that I'm only using three of the four ports, so without much effort I could buy 8 more drives (of any size I suppose), put them into another 8 bay chassis, and just add vdev to my RAID. This particular PCIe USB card was $41 on eBay.
I use ZFS for a software RAID.
This sort of happened accidentally; I didn't used to have a rack mount server, I was using a bunch of Nvidia Jetson Nanos, and as such there wasn't a real way directly plug hard drives in, so I ended up having to use the external chassis. Eventually I bought three of those cases, had all the drives in there, and when I decided to buy a "real" server it was substantially easier and cheaper to just add proper USB support to a server than it was to find a decent 24-bay server at a decent price, particularly since I already had the cases.
In hindsight I probably should have got chassis that have eSATA support, but they've been getting decent speed and it's an extremely convenient system. It's also come in handy once when I accidentally broke my server, and I needed to get files off the RAID; all I had to do was plug them into my laptop and mount the ZFS RAID there.
ETA:
Forgot to answer what I use to manage it. I installed NixOS on my server with a pretty vanilla ZFS on Linux install and an NFS mount. I do the tmpfs-on-root trick, so the root partition gets nuked on every reboot. https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/
I don't use any fancy GUI tools or anything; NixOS does a good enough automating away the un-fun parts of maintaining a server, and if I break my server I can very easily roll back to a previous state, and if I really have to, reinstalling it from scratch takes like 20 minutes since I can just copy over my configuration file (which I back up on Gitlab) and have everything automatically set up again. NixOS is pretty cool.
ETAA:
According to `dmidecode` and `lscpu`, server model is `ProLiant DL380 G7`, 24 cores, 128GB of RAM. I think it's actually 12 cores, and it says 24 because of hyperthreading.
It's pretty unremarkable. I like it a lot, but it's a fairly typical and bog-standard used server.
Can you talk more about this? I personally have a Dell R720 and about 90TB available for use, and use 40TB of it for my media server. I'm wondering if that's some use case I want to look into.
I wanted to play with paper trading strategies. I listen on websocket streams of cryptocurrency and stock ticker stuff [1], and the service listening on that socket just plops it into Kafka, with the partition key being the ticker name.
The reason I bother adding Kafka largely comes down to the ability to use the Kafka Streams API. Kafka Streams allows me to do time windows of different trades, or lets me do a sql-style join across two different streams, or lets me filter out data that I don't think is relevant.
Kafka Streams gives you most of the fun of a map-reduce framework, but without having to administer a map-reduce server; it's even smart enough to handle internal state by creating intermediate topics and/or creating local RocksDB instances. It's pretty cool.
There's also has the advantage of allowing me to set retention, so I don't have to worry about doing any kind of manual cleanup; I only have to worry about having N days of stuff on the server, and it's trivial to change that.
Also, I have a number of topics, each with 32 partitions, meaning that if I do find any kind of strategy that works, I can very easily scale it up without changing any code.
I'm basically using Kafka as a streaming-only database. I think it's neat but I haven't actually found any strategies that make or lose money. It's just been a fun way to play around with different bits of server stuff.
[1] https://docs.alpaca.markets/docs/real-time-stock-pricing-dat...
I can read about all the fun theory behind trading math and for distributed systems, and that has some value, but I will understand stuff a lot better if I give myself a real project to do stuff with. I guess I am more of an engineer than a mathematician.
However, if you want to rip 4K blurays, you've got to flash the firmware on your bluray drive and then run MakeMkV + handbrake (or much harder CLI stuff) to process a disc into a useable media file which can THEN go into Plex. All of that takes time and effort, and usually some trial and error.
I also keep a bluray player next to the NAS. Simply because it's too much of a pain to deal with ripping sometimes. I still have to get through my 30-disc Ingmar Bergman Criterion Boxset that I bought 2 years ago. Much easier to pop a disc in the player!
Honestly, I've been happy enough with 1080p. I think 1080p blu-rays still hold up pretty well visually. My biggest TV is 70", and I watch it from like 15 feet away, so I don't think I'd benefit much from the increased resolution.
I know that there's other variables that are enhanced with a 4k blu-ray, but they haven't been significant enough for me to bother collecting/ripping them. When I buy a 4k blu-ray, I've just been ripping the regular 1080p blu-ray that always comes with it.
I’ve heard that some setups struggle with higher resolutions and bitrates, which is compounded on 4K (HDR also with its own quirks for downsampling in post processing for playback on unsupported devices) versus 1080p. These factors are exasperated when direct play isn’t possible due to incompatible files, unsupported container formats and/or codecs, or insufficient bandwidth, especially when remote mounts, caching, and transcoding come into play.
Apparently as of recently at [CCC 37C3: Unlocked] (December 2023) Adam Batori demonstrated a side channel attack against Intel SGX and AACSv2 in his talk called "Full AACSess: Exposing and exploiting AACSv2 UHD DRM for your viewing pleasure" although that has nothing to do with the specific drive used to my knowledge, though the specific Intel CPU used may have quirks which may make their particular implementation CPU and/or CPU microcode and/or motherboard dependent.
Previously on HN:
Full AACSess: Exposing and exploiting AACSv2 UHD DRM for your viewing pleasure [video] 6 points by hcadam | 2023-12-29 | 1 comment
https://news.ycombinator.com/item?id=38810011
https://media.ccc.de/v/37c3-12296-full_aacsess_exposing_and_...
Video of the live demo above. Additionally, I have included a version of video on from the official media.ccc.de YouTube channel:
https://www.youtube.com/watch?v=SEBuiecLZGg
Referenced by the above talk is prior work on defeating/bypassing AACS[v1] by Peter Eckersley in his talk titled "A Spotter's Guide to AACS Keys" at [CCC 24C3: Volldampf voraus!] in December 2007, which I also link below:
https://media.ccc.de/v/24c3-2331-en-spotter_guide_to_aacs_ke...
Also of interest from [CCC 36C3: Resource Exhaustion] is a talk by oct0xor titled "Hacking Sony PlayStation Blu-ray Drives" in December 2019:
https://media.ccc.de/v/36c3-10567-hacking_sony_playstation_b...
https://www.youtube.com/watch?v=WW39dsbffMw
Apologies for lateness of my reply! I hope that helps.
i.e. one doesn't need "crazy" (i.e. more than a handful of consumer hard drives) amount of storage to store a lot of bluray movies and tv shows and keep them online available to you.
That said, since I have an elaborate tape backup system now, I've debated keeping the "masters" as archives on the tapes, and keeping compressed versions for streaming, but laziness has kind of won out on that.
I've entertained setting up a tape drive local backup for my NAS (4x 14 TB) for a few years. Is it worthwhile from your experience?
I have two LTO-6 tape drives. They only connect via SAS, which is a little annoying, but I bought a 2-port used SAS card on ebay for $20. I ended up buying a pack of like 100 LTO-6 tapes on eBay two years ago for a steal, like $150 in total. Each tape is 2.5TB, so I can pretty easily back up my movies. Each movies is written to two separate tapes. I keep a spreadsheet of which tape every movie lives, in case I need to restore a specific backup.
I would really like to get an LTO-7 or LTO-8 drive with Thunderbolt, but they cost an arm and a leg even used on eBay right now. If you get lucky, an external SAS LTO-6 drive can be had for about $200-$300.
It's a slightly expensive up-front cost I'll admit, however, it came after I was using Google archive storage, and when I did a restore of 21TB of movies, the next day I had a bill for like $700!
It's saved my ass one time thus far; I accidentally broke my ZFS RAID and lost everything, but fortunately I was able to restore all my blu-ray rips from tape, and it was pretty easy, and more importantly, didn't cost me anything extra.
If you decide to go down this road, I do recommend getting one that has support for LTFS; otherwise you're going to be stuck using tar commands I think.
4k, something like 25GB-45GB, depending, but I have several in the teens of GB that I wouldn’t guess weren’t full quality.
I reserve full-quality rips for movies I both care about a lot and that benefit from it. I don’t need more than 4.5GB dedicated to Happy Gilmore or what have you.
You could just buy DVD's if you want to save space and are willing to take a hit to image quality.
Let's imagine an 8TB drive costs $100 and can store 200 movies as their raw bluray encoding. that means each movie costs 50 cents to store without any real work. If one can divide the space needed by half, one only saved 25c.
But how much did it cost to save that 25c.
Lets imagine one can encode a movie with x264/5, but it will consume 100watts of cpu power over 10 hours to do that (or a kilowatt total). If one is paying 20c a killowat hour (US national average is 19c a kilowatt hour, california is 33c per google) one just paid 20c more to encode it, so if one cut the size in half, one in practice only saved 5c. If one spent 5-10 minutes setting up the encode, did one spend 5c of one's time doing that? Personally, the setup for me was also 25c worth of time.
Now, one can "save more", if one is storing duplicate copies. (i.e. I'm storing it on 2 or more hard drives, not 1), but its still not really that significant.
Now, if one isn't spending one's own time and energy to encode it (i.e. is pirating it as others did the work), then perhaps one can argue that a "good" encode vs the raw is better and makes more efficient usage of one's space. But the argument here isn't about pirating, but about storing one's own legally obtained media.
As I've argued elsewhere, the only reasonable reason I've found for encoding is that one runs into "physical" space limitations (i.e. how many hard drives one can keep online). I'm just unconvinced that should override the time and cost of doing the encoding oneself.