My home lab setup for highly-available Internet
github.com
github.com
I do have a UPS on the modems and main access point.. but after reading this post, I may invest in diesel generator and a 5,000 gallon subterranean tank.
Their IPv6 situation was even worse. They used 6rd and I swear, the translation box was probably a single router or Linux box with a 100 Mbit NIC in a rack somewhere. If you bothered enabling 6rd, every v6 site would be awfully slow. Even the browser projects to automate the selection of v6/v4 didn't help.
When I finally moved away and cancelled the service, I mailed my modem back as directed. A few months later, they sent my account to a collections agency over the cost of a modem, which their system claimed to have not received. I spent hours on endless phone calls but ended up just paying them the $250 or whatever to save my credit and stop the madness.
Seriously, they were the worst provider I ever had.
In my experience, nothing happens until you actually make contact. Legal isn’t usually pleasant. But they’re almost always competent.
We all knew that 95% of all legal mumbo jumbo threats where all bark and no bite, and the other 10% was someone else's problem. So threaten away and know that you're making reps days.
They still come by to sell me fiber a few times a year, and I explain I would try again if they "forgive" the money I owe, the salesperson says "no problem", they spend 20 minutes on the phone to HQ, then say it's impossible :)
Oh yeah, and I had the exact same oversubscription issue. The service was fast as hell, until the middleschoolers got home at 2:30ish, then it slowed to almost unusable speeds.
(Hm. Bandwidth costs might throw that idea out the window, but it might be absolutely worth it for gaming - or maybe you could sign up for game server v6 alpha/beta testing, heh)
Oh well, let's keep gifting them billions in CAF funding to build out more poorly-maintained private networks and allow more monopoly mergers with other tier 1 providers. I'm sure that'll help and not cause decades of stagnation.
[1]: https://www.dslreports.com/forum/r31539668-Awful-YouTube-Con...
Reliability wise, I've had 1 outage in two years, which happened at 1am on a weekday for slightly under 2 hours due to a PPPoE Aggregator failing. Far better than Comcast or Wave ever were, and at our usage I don't think either of the cable providers would be viable. We hit 12TB last month iirc, not even a peep from Centurylink.
Speaking of UDP Multicast, PFSense dropped all support for it without warning before updating, which was the end of me using PFSense at home as it broke TV until I could replace it with OpenWRT.
I highly recommend propane-fueled gensets. Fuel storage is much less hassle, and propane doesn't go bad. You won't get the runtime of a huge diesel tank. But there's often little point in that, because an extended power outage will also take down the telecom infrastructure. As I recall, a ~7kW genset at ~70% capacity went through ~40kg propane per day. That was running well pump, sump pump, refrigerator, CFLs, microwave, fans, and a ~3kW UPS for several computers.
Edit: Make sure to get a UPS that accepts genset power. That usually means full online aka double conversion. And everything must be grounded properly. Plus at least a manual transfer switch, to avoid injuring utility workers.
IIRC that's how at least one wireless ISP measures hotspot data different than from-phone data.
so what, increment by 1?
https://www.amazon.com/Omnicharge-Portable-Power-Bank-connec...
Any resources, books, links, youtubes especially, that you can point me to?
Also, in his set up, theres no router? He says hes using a VM? What does that mean?
Go to your router's config page, and google all the words/acronyms you don't know. Read the Wikipedia pages too. That should put you in a position where you can ask better questions.
Router is a generic term for something that takes wired Ethernet and performs NAT and creates a wifi network( something an access point aka AP does). The NAT here is handled by a different device(the by the 'VM', which is a program that runs an entire OS by simulating a computer) and the AP here is multiple UniFi devices[1].
I would say get familiar with googling technical terms, because there won't always be someone willing to answer questions
not to nitpick, but this not actually true. for home use, perhaps it is conflated to mean this, but really it is one machine that reroutes traffic on behalf of another - NAT/wifi or other media transformation is not necessarily required here (though definitely could be a part of it)
Most people are familiar with home routers which do Network Address Translation and have a built in Wireless access point. Neither of those things are required for something to be a router. In fact routers with IPv6 support do not perform NAT between your local network and the Internet for IPv6.
Similarly, DHCP, DNS, and various other things that home routers do can actually be handled by totally separate hosts on the network. That's what he's doing.
If you're looking for more information about how networking works, I would highly recommend Computer Networks by Andrew Tanenbaum. It's more abstract than the typical "Understand TCP/IP in 600 pages" books that are available but it provides a good high-level overview of how networking works, what protocols matter, and how everything fits together.
Edit: When he says he's using VMs, he means that he's using Virtual Machines to run multiple operating systems on the same server. Each of these operating systems runs one or more servers for DHCP or DNS or other networking services. I assume that he's using his virtualization platform to mirror the virtual machines between his servers and provide hot spares, so that if one VM goes down another spare can step in.
Download this: https://www.pfsense.org/download/
Have a read of this: https://www.netgate.com/docs/pfsense/
Hang out here: https://forum.netgate.com/
The VM is a router. Provided you ensure that traffic has to go through the VM and that the VM is able to route etc then it is a router 8) OP is using Linux whereas pfSense is FreeBSD based but pfSense is pretty much the only (near enough) turnkey product that does multi WAN and CARP properly. I should mention OPNsense as well here for fairness.
I believe they asked, what a VM actually stands for and what is means, rather than what it's for.
For example, I'm diving into Home Assistant, I don't think I'm daft but I ended up posting what turned out to be a really silly request for help because I was not a local and used to the scenery. I'd read all the docs, which I will soon subtly alter, but missed an implied (if you knew the system) point.
How the heck do you describe what a VM router does, quickly? 8)
It's unfortunate anyone has to go through so much trouble to prove to Century Link what their inventory system is probably telling them anyway but it's always best to protect yourself.
Any reason not to do that?
That doesn't mean that there can't be cost savings from automation. For example if it costs X in lost business due to a misunderstanding by the remote hands that extends the outage unnecessarily, then a certain number of times avoiding that X cost would pay for the investment in automation. You pay the "sunk" and the new cost but you avoid unnecessary costs in the long run.
It's all a matter of fully modeling your costs and benefits. Noting that certain costs are sunk is a partial model.
You can make of course get it to be per-packet load balanced, but as you note, there are issues with that when you don't control both ends.
Using source ip to round robin on active wan connections is the safest.
Can't you keep both connected to a router and have a script do the switching instead?
Anyhow, still impressive.
I'm not sure if that's a joke or not, mainly because after reading it I'm thinking "It's a stupid idea, but ... no it's a really stupid idea, but ..."
That was followed by a couple years of 100+ hour/year cumulative outages due to HACMP stability issues, and an environment that everyone was deathly afraid to touch.
The hardcore network engineer in me appreciates the detail in these kinds of solutions, but these days the practical side of me is satisfied with usability and maintainability of SPOF cable access with a manual failover to mobile hotspot on the rare occasions that drops offline.
Even ISPs and CDNs I worked with sometimes have surprisingly uncomplicated redundancy systems (sometimes just a handful of small routers they are very much ready to power down to cut over to backup paths or bring up new paths) and often they do not use the more complicated methods.
The catch with complicated redundancy is there is always a very close relationship or protocol or something between redundant components, bet it storage systems, network systems, anything. Inevitably a system goes down or loses its mind and takes it's redundant peers with it.... every new system you introduce is one more piece that could reach out and take everyone else with it. I saw it time and again, and again...
I’ve also seen well built and maintained HA systems work exactly as desired.
As a general rule, the cost of building and operating a reliable HA solution is not 2x, but at least 10x. If the system being protected is not worth that, you’ll very likely find the MTTR acronym far easier to catch than the rather more slippery HA.
My home network is built with Mikrotik kit which is priced where it's affordable to have spares. I have yet to encounter a failure, but could drop in a new router in a couple of minutes with the saved configs.
I have SNMP monitoring feeding from telegraf into influxdb on an RPI. Dashboard rendered with Grafana on PC. Also have telegraf pinging to all 24x7 devices and collecting data from electricity meter, smartplugs, and Nests. It's been fun to do.
Web, Power, Internet, Network, Military systems at scale use reliable redundancy and work w/ very little downtime.
[1] i.e. the systems that interconnect the multiple redundant system, detect failures, redirect traffic, etc.
Turtles all the way down, I guess.
Indeed it is important in this case of course that this does not happen :) To see the increased reliability and P(glue failure)<P(single failure) you need to assure the glue systems are very simple and well built -- and preferably they need to be much smaller than the system you're protecting.
Another adequate expression to apply here is
"Who watches the watchmen?"
The answer again is the watchmen must watch themselves and be very reliable.
On this topic I recommend von Neumann's (the brilliant mathematician) "Computer and the brain" book, where he explores how computing systems can be reliably interconnected and how those failure probabilities interact. He was interested on how the brain could be so robust to failure -- don't worry there's no time spent speculating on how the brain works, instead he derives from first principles properties of reliable computing components, and possible reliable designs (the brain's unknown internal workings at the time, and now to a lesser extent, would follow as a special case). He used this same approach in analyzing the principles of life, where he came up with a self-replicating machine with a tape encoding of itself, predating the discovery of DNA -- it's a very inspiring and powerful approach. Unfortunately he could not complete 'Computer and the Brain', he was in declining health due to cancer and died while writing it. What was left is still very interesting imo. He is one of those giants whose shoulders we can sit on to peek over the horizon :)
As a caution against tenanting the deployment tools in-band, I'm reminded of an incident I witnessed about five years back. Company was moving their compute from on-prem to colo datacenters. Pretty good, mature setup: Almost entirely virtualized, 10Gb iSCSI SAN, credentials managed via a dedicated COTS tool, etc. They got most things over-the-wire to the DC. But the final migration had to be done cold - Shut the last bits down that were keeping everything running, move them to the DC and power back on.
Everything went very well until the SAN wouldn't come up. To get into the SAN and troubleshoot they needed the domain, which wasn't available. They had a local account on the SAN, the key for that was safely stored in the password manager. Which was a virtual machine. On the hyper visors. That wouldn't come up until the SAN was booted. Oops!
OK, that's a very obvious foot-in-mouth, in hindsight. As a more likely example, how about the Amazon S3 outage a few years back that wasn't reported on the status page, because the images for the status page were stored on... S3 :D
>you need to assure the glue systems are very simple and well built -- and preferably they need to be much smaller than the system you're protecting.
Absolutely agree.
Most half-baked redundant systems I've seen are a result of "I want four nines, but I only want it to cost 20% more than a two or three nines solution" type thinking.
With my luck, it would catastrophically fail while out of town, leaving the wife and kids without internet.
My dad set up a lot of complicated stuff like this. As people are prone to do, eventually he died, and it just made it difficult to troubleshoot technical problems for mom. So now the equipment sits in some corner, unused, because we replaced it all with something your average AT&T technician could troubleshoot.
Two ISPs, two networks. One called "main", one called "backup".
If "Main" fails, move over to "Backup", either with a cable, or on a different SSID.
That's not an unreasonable solution, considering most people already pay two ISPs (one fixed, and another for their phone/tablet). When your home wifi goes down, you're going to fall-back to your mobile anyway. I'm thinking of getting an extra data SIM, an LTE modem and do auto-failover.
--edit--
My needs are somewhat unique - my traveling laptop is on its last legs (and will be replaced by a cheap chromebook. Desktops/servers get better bang for the buck compared to laptops. Go figure!), so I tunnel onto a server at home for heavy-lift computing. If the internet fails when I'm not home, I'd be left stranded (and this has happened).
From a practical point of view I think it's silly to do such a thing for a residential situation, but I can appreciate using it as a learning experience for building systems like this.
3g is good enough backup for me, but for the office we go for two routers two isps and vrrp on the lan side, load balance across the wans, with failover to the other one.
I would not discount the possibility completely. But I judge it unlikely.
Then I'd put the primary router on the wired line, the other one on a 4G sim which did nothing but heartbeats unless the wired line went down. If the wired line shut down, traffic would reroute via 4G within 10 seconds or so. If the primary router went down, the backup router would take over in a similar time frame. Might put some capping on the 4G router to the netflix/etc boxes to keep bandwidth costs down.
UPS would be about 10W, so £45 for a 4 hour one. Possibly look at renewable energy of some sort to keep the UPS going during an extended outage.
I'd then VRRP on the lan side with primary on the main router (which would have a backup route via the secondary router)
Cloud based VM to do monitoring/alerting and land outgoing openvpn tunnels from both routers to allow secure remote access.
£170, £10 a month plus main ISP, and an hour of config.
However in reality having an ISP provided router and showing them how to tether in a problem works fine. OK, they lose their devices if the main circuit goes off, but running those over 4G can be pricey.
Link to the story online.
>because of its own organizational flaws and its willingness to discard old technology without having fully perfected the new.
ADSL Modem > Firewall > Router > Web/DB servers
It was basic, but it worked. Our web servers were mission critical, but as a B2B business they, and the ADSL connection, didn't sustain a heavy load. The only issues we had over several years were with the ADSL modem. Everything else just worked.
When we moved office we moved our servers to a co-hosting centre with an upgraded network setup with all sorts of backup and redundancy. Every week something went wrong. Sometimes simple is best.
Nobody understood exactly how the cluster worked to the point that a correction my boss made on the physical connections, made us loose a couple of million of dollars in transactions not processed.
The funny part is, when the cluster was working fine, a takeover took at least 20 minutes. During that time nothing was "available". The thing is, no matter what, SWIFT Alliance took that time to properly close and open the DB.
I've set up ISP redundancy on my home network before, I should probably test to verify that it still works after my update some months back. It's a truly high-tech solution: A Netgear WNDR3700v2 router (5x Gigabit, dual-band, circa 2011) running LEDE (previously OpenWRT).
It's not automatic, but I can set it to act as a wifi client, so if my regular Internet goes down I can simply connect into the router, connect to a phone hotspot, and continue providing internal network access. I don't recall if it's able to act as both a client and an AP on the same frequency at the same time, but since my wife's Kindle and Chumby are the only 2.4-only devices in the house I'm not really that concerned about it either.
And yes, the Chumby does still work though it's just a clock these days.
Like the guys who make videos of sharpening a grocery store knife to an atom width.
And maybe you'll find this interesting: Sharpening a wooden knife: https://www.youtube.com/watch?v=kKH63_r0OCA
And his cats are remarkably well behaved.
It is hard to beat a stock, as supplied by the telco, router with a generic Android phone for maximum uptime. If one connection is wired and the other is wifi then the computer handles broadband difficulties with no problems.
If you are actually serious about 'single point of failure' then you just need to live with someone that is likely to not pay the bills for electricity or broadband. Being insufficiently creditworthy to have better than a pay as you go burner phone helps too as every byte costs $$$. Living in an area where any nice toys will get stolen/destroyed also 'helps' as a refurbished laptop running linux is then only practical option. Congested wifi 'helps' too, a basic wifi booster with ethernet out becomes truly useful for 'blazing speeds', particularly if wanting your backup network to come from the local cafe or some neighbour with an easily Googlable password.
Having a local server for development and version control means that you are good to go when it comes to useful work even if there is no connectivity going.
For entertainment a regular FM radio works fine. Two refurbished laptops and a USB stick for bulk transfer of current project stuff makes it fully possible to pull an all-nighter even if there is no electricity due to bills-not-being paid reasons. A nice add is a Chromebook, those things designed for nine year olds with a battery that lasts 10 hours with no difficulty does the job with better wifi than any normal laptop, no fans and no thermal runaway.
Even better, the whole kit can be put in a modest backpack and a bit of couch-surfing later one can be back in business.
It is much more satisfying to do more with less, I would probably hate myself if I had a basement full of servers and only whiled away the hours on social media rather than do 'work'.
This budget ethos is anti-pattern but why should it be? The carbon footprint of operating on low-power refurbished hardware is penguin friendly and cheap. If your apps are supposed to be compatible with regular consumer PCs then it doesn't really help to have a beast of a machine with 4K screen, 32Gb or RAM and some quad Xeon. Maybe a linux toolchain with no virtualisation is better for making one's code performant on target devices. Obviously an SSD helps.
The kids and the grandparents can read books together if the devices are down. They can also listen to the FM radio. What's not to like?
Thank goodness I don't do company IT. Yes it would consist of two refurbished laptops hidden under the floorboards, servicing 50-100 office workers without any difficulty.
One of those quirky little devices that existed in this weird span of time when computing power was small and cheap, but our phones had not yet come to rule everything in our lives. RIP 2007-2012
Any resources, books, links, youtubes especially, that you can point me to?
Also, in his set up, theres no router? He says hes using a VM? What does that mean?
I assume you're not running a full BGP handoff to each ISP, so any existing sessions will die should your WAN die (as your lan get natted behind a different IP address). Presumably your nat state will move over in the case of router failure as it's a floating VM of some sort, so what's the failover time for each component? How does it compare to using say VRRP?
How are you detecting ISP failures -- are you pinging beyond the next hop, or are you assuming if you can ping/arp the upstream router, it's working? I've had failure scenarios with ISPs where the next hop works, but nothing past that.
What benefits are there of tcpproxy over something like nginx (for http/s) or dst-nat (for other connections)?
It looks like all your traffic defaults to WAN1, and only uses WAN2 in certain cases. Do you have the ability to send traffic for a given client to WAN2 by default?
What type of queuing are you using -- can 1 client hog all the bandwidth?
And finally, what keyboard layout is 6 above N?
I'm using this setup in my office. Easier than finding a last-mile type ISP that supports BGP.
Next hop checking isn't always good enough. I had a 7 minute outage on one line last week, next hop was fine, but outside the ISP network it all fell apart.
ARIN, at least, will happily assign you an ASN assuming you 1) meet the multi-homing requirements and 2) pay the bill for it.
Getting an AS is easy. Getting portable IPv4 address space or an LOA to readvertise is more tricky.
(Of course, you can start your own company for something like 13 quid/year. Now that I have one maybe I should revisit that.)
Seems like a lot of effort to ensure your ssh session doesn't drop
"Above" here is kind of incorrect, it's actually "beyond". Colloquially we say the keys are above and below each other.
Most of the gear was used for lab scenarios and such for various (Cisco, Juniper, et al) networking certs and was (mostly, but not completely) isolated from my "real" network. IIRC, I had ~35 VLANs at one point.
My extremely over-engineered home lab certainly served its purpose but I think I spent as much time maintaining it as I did actually using it, although it really came in handy for building out PoCs for projects I was handling at $work (my test/lab network at $work wasn't nearly as well-equipped as my home lab was!).
For the last several years, though, I've managed to get by with a single subnet that is shared by everything -- a few laptops, a couple desktops, a server hosting the handful of obligatory VMs, and, of course, the various phones, tablets, and streaming devices that are ubiquitous in all of our homes nowadays.
Just within the last few weeks, however, I've acquired a new server (2 x 10-core Xeons, 256 GB RAM, 4 "Enterprise" SSDs and 12 "Enterprise" HDDs (600 GB 15k SAS)), dug a couple switches out of storage in the garage, replaced my Internet router with a small industrial box running OpenBSD, and started building out a few more subnets for proper separation of various devices (I've twice been offered a 42U cabinet recently but, thus far, managed to say no!). Like probably most HN'ers, I've got a few VPSes spread out here and there as well. Finally, I've got a decent (but was over-built) 2U box in a rack at $work ($work == ISP) that I am planning to use to tie all of this together (using Wireguard, of course).
Yes, I'm fully aware that I'm in the beginning stages of a relapse. After these upcoming changes, however, I don't intend to "grow" this lab much larger (although this kinda stuff does just creep up on you sometimes).
I used to also have a 42U cabinet in my garage for several years. It housed a bunch of servers, mostly Dell poweredge but also some no-name boxes, plus some switches and other miscellaneous gear.
The power draw was too strong for my poorly garage circuit and after any power outage I had to power up the rack one device at a time - it was a massive pain. I also spent WAY too much time tinkering with it all, instead of actually using it in anger. Sure, it help me immensely doing PoCs for work or for my own learning, but it was always overkill. Funnily enough though, every other tech-head that saw it was envious, until I started detailing the horror stories of keeping it all running.
Thankfully Virtualisation became a usable and affordable platform for tinkerers, and I migrated everything (via a streamlined custom P2V process) to ESX, then later on migrated/rebuilt the VMs over to Hyper-V.
I now just run 2x Tower servers (HP 8xxx series workstations - dual Xeon based) and run 20+ VMs on each. Plus a single NAS for file storage. Life is so much easier... and the Garage is so much quieter.
What box and how's it performing?
Out of the box, pfSense can do multi WAN and CARP (similar to VRRP) clustering. At the office I have two older servers with lots of NICs and five WANs. Inbound redundancy is provided by dynamic DNS and SRV records etc. Note that to do CARP/VRRP, you do need at least a /29 IPv4 allocation. You need an address per box plus the virtual one that is actually used by services. PPPoA/E is harder to deal with than cable/leased line etc but it turns out that low cost Billion 8800NLR2 can do external IPv4 pass through as well as do the PPPoA/E. They will need an address as well from your range. You need something like them in this case because only one device can be the PPPoA/E dial up system at a time. Unless you have some very fancy secret sauce, your clustered routers' pppd or whatever are going to get confused as to who does what.
I notice you have a cloud key. Unifi on an Ubuntu VM is easy, and much easier to backup and snapshot before upgrades, so is safer. You can also front it with HA Proxy for simple URLs and perhaps Lets Encrypt. pfSense has a HA Proxy package with a GUI and I believe it is CARP friendly as well ...
I use Draytek 120 or 130s modems for single ADSL or FTTC connections but for CARP clusters, I use Billion Bipac 8800NLR2, so I am not doing the PPPoA/E on the pfSense boxes. The Billions are able pass through bits of a /29 and do the PPPoA/E themselves - the only cheap router (~£60) I've found to do this.
I've been running this thing for about four years now. PPPox is a complex beast and there are a few things to look out for such as MTU. PPPoE imposes an eight byte overhead (hence 1492) and back in the day some ill advised auth mechanism required setting a 1458 byte MTU. Apparently, some BT kit supports mini Jumbo frames of 1508 bytes which means that you could set your MTU to 1500 instead of 1492 - good luck with that as a rule of thumb. $DEITY only knows what an ISP in WA has arbitrarily decided to mandate. Here in the UK we have a near monopoly for the infrastructure but lots of providers that use it and so it should be simple. To be fair, I bet you don't get docs like this: https://www.btplc.com/SINet/SINs/index.htm (498 is FTTC)
Anyway, if you are happy maintaining your firewall rule set manually then crack on but nowadays it is hard to do that. pfSense has a lot of quite vociferous users who kick the tyres on a regular basis. It even looks quite pretty these days - all bootstrapped up and stuff, the red thing is long gone.
What's the "-1" in "80/20Mbs-1" and "100Mbs-1" signify? I've never seen this "syntax" or formst used before but maybe it's an EU/UK thing (I'm in .us, FWIW)?
-1 is meant as "to the power of -1". Thus, s-1 becomes 1/s, and the entire thing Mb/s
Never seen that either
For those that assume that was just a joke on escalating size, the joke was actually made it a real thing by Netflix when they actually named the component that randomly shuts down not just services, but entire AWS availability zones of Netflix services.
Child with a water balloon? Hope you have multiple data-closets in your house...
I run a similar set of WiFi gear. I've a couple PoE powered Unifi UAP-AC-Pro spread around the house, all connected to an 8-port Unifi PoE GigE switch. Routing is done with an EdgeRouter lite, which as it turns out is capable of line rate GigE.
I have a low power industrial computer with 4 cores and 8GB memory that runs various services mostly via docker or vagrant. It consumes about 12w.
It's all powered by a 750VA APC SmartUPS. I get almost an hour of runtime on the internal batteries. I may add some external batteries at some point, but most power outages in my area don't last longer than 20-30 minutes.
Not trying to be a dick, but does that count as "fairly common"?
Shorter power outages are more common; 10-ish power interruptions of less than ~2 hours per year.
It's not like, developing or failing nation bad, but it's not great, especially when the problem is always "a tree limb fell on a wire".
My home setup:
hardwired all the desktops and a few access points via cheap 1gbit hardware (literally found some at the thrift store/ebay), usually using tomato/shibby.
have a backup router.
battery backup on main routers/modem.
large external battery wire nutted to my desktop UPS.
NAS is an old laptop with battery intact, doubles as second display/machine.
use my phone via usb on my desktop if all else fails.
total cost, probably less than $100.
Oh, and I use a $5/month server for stuff that absolutely needs to be on full time. Otherwise the only external access is me occasionally remoting into my desktop and I am happy to stop and smell the flowers if that is interrupted briefly.
My laptop is enough for me to stay productive (it's a ThinkPad 25! very productive). Everything that needs to be online is on a Hetzner server I rent for all sorts of purposes so the 51 EUR monthly bill kind of spreads out.
I went with desktop because I wanted everyone in the house to have a decent machine and I could get several I5s for less than $70 apiece (5 machines, one in each bedroom) and wanted easy/cheap upgrades for some of them, and they are all the same optiplex model, which makes my life easier.
I like my desktop setup a lot though, 3.3ghz I-5, 27" 1080, 16 gig ram, 1tb ssd, 8tb in "cold storage", g402 mouse, gt710 vid, clicky keyboard, Nubwo N2 headset, decent posture, 100+ fps gaming. Probably threw $500 at it above the initial $70 though, but most of the machines didn't get that treatment, but their users aren't using it to make a living either.
I used to use a dual-WAN setup with cable modem + DSL backup. It worked well with automatic failover. I use a pfSense APU based router and, with no moving parts, it's been very reliable, nearly 4 years without any unscheduled downtime.
Then I moved and only had a single ISP to choose from, so my backup is to manually turn on a Wifi hotspot. I thought about using a cellular router with ethernet or a wifi connection to the hotspot for auto-failover, but it just wasn't worth the time and/or money to set it up -- if I'm home when the internet goes down, I can just switch to the hotspot, if I'm not home, then all I really lose is the ability to control the lights and thermostat remotely, not exactly a critical function.
I think that's quite the understatement. The thing that really stands out to me is the claim that all of that is only drawing 220W at idle. I'm curious if he means truly idle, like literally just booted up and not doing anything at all, zero traffic, etc. Or if that's the draw with stuff actually being used. Because 220W just for your home network is hilarious. I mean I feel dumb often because my little pfsense box pulls about 15W.
You can do load balancing using PF as well, which is what we were mostly offering, cheap fault tolerant hosting for colocated customers.
Having VMs float around with shared storage makes complexity elsewhere go away. i.e. I don't need to deal with CARP, VRRP, etc.
The main thing that might make me shy away is the added exposure at the edge. If the VM hosting is dedicated to just the network failover/firewall, it seems wasteful, and if it isn't it seems unnecessary exposed.
The only other thing I'm not sure of, since I'm not too familiar with AL the VM solutions nowadays, is whether an actual hardware failure of the active VM hardware allows seamless failover (which you do get with what we were doing back in the day).
Edit: although, it's not hard to emulate the stuff we were doing using some OpenBSD virts on those two boxes, which even if they don't support full hardware failure with the current setup they then would. Since you're playing with the for fun, you might be interested in trying it. If you find OpenBSD intimidating, you can use pfsense to do the same, which is a dedicated GUI configured FreeBSD distro that offers much the same (there were some CARP implementation differences/bugs in FreeBSD way back, but I think they got fixed up long ago).
* Cantenna/laser link to a house some blocks away to avoid local WAN link disruption
* For less performance-intense networks, remove the physical impediments: 2 routers, each with 1 APC, connected to 2 separate power circuits, connected to 2 WAN links, providing 2 radios each. No switch to go down or cables to trip over, redundancy of access point, redundancy of frequency/radio, redundancy of WAN link, redundancy of power. Hardware-wise this is pretty cheap and still highly available. If the routers are cheap, use a hardware watchdog.
I attempted something similar to this in a 20U cabinet some time back. The biggest issue is the fan noise that 1U form factor servers and network gear produce, with their rather high RPMs. One can hear the noise across the other side of the house.
We've since switched to fanless network gear and ATX form factor servers with large diameter fans to keep the family happy. It definitely doesn't look as nice, though.
Not as cool though, and clearly not running any servers, but that's what things like AWS or Linode are for -- or for low power stuff, something like a fitlet [0]
If your home is directly connected to their datacenter...
Not everyone has 10 Gbit upload with best peering!
I'm happy with a QNap as the only home server I need.
It's insane how quiet you can go with this approach, while remaining air-cooled. I know when my home server is running backup scripts because the noise increases at least tenfold when the hard drives spin. Fortunately, I have coordinated that to be only once a day -- the rest of the time the drives are in standby.
It's not whisper silent, especially during the summer when the fans on the R320 speed up to around 6000RPM (and this is with a E5-2430L) - but that's mostly due to my office remaining closed from the rest of the house leaving the ambient intake temperature around 75-80F (rest of the house stays at 72F). I'm probably going to stick with 2U's (probably R520's) when I start expanding again to lower the noise at higher temps, since the more equipment I add the more heat gets trapped in the room.
Clearly hasn't been bitten by it, yet.
I mean... I love Ceph, too, but I don't ever want to run it again.
Recovery is difficult and there's no support unless you have a subscription from Redhat and also run RHEL plus their stable distro of Ceph (RedHat Storage or whatever). IIRC, they quoted me $90k for a petabyte of raw disk.
I haven't messed with it much in the last couple of years. Bluestore looked really promising. I've thought about taking a look at rook, but haven't yet.
If I were in a position to deploy a bunch of storage on bare metal again, I'd likely go with ceph. I do know that $GLORIOUS_FORMER_EMPLOYER ended up making the migration to ScaleIO and report being happy with it and having good performance.
This is more of a homelab tinkering setup to learn.
And thanks for all the Go code. It's awesome! I'm building 1.10.3 on an old Beagle Bone Black right now ;)
I used to use a Soekris net6501 as my home gateway, but its CPU maxes out NAT'ing about 300 Mbps, sadly, so I started looking at alternatives when I got Centurylink fiber.
I used to use a UniFi Security Gateway Pro but it failed one day and wouldn't power on any more. Dave had a backup for me handy, but the Unifi controller software wedged itself and wouldn't let me remove the old (dead) one ..."
There is much adoration of Ubiquiti hardware on forums and message boards. I do not doubt for a moment it has been well-deserved.
However, I have a question about the software. I would like to use own kernel and custom utilities.
If I understand correctly, installing one's own choice of OS on Ubiquiti hardware is not always possible and even if successful it carries a penalty in terms of performance versus retaining the Ubiquiti pre-installed proprietary OS.
Soekris made it easy for the user to install the OS of her choice. Tradeoff: More user "control", but a slower router.
The question is: Are there other alternatives to Soekris that can exceed 300mbps and allow for user-chosen OS?
This is another line of (faster) routers where the vendor has allowed for easy installation of user-chosen OS.
https://protectli.com/product-comparison/
There are comments in some other forums and message boards about these computers but I have not seen this company discussed on HN before.
Note the website claims models FW1, 2 and 4 have no Intel ME, SPS or TXE.
https://protectli.com/kb/intel-management-engine-vulnerabili...
I don't think Intel ME is at the top of my threat model - by the time someone's using that kind of stuff on me I'm screwed anyway. I do, however, pay insane prices for power (28-34 cents AUD per kWh). This has pretty much meant I look for ARM and MIPS devices everywhere, but the latest gen Intel stuff is looking good.
I hadn't seen those Protectli boards before and they look quite cool - I'll keep them in mind. At full tilt, it'd cost me about $85 AUD per year to run.
If Marvell ever open sources the switch drivers for the Espressobin [1] [2] then that may be an option to exceed 300mbps.
0: https://omnia.turris.cz/en/
1: http://wiki.espressobin.net/tiki-index.php?page=Topaz+Switch
Don't get me wrong, I still have highly available Internet at my house - I just tether my laptop to my phone and I'm done.
Maybe you missed the New Yorker article entitled 'The Really Big One' [1]
When that fails I switch to my iPhone. :)
(On a more serious note, I’d like to see the basement or whatever with raised floors. Come on Brad. ;)
My setup is Comcast going into a simple, reliable Surfboard modem, feeding a Google Wifi setup. If it goes down, which it just really doesn't do, we can use cellular data.
Complexity is the enemy of availability. Keep it as simple as possible, but no simpler.
I can hardly remember the last time that my internet connection cut out... but if I had to guess... it was probably during the peak of a 100 year storm we had a few years back that put the entire area underwater for about 48 hours.
Transformers were blowing up all over the place, the power was out for days in some areas, and yes the internet went out as well at that point.
I live in the GTA FYI.
Here is my top down take on a more traditional (cheaper) approach. * 2 1G 5 port edge switches * IDS * vrrpd balanced cots NAT routers -w- RIPng + nginx as generic and web proxy. * LAN 1G 12 port switches (1 hot, 1 cold) * 2 synology NAS (redundant, manual failover). * etc...
https://www.electricitylocal.com/states/washington/quincy/
The average residential electricity rate in Quincy is 4.85¢/kWh.[1]
4.85 << 9.74
https://www.netgate.com/docs/pfsense/highavailability/config...
If you're asking about in general would this be a good thing for a Twitch streamer... then I would say no. Mostly because most Twitch streamers are not going to know how to maintain something like this and they don't need all the servers.
If someone not so technical, Twitch streamers included, needed the redundant internet I would recommend something more along the lines of two ISPs like this guy (specifically over two technologies if possible: fiber and wifi, but that comes down to bandwidth requirements) but instead of going into multiple switches and having 3 servers running with VMs moving around just plug the two ISPs into something like the Unifi Security Gateway (USG) or USG Pro.
A very outdated post about my setup : https://www.sajalkayan.com/post/fun-with-mptcp.html
I now have 2 broadband ISPs, and optionally I can hook in my phone's 4g into the mix.
Multipath TCP allows me to "mix" bandwidth of both ISPs at the same time.
Yet plenty of folks (myself included) have been doing exactly that for well over two decades without any major issues to speak of.
You are likely correct that it was "a software / configuration problem" but the lack of any actual details or useful information makes it impossible to offer any potential insight; baseless speculation is the best you may hope to receive.
> to have a highly-available home Internet setup, with no SPOF (Single Point of Failure)
> to learn and have fun.
I was researching and experimenting: What hyper visor is out there providing a good file system (zfs) and also full disc encryption at the hyprvisor level?
tldr: FreeNAS
And it came out that this is not that trivial.
You can buy a Vsphere/ESXi license for encryption, but (probably) don’t have the same capabilities as ZFS.
You could use Hyper-V and have encryption but no ZFS.
On the other side there is Promox (Debian 9 stretch) which has an installer which uses ZFS (but no encryption). You can jump to some hoops and make a manual Debian 9 Installation with ZFS and luks (for the encryption) and then install Proxmox. Then you have to watch out to use the ZFS version Proxmox uses (instead of the Debian version)
You could use OmniOS, SmartOS to get ZFS, but again no encryption out of the box.
Solaris 11 has the ZFS and encryption part figured out, but the hypervisor part is not clear to me.
So FreeBSD has ZFS and encryption (GELI) figured out as well. For the hypervisor bhyve. Still there is manual work.
Then there is FreeNAS. It has ZFS, Encryption -and- hypervisor streamlined. :)
Some people use it as a VM guest inside Proxmox/ESXi, pass through their discs and from FreeNAS Export either NFS or ZFS over iSCSI back to the hypervisor to use as a storage pool.
Or as I found out, FreeNAS 11 has the bhyve hypervisor built in. You can have FreeBSD jails for BSD and Linux, or full VM guests via bhyve like Windows or Docker/Kubernetes.
FreeNAS ships with RancherOS as the minimal Linux vom, which can act as a Docker host.(if you don’t want to setup your own)
So for our use case of having a safe file system and full disc encryption and be able to launch VMs, and to have this very easily installed on an USB stick with minimal configuration and excellent documentation, I would recommend trying it out.
Of course Proxmox has live migrations, which is not figured out here. Probably Kubernetes would help.
Probably the other good way would be to have drives and a mainboard which support encryption at the hardware-level. Or wait until zfs on Linux v.0.8 is more in use. It contains encryption support.
I think you'll still hit bottlenecks with the switch on the Espressobin - Marvell hasn't enabled hardware acceleration, at least for the open source parts.
Can someone write up exactly how to set something like that up, maybe show us some urls?
With three servers, if you have two power failures then the Ceph monitors will no longer be able to achieve a quorum.
In practice using a Wifi-router with 4G fallback would achieve similar availability at a fraction of the cost and power consumption.
What would you cache & how much?
OK, it went down once right after install but that was due to a tech accidentally disconnecting me at the node while connecting a neighbor.
I'm in the Atlanta area. $0.07181/kWh.
For example, Grant County PUD for residential customers: $0.04547 per kWh
His gig internet is $80 a month: https://www.centurylink.com/fiber/plans-and-pricing/seattle-...
His wifi backup internet is $40: http://www.gigabitseattle.com/residential-services
He specifically states the setup draws 220 watts at idle and that his electricity costs $0.0974/kWh. So 22024/10000.0974 = 0.514272 per day, or about $15.40 a month at idle.
So around $135 a month.
yea if it's idle the entire month, which is doubtful. but even if it's not, it's not likely to be too much more than the $135 you calculated. I figured the internet service would have been more, since the rest of us get screwed by our ISPs on costs.
No modem, just an Ethernet drop into your home.
I use the UAP-AC and an EdgeRouter. The EdgeRouter has relatively sophisticated capabilities for a piece of consumer great network gear. I have GigE fiber to the home and get ~900Mbps through the router (and ~400Mpbs through the access points).
I generally don't recommend the USG, which has similar functionality and is integrated with the Unifi management platform.
What advantage do you get with that versus just running a bunch of APs with the same SSID/password bridging to a single router? I do that with 3 cheap tp-link routers (1 as router, 2 as APs) and LEDE and both my laptop and phone work seamlessly. At one point I considered actually doing full 802.11r AP roaming but the only actual use case I had for that was doing VoIP calls while roaming between APs with no drops. Everything else works fine with the small interruption of switching APs.
> Note that UniFi Fast Roaming is not a direct implementation of 802.11r - it is a solution taking inspiration from 802.11r, with a few key proprietary differences. We've found that Fast Roaming provides about 90% of the roaming improvement offered by BSS Transition. However, Fast Roaming does not require client support, allowing backwards compatibility with all clients.
https://help.ubnt.com/hc/en-us/articles/115004662107-UniFi-F...
The app can be self-hosted, run in the cloud, or on something they call a "cloud key" that's not much more than a Raspberry Pi.
I've run it on a Pi3, and it's a little bit laggy, but tolerable. I prefer to run it on my little x86 server.
Both the self-hosted and the cloud version can be managed remotely, which is neat.
There are a host of other benefits, but given the price (around $100 per AP), I see no reason to use the more commonplace consumer grade stuff. Check out the unifi demo here:
APs can provide the client with a list of neighboring APs and things like 802.11r speed up the reassociation process when there's a complex backend authentication process like 802.1x, but it's still entirely up to the client when to roam. Ubiquiti's early products did provide a "zero roaming" option which faked one big network, but this was very inefficient bandwidth-wise as it required all APs to share a single channel.
Proper network design such as minimizing TX power to encourage a client to roam due to a "weak" signal and maximizing coverage with lots of APs typically is the best way to get clients to roam reliably.
I went with a US8-60W and a UAP to start doing Power over Ethernet everywhere I could (it's annoying that the USG doesn't support this). The USG hasn't arrived yet, but it'll let me do PPPoE, firewalling and stats for the network.
However, I've had a couple of the USG fail under certain circumstances with enough bandwidth (>200mbps) or tickling a certain feature of the firewall. The EdgeRouter has been more reliable for me.
I'm not recommending against them, I suppose. I've just been bitten on a couple issues that I imagine have likely been corrected at this point. I was a bit disappointed that it took Ubiquiti a couple of years to stabilize the product.
I do have one on hand that I intend to replace my EdgeRouter with at some point so that I can manage all the devices with one app.
(edit) I think I was also waiting on IPv6 which it looks like they've added at this point:
https://medium.com/@poolski/setting-up-ipv6-on-unifi-securit...
Ah, if only that were possible here.
It's in my garage, though, so I don't care. But it's not annoyingly loud. I used to have a 1U server in this same garage that was annoying... old Xeon that drew 200+ watts idle with killer fans.
This setup runs at about $50/mo, or $600/year, when idle. Do I have this right?
Doesn’t sound cheap to me.