Resolving an unusual WiFi issue
blog.ando.fyi
blog.ando.fyi
Qt should've posted a public advisory announcement when this was discovered. Instead it has plagued users for nearly a decade now because very few Qt developers even know that their Qt5 apps are ruining the WiFi connections of their users.
Why addressing this took so long I think it's worth taking a step back to look into. This isn't some small UI change it impacts system performance in a negative way for so many users.
The source of the problem is Bearer Management which was a feature of Qt5 where it occasionally polls the OS for WiFi networks to speed up WiFi roaming: https://doc.qt.io/qt-5/bearer-management.html#roaming
However the result of those polling requests is that the OS might opt to drop everything else to complete the network scan. Some incomplete workarounds were implemented that mainly revolved around disabling the feature on affected systems: https://codereview.qt-project.org/c/qt/qtbase/+/214029
Without reopening old wounds and _days_ of wasted dev effort. Defective design is still a bug.
Anyway whenever I switch it on my wifi goes to shit. Apparently it does some SSID scanning every 5 seconds and then keeps sending the scan result to the "mothership". So I switch it on once or twice a week for an hour or so to do its spying thing.
what would it say? how ridiculous.
I'll bet you can't insert USB flash drives either.
We keep the monitoring to as minimal as needed to meet that requirement.
If your workstation stops checking in, or tells us your workstation doesn't comply then we will ping you to fix it.
It's always polite: Your machine is reporting X please fix that.
E: I should point out that our monitoring doesn't involve wifi like the person you replied to.
The main issue I had was that most "consumer" access points don't support multiple SSIDs with separate VLANs. In the end, I went with a Netgear WAX something that can support 4 SSIDs, each with a dedicated VLAN (+ a separate management VLAN). But it's more expensive than "normal" APs with similar performance.
I've had something trigger every device into asking 'who has xyz ip tell [Mac]'. It made the network unusable and even rebooting systems it would come back since as soon as one device asked the question (sent the broken arp packet) all other devices decided they too needed to know.
The solution that worked was to flip the circuit breaker for the whole house and reset every network device at once.
Usb ethernet adapters will send these pause frames, waiting for the computer to reconnect.. but they end up flooding dumb switches..
Bug bounty opportunity missed?
Think of the number of meetings that could be cut short by the ohp system crashing, the many hours that's could be saved :p
Obviously an actual broadcast storm can take down an entire network, but excessive broadcast traffic on WiFi specifically can also suck up a huge amount of airtime for little bandwidth. Every single device has to go to the slowest speed and stop what they're doing to listen and make sure no one is left out. Using STP/RSTP with proper values set and LACP for aggregated interfaces can help prevent inadvertent network loops. Many switches also support some kind of port isolation and explicit per port storm control restricting max numbers of packets/second for unicast/broadcast/multicast traffic. WiFi APs can use proxy ARP to cut it in their domain too. The WAP already of course knows the MACs of every device connected to it by definition, so there isn't generally any reason not to have it answer ARP requests on their behalf then forward the traffic itself.
For home wireless another trick, assuming you have a higher end model, is setting it to a different network than the wired. If you're not in particular need for slow service at the very edge of the coverage zone you can also raise the minimum data rate and then BUM traffic and wireless management frames won't be as painful even when they do occur.
While I agree with the general idea, I think that there are only so many hours in a day. If the broken device isn't easily fixable, like most IoT junk out there, it's nice to be able to limit the damage.
Storm control has more a "John brought a home Netgear (which ignores but doesn't forward spanning tree packets) and created a loop between two wallports despite spanning tree being enabled on them. Thanks to storm control Jane is still able use her IP phone for a meeting while the network admin investigates a monitoring alert about excessive broadcast drops on John's port which is causing slowdowns in that network" type use case. It's not meant to be the long term solution to anything it's just there for when other long term solutions fail to kick in and you don't want a full outage until you can fix them.
I'd actually originally (and still at many sites) intended to hold off and wait for WiFi 7 gear, because at that point a bunch of clients (and myself for that matter) will be interested in replacing WAPs anyway which is a very natural point to consider changing manufacturer as well. But a breaking point has come at a few places with a final feature which is PPSK, allowing the system to have many different passwords for an SSID that can be assigned different tags. Basically it allows having many of the benefits of WPA-Enterprise in terms of segmenting different clients onto different VLANs and revoking credentials and the like with more security and less manual work than MAB (MAC bypass) while still looking like a normal PSK scheme, which means the vast universe of brand new stuff which doesn't support 802.1x and never will works with it happily (by the same token none of that is going to play directly with using a secure virtual network or other better systems either sadly). Lower overhead and better compatibility than captive portals for non or semi-interactive devices as well. Someone hacked together a demo showing this could work on UniFi WAPs like four freaking years ago and Ubiquiti never did anything with it in favor of endless bikeshedding GUI changes to add more white space and hide important features and information (yes I'm a touch bitter).
So I'm not in the position of wholeheartedly recommending Omada yet, I don't have years under my belt there and it's relatively speaking fairly new. It has its own warts and rough edges for sure, from the software to the hardware physical design. But it can be self-hosted and the trajectory looks massively better, has already had more meaningful improvement in months than UniFi has had in years, seems to perform much better so far as well.
Of course the Venn diagram of self-hosting, herding lots of hardware with single pane, fully networking features, ecosystem richness and so on is pretty minimal in the overlap. Take away any one or multiple of those and options expand a lot, Aruba InstantOn for example.
And welp, this didn't end up "basically" at all did it, sorry about that. I am bummed by the sheer wasted potential with Ubiquiti. So it goes in tech over and over again though, we've all seen this movie many, many times.
As far as tips, I would suggest if you plan to stay on the managing-your-own-networks route to very strongly consider having the router/gateway stuff be separate and fully open source as I ended up. Doesn't have to be OPNsense, could be VyOS or plain OpenBSD or whatever else you're most comfortable with and depending on how you want to manage stuff and what needs there are for others to take over. But it's very, very pleasant to have the full spectrum of quality PC hardware available, you can get far more power for less, and you're never stuck with a critical aspect. I'd still suggest generally running that on metal rather than virtualizing it in a (semi)production network, but opinions vary there.
----
I've also soured on UniFi a lot over the last few years. It's crazy. The SMB / MSP market is massively underserved and instead of staying focused and capturing it Unifi is trying to compete in the enterprise market where, IMHO, they aren't going to succeed.
If we try out Omada and feel like it gives us everything Unifi does without all the feature creep then we'll do the same thing as you; EoL sites will move from Unifi to Omada. I'll always take fewer features and stability over continuous new features.
> I would suggest if you plan to stay on the managing-your-own-networks route to very strongly consider having the router/gateway stuff be separate and fully open source as I ended up.
We've used pfSense forever and have been happy with it. However, 2.6.0 has some gateway/dpinger issues where the bug fixes are slotted for 2.7.0 on the CE side, so it feels like the inevitable neglect of the CE version might be starting to kick in there. I tried to explain to them the realities of the market we're in (small businesses) along with real numbers in terms of what kind of money they could extract from businesses like ours and all it got me was an offer to talk to someone in sales and we're not buying into subscriptions (ever).
It would cost us $15k+ / year to switch to pfSense+. The software only subscription is nearly 2x the cost of buying their entry level appliance (assuming a 5 year lifecycle). IMHO, that's an indicator they don't want to support 3rd party hardware and there's a risk they'll eventually drop it.
I wish VyOS had less crazy pricing because their stuff looks nice. They used to have a $600 / year professional subscription which they don't appear to offer anymore. I think the corporate subscription used to be <$4k / year and now it's $6k. I wonder why no one wants to commit to subscriptions. How can I sell anything to a customer when I can't give a predictable price for ongoing maintenance a year into the future?
Our customers will literally go back to using their ISP supplied modems instead of proper firewalls before they'll pay the prices everyone is asking for. I don't blame them either. They only need a fraction of the functionality being sold and all of the fancy, expensive features everyone is using to justify sky high pricing are simply bad value in that context because they'll basically unneeded bloat.
No problem, glad to chat about this and commiserate a bit with someone in a similar boat.
>I've also soured on UniFi a lot over the last few years. It's crazy. The SMB / MSP market is massively underserved and instead of staying focused and capturing it Unifi is trying to compete in the enterprise market where, IMHO, they aren't going to succeed.
Yeah, it's such a huge waste. But the CEO is a classic trouble case with near total control (owns majority of the stock and thus effectively owns the board, a few caveats in terms of protection for minority shareholders in public companies don't really stop the trend).
>If we try out Omada and feel like it gives us everything Unifi does without all the feature creep then we'll do the same thing as you; EoL sites will move from Unifi to Omada. I'll always take fewer features and stability over continuous new features.
UniFi hasn't even had "continuous new features" though! Or rather, their "features" have been stuff like reskinning the GUI or doing their own entire custom silly speedtest stack for a while, not "features" like PPSK, or for that matter just having normal DHCP and DNS management! Or any sort of certificate management or customization, or, like, anything of substance. Or on hardware, they didn't even bother to upgrade their UniFi Security Gateways literally ever, not once! The ones still selling now are the exact same hardware as 2014 launch, no new updates in forever. They intro new stuff without ever sunsetting older stuff in general really. Use "EA" as a popularity poll test. The dysfunction goes on and on.
I don't know if Omada will be the answer yet but worth a spin IMO. I'd definitely hate having to either go back to lots of individual management or something cloud-based. I am stick with UI for PtP/PtMP stuff for the time being though, even if some rot is creeping in there too.
>We've used pfSense forever and have been happy with it. However, 2.6.0 has some gateway/dpinger issues where the bug fixes are slotted for 2.7.0 on the CE side, so it feels like the inevitable neglect of the CE version might be starting to kick in there.
I'm afraid that company has a history of scummy behavior too :(. I think OPNsense is just plain better but ymmv. FWIW, while it's OSS there is at least one company (Deciso) that does a "Business Edition" at a relatively sane rate (~$150/yr, less for a 3 year) with no hardware reqs (they sell hardware too but I don't suggest getting it) and does business support contracts as well. And Sunny Valley Networks does an interesting little layer 7 inspector that can plugin to it. It conversely might not be good enough for your needs either, but it's not crazy. And the free version is still quite solid. As I said in another comment the docs are pretty decent so might be worth giving them a glance and firing up a system in a VM.
At any rate my clients are happy now. And despite rough edges I'm grateful there are a number of essentially decent options available.
But basically Ubiquiti has become a toxic dumpster fire of a company and their product lines (UniFi in particular) on a downward trajectory in terms of performance, features and stability for quite a while.
But it's very, very pleasant to have the full spectrum of quality PC hardware available, you can get far more power for less, and you're never stuck with a critical aspect.
The big problem is that homelab/SOHO folks (myself included) don't want to pay the upfront cost of supporting or developing/testing modern hardware. At the price point Ubiquiti is targeting I think any entrant will be doomed to failure.
4-6 years ago things still looked quite promising, high energy great community and forums (gone now), a basic but workable issue/feature tracker (gone), lots of clearly talented engineers who were responsive and right there (gone), etc. Very good price/performance, quality and so on particularly compared to other stuff of the time. There was a reason it got a lot of recommendations. That was certainly a while ago now though.
They've always seemed like the MongoDB of network hardware.
Eh. Though a grim bit of irony is how old a version of MongoDB they've stuck on too amongst other things.
>COTS hardware tends to be more expensive and power hungry
Not enough to me to matter I guess. For applications like this one can find solid low power hardware, Xeon-Ds for example, that consume tens of watts at max load and idle plenty low. Or truly embedded stuff a single digits. There are enormous floods of stuff available that is a few years old for dimes or even pennies on the dollar. I replaced all my USG-4Ps with a few years old Supermicro Xeon-D front-network port systems I got for around $350. An extra 20W 24/7/365 is something like $20-30/year, for a device this important that's well worth it to me. And there's no comparison, it's a slaughter. Real systems have actual BMC/IPMI which is super handy if things go wrong. The performance is just ludicrously higher. It's trivial to have more RAM, the system runs off of mirrored ZFS drives I can count on, it can do HA, etc etc. All the normal tools are available.
I'd even take an old Celeron or Atom system over a USG though. Even without going to second hand market, something like Protectli's basic J3060 based FW2B would still be better IMO and has a max power of 12W, fanless. "Purpose built network hardware" doesn't mean much, hardware vlan filtering or offload for CRC/LRO/TSO are something a decent NIC can do, or even plenty of built-in stuff.
As for Ubiquiti, I remember them thumbing their noses at the GPL and introducing security bugs in u-boot or having no idea how to fix the issues with RADIUS. They had no clue how to work around Cavium bugs (or get support from Cavium). I remember a company that shipped products based on a lightly skinned Vyatta using an end-of-life'd version of Debian (note that Debian ended support specifically for MIPS). I remember a company that shipped products (ER-X, ER-L) that ran hot enough to reliably cook themselves. And, yeah, Mongo. It was just a mess top-to-bottom.
This is very true. However if you bypass the Dream Machine and just use some other router or an EdgeRouter it’s ridiculously reliable in the home setting.
There are a heap of nice features in the UDM line, but wow does it need polish and stability.
My wifi is the best I’ve every used by a million miles, but the UDMP crashes and has weird behaviour that requires heroics to repair.
Anyways here's the regex blacklist I use in PiHole to prevent Apple's reverse DNS spam -
^(?:(?:\d+\.)*\d+(?:-in\.addr\.arpa\.)?)$there's been reboot the sun plot. there's been reboot the earth's core plot. there's been reboot the machine running the city plot.
https://mnpn.github.io/blog/airplay-network-disaster
Previous discussion:
It's the source of so many weird issues.
For example, locked down mac, using wifi at a friend's house and their LG tv shows up as an airplay mirroring device. Why should my machine be discovering that TV without me asking? When I'm on a public network, I'd like to make my machine output-only, not promiscuous in this way.
there was also an issue where a macbook would randomly lose its onboard sound and somehow default to using a nearby appletv as the output device.
The TV advertises itself on the network so it's the one being promiscuous. Your machine is still being passive, it just shows you the devices that are have advertised themselves.
No idea about the sound thing though, I don't use any Mac stuff :)
I recall disabling the `awdl` adapter can stop a lot of this behavior, at the cost of breaking these systems. I used to do this when using Moonlight, as it would cause periodic ping spikes.
At least it finally got fixed after 6 years...
Turned out to have multiple antennas, and the transmit antenna was broken, so it could receive just fine, but not transmit over anything but short distances.
Sometimes the physical layer is the problem, even if the logical layer says everything is fine.
I've had the same issue without any antenna troubles - Mac would constantly connect to the 5GHz network and struggle to send any packets out, yet the displayed signal strength was good. It turns out it was able to "hear" the AP just fine, but the AP had trouble hearing back, yet somehow there's no feedback mechanism for it to know.
At the moment there's this really weird network issue we're having where iPhones are unable to play Netflix on the Wifi. Every other device works fine but iphone 7, 8 (2 devices) and SE can't stream Netflix. I noticed there is other things they can't do, for example the page for the fast.com speedtest loads but the speedtest cant be performed. Same with the Google speedtest. The phones also can't access Apples update server on the wifi. Other network stuff does work fine, youtube works, browsing works, etc. The behaviour is consistent across the iphones and all these things work fine on multiple other devices on the same network.
I can't make sense of it at all.
Called internet provider and they didnt know either apparently other people had the same issue but nothing has changed from their side of things. Called Apple support and they are putting the blame on the network provider.
Tested one of the devices on a different wifi network and works fine.
AFAIK if you're an internet router the packets look the same no matter what device is being used so I think this must be some Apple software issue. Or maybe my router is cursed.
All your symptoms are explained by this (some but not all devices are affected, some but not all services are unreachable).
It doesn't help that FreeBSD used to do MSS sensibly (thanks to a patch from Microsoft), and switched to doing it the way everyone else does. There's not a good reason to send your naive MSS on SYN+ACK if it's bigger than the received MSS, it's highly likely that the other end can't successfully send a packet larger than they can receive, but it's not unlikely that something in the middle mangles SYN to reflect the true MSS, but doesn't mangle SYN+ACK. I could rant all day.
You can also get very good results by reflecting the received RSS minus 8 (assumed PPPoE), 20 (assumed IPIP tunnel), or 28 (assumed PPPoE over IPIP tunnel).
For me, it's always DNS until proven otherwise. But the difference of some sites loading, but others not makes me suspect there's a split somewhere and IPv4 vs 6 seems as likely as anything.
It has caused some significant weirdness, and I've had to relax the rules just to get certain things to work at all.
Fielded a lot of grumpy calls that Christmas morning.
the very first thing I do when I have a problem on wifi, is to remove wifi from the equation. wired Ethernet is so much better, and so far, the problem always disappears.
I find it stupid that I have to employ such a janky solution to this.
Citrix Workspace on the Mac does not have this problem, but using an old MacBook Pro for this was undesirable for various reasons.
https://www.reddit.com/r/GlobalOffensive/comments/3ahg59/fix...
Turn autoconfig back on only when you restart your PC or disconnect from the network (maybe someone can automate this by checking connectivity without scanning networks, enabling autoconfig, and then turning it back off)
If so, is there a better routine to break on in the debugger to see it actually initiating a scan?
Knowing nothing about this scanning process, I’m just assuming they first enumerated wireless adapters. So you could start with iphelper and then explore deeper into how you tell the card to scan. There’s probably some API for that as well.
So now I pretty much exclusively buy USB-2 hubs. Very little saturates 2.0 speed anyway.
Two wrongs don't make a right.
Only after much online searching did I stumble on another user with the same issue and after uninstalling it and reverting to an earlier version the issue immediately went away. I've since kept a more open mind as to what may be causing network issues.
Most wireless links are constantly in a barely working mode that's only made usable by TCP and various retransmission/error-correction algorithms on top, so a lot of connections that people think are "fine" (especially when the signal strength meter is maxed out) are actually not fine at all. There is no magic and those workarounds don't work for real-time streams such as calls/games/etc, so those "fine" connections will crap out miserably as soon as users jump on a call or game.
I'm not sure if there's a way for the OS to get any more info about the link quality/stability from the network card itself, but a purely software-level workaround could be to just track all TCP sessions (should be easy, as that's already managed by the kernel), average the percentage of retransmissions across all of them (to prevent a single bad server from skewing the result) and factor that into the "signal strength" meter.
An easy way to test this yourself is to run a ping towards your default gateway (the IP of your router). You should see a consistently low latency (in the low single-digits range) and no packet loss. If you're seeing losses or significant variation in response times, your connection is bad and you should do something about it (get closer to the AP or install a second one).
I didn't dig too far into it beyond verifying that uninstalling Alibaba fixed the problem. But it's super obnoxious that userspace apps are given handles on enough of the wireless stack such that they can break things.
Can anyone tell me why this causes the network issue? Don't understand how querying the registry can cause this.
The worst part is that I have no clue what causes it and no clue how I would even get started analyzing why the router crashes...seconds after the Ubuntu machine connects to the network, via WiFi.
I just gave up connecting to that network altogether. It works fine if I use it on other networks. lol
It is a bit of a trade off since if you want to see every possible available AP, even the shitty ones with signal levels at like -80, you can't be noisy on your own radio at the same time as you scan the band.
Remember it's a half duplex medium.
It does it even more if you hold down option and click the wifi menu bar, to get detailed signal strength/info on the AP you're presently connected to.
1. I can't see how you'd enumerate wifi networks without degrading network performance.
2. The user has specifically initiated a wifi-related action.
3. There's not generally any need to use that menu unless the network is already not working.