Comment by api | Hacker News Reader

Comment by api | Hacker News Reader

Nice.

It'd be neat to include options to simulate other pathological conditions often encountered in the field like:

- Multiple layers of NAT - overlapping un-synchronized timeouts are one thing ... but the REAL fun comes in when intermediate layers have the same IP ranges as IPs you are trying to reach on the "outside" of the NAT sandwich. All kinds of "interesting" things can happen, like the "software laser": http://catb.org/jargon/html/S/software-laser.html

- Stateful NATs/firewalls with bizarrely short connection / UDP association timeouts or that randomly forget connection state

- Shitty NATs/routers that go into bizarre failure modes when there are too many open connections, like forgetting early ones because they remember connections via a small ring buffer

- People who block all ICMP "because security."

- NAT + short DHCP leases = musical external IP addresses

- Small MTUs (<1500) like those imposed by PPPoE and other nasty encapsulation protocols, sometimes without ICMP errors for big packets because some jackhole blocked ICMP "because security."

All these things are common in the field. WHYWHYWHYWHYWHY

My favorite enterprise network pathologies:

- Randomly block specific ports "because security"

- Limit RPC port ranges to some random small number.

- Strange network optimizations. Example: Optimize TCP Window Size to support NT4 clients on 56k frame relay circuits.

- Very Slow DNS response

- Optimize core switching rules to fully utilize switch CPU. Avoid configurations that take place in an asic.

- Long DHCP Leases + Undersized DHCP scopes.

I also forgot TCP through TCP tunnels, which causes strange timing and flow control behaviors due to never actually dropping packets on congestion and double-ACKing.

It makes me wonder at how Skype manages to tunnel though almost anything.

Primer I wrote a while back: https://www.zerotier.com/blog/?p=226

Skype has abandoned P2P, though I'm not sure why. One possibility is that MS now has middle boxes deployed at so many interchange and peering points that there's no benefit in maintaining the added complexity. Another is that it was to comply with surveillance/tapping requirements.

IIRC they said that it's because of the shift to mobile. You can't really do P2P when the majority of your clients sit on phones.

EDIT: [I can't reply to the comment below, so I'll add here] It's most likely not a technological problem, but rather than data usage is limited on mobile and you don't want your user to pay for traffic they didn't use.

I keep hearing a categorical "you can't do P2P on mobile." I intend to fling myself at this problem like a drunk seagull soon, so I'll blog on it. I think it's possible, but with a number of special considerations around wakeup-quantization and general power management, some protocol augmentation, and a high tolerance for nodes appearing and disappearing.

Primer concept: http://en.wikipedia.org/wiki/Interrupt_coalescing

The problem isn't power (well it is; but that can be overcome), it's spectrum and pricing.

Pricing: Most people in the US are charged for the data they use on their mobile devices, and thus would not want P2P used on their phone because it costs them money.

Spectrum: P2P is not a very efficient distribution model in a world where most clients are on asynchronous connections. Asynchronous connections exist because transmission spectrum is limited, so to maximize spectrum usage, telcos allocate more spectrum for downstream transmissions than upstream. But if everyone's phone is chattering all the time with P2P traffic, you're going to saturate the spectrum and reduce overall data speeds. This is why mobile is still charged on a usage basis: it discourages overly chatty applications.

Good point on spectrum.

Verizon and AT&T monitor for super-chatty apps and in some cases can rate-limit or otherwise impair them.

P2P doesn't necessarily mean cooperative relaying or swarm distribution like BitTorrent. I agree that those applications are mobile-unfriendly with current batteries and cell networks. It just means you are talking directly to your peers instead of back-hauling to the cloud. You can have P2P where the only traffic you handle is your own. In that scenario total aggregate bandwidth shouldn't be that different from a cloud-backhauled app -- the only difference you'd see is in how many endpoints you're talking to. So instead of seeing 25mb transferred to/from one IP, you'd see it sprayed across a few dozen IPs.

The big hurdles I see are (in no particular order):

- Connection maintenance and keep alive. Basically this is bad, so you want to be more aggressive about shutting down unnecessary P2P links on mobile than you need to on desktop/server. Keep alive requirements generally suck anyway, and are one way NAT murders kittens.

- Restrictions around background tasks on mobile OSes (iOS is particularly onerous).

- Squelching inbound traffic from badly behaved or broken peers to avoid inbound flooding.

- Battery life and related concerns.

I understand it this way: cellular networks currently have their own "netiquette." It includes things like don't be too chatty, try to coalesce instead of spewing packets at excessively random times, etc. These things are less important on wired networks since they don't have the same resource constraints or bandwidth issues.

I guess a related question is why you would do P2P and not backhaul to the cloud? I can think of many:

- Reduced latency for things like AR and VR where latency matters a lot.

- Reduced bandwidth cost due to lack of back-haul, and back-hauling a pic being sent between two people in the same city 2000 miles to a cloud server is just offensively stupid anyway.

- Privacy and security.

- With P2P you could have a more open app model where apps aren't wedded to proprietary cloud infrastructure. They still work even without someone's cloud, etc.

If you're talking about P2P over cellular, I think it's just in general a bad idea. The way cellular networks are operated, you can't do device-to-device connections - they have to go through the tower for a number of (very good) reasons. That's where the latency comes in, and the wireless transmission latency can be an order of magnitude higher than the latency on a cross-country connection. Furthermore, the resource that is constrained is the amount of available spectrum - so we need to optimize for that. Bandwidth is effectively infinite from the tower to the cloud, but bandwidth from the tower to the device is the constraint.

P2P over non-cellular (i.e. Wi-Fi or Bluetooth) where you can actually make a direct device-to-device connection seems like it may have some use cases (messaging apps, etc.) But they're edge cases and by no means a common use case because it just isn't reliable enough.

Alex Pankratov, the developer of Hamachi, discusses his approach here: https://swapped.cc/#!/hamachi

What do you mean randomly block specific ports because "security"? All ports should be denied access except ports which have a justified business reason. Got a web app? The only thing open should be 80/443. There's no reason for SMTP to be open on the web server. Anything doing mail should be on its own MTA server. Least functionality per server. That's not even security. That's just good system administration.

At one of the enterprises that I've had the pleasure to work at, the network guys would randomly come up with some "concerns" about your firewall requests, and would just not include certain parts of your request.

So you might request ports 4000-4100, and find that 4007 is blocked, "because security".

I'm pretty sure the reality was that the firewall rules were a big hairball, and they were stepping in some other rule out in place a long time ago.

Don't forget jitter, and simulating an ISP that will shut off your DNS traffic at random (literally anything DNS, not just their own servers).

I forgot about DNS munging shenanigans. Not only do some people selectively block DNS, but there are also cases where middle boxes and proxies pretend to be remote DNS servers but really aren't... or rewrite DNS traffic, insert fields, etc.

I take it you've seen core switches on fire off the Jupiter gate?

Juniper gate.

:)

Juniper leads to gin?

Tannhauser gate

>WHYWHYWHYWHYWHY

Are there any network administrators who don't consider NAT traversal to be a security breach? Making it difficult would seem to be a feature, not a bug, in most enterprises.

NAT traversal is a security breach that became a de-facto standard and is part of the SIP, STUN, and IPSec (extended) standards among other RFCs. It exists because NAT itself is an abomination.

Reply on news.ycombinator.com