'Anti-authority' tech rebels take on ISPs, connect NYC with cheap Wi-Fi | Hacker News Reader

'Anti-authority' tech rebels take on ISPs, connect NYC with cheap Wi-Fi

cbc.ca

417 pointsstandeven228 comments

There's an impressive amount of negativity in these comments. A community-run mesh network? HN should be all over it, but it appears to have turned into a bunch of network engineers telling us why they are very smart and it won't work, or is "insignificant".

More power to you, NYC Mesh.

It's OK to be optimistic about something different. It's also ok to discuss the reality, that's not necessarily negativity towards a project. Are we only supposed to discuss one side of the argument and ignore reality just because it makes us feel nice inside?

There are bazillion reasons that this is nothing more than a cute side project that will fall on its face when its time to scale in any meaningful way. Unless they have discovered a new technology they are not taking on ISPs by any meaningful way. Especially in NYC.

There is a difference between, just "connecting to the internet" and having a "reliable connectivity to the internet", the difference is a massive investment in infrastructure. Wifi p2p mesh networks don't work on magic. They are very unreliable and hard/impossible to scale to 100,000 users, let alone millions of users. In some places, wifi p2p connection hardware actually costs more money than a traditional wired network.

Mesh networks have been around for a while, technical and reliability aspect of this is well understood within the networking industry. There is a reason why it's not widely used, especially in cities. In remote areas, this kind of network makes sense, if you don't have a choice or an alternative.

Don't brush of educated criticisms from people within the industry. I own an ISP for 5+ years (mostly wired). I am not in US and I am not at all worried if this kind of network comes in my country/area. We are not worried because we have tried and deployed this kind of network in cities and they are extremely unreliable and costs real money.

I also don't understand the "Anti-authority" angle of the argument, unless I live in a different universe, they still have to get license/permit from the local government to do this business (non-profit or not), they still have to follow the same rules enforced by local and federal government when it comes to log/customer information sharing to LEO. So not sure how they are Anti-authority? Unless they mean anti-establishment?

Once upon a time the network, storage and data center guys thought Facebook/YouTubes one to many broadcasting of content would never scale too. Now they fall over themselves to work there.

They scaled by making a huge investment in their network. They didn't make a fundamental new technical discovery that didn't exist before. They had money, they threw money at the problem, as expected it worked. The problem was an investment (which takes time), not technical.

https://peering.google.com/#/ https://code.facebook.com/posts/565767133547005/steering-oce...

Any ISP (if big enough), can host their own Google, Akamai, Facebook, Netflix nodes on their premise.

I think pavs is referring to physical limitations of the technology that prevent it from scaling.

It has been a few years since I learned about Guifi.net (non-English site; English portal:guifi.net/en/node/38392) in Catalonia/Valencia - a 2004 startup. WP says that (as of a year ago) it has 'over 33,000 active nodes and about 46,000 km of wireless links.' It's described (in English) in this 2016 paper

http://dsg.ac.upc.edu/sites/default/files/dsg/acm-sigcomm-ga...

as 'known to be the largest community network in the world ... and is still growing while successfully evading the collapse of the tragedy of the commons. Undoubtedly, this is due to the governance model that the foundation has developed....'

Since it's been around for a while, and still viable, that says something about the idea worth attending to.

More info: http://www.dtic.upf.edu/~jbarcelo/tmp/guifi.net.pdf

The weirdest part of HN for me is the optimism. Engineers are pessimistic because engineering is hard and physics is unforgiving. Engineers like to point out things won't work because it probably won't and engineers like to be right. Mesh networking is incredibly hard to get right to the point where I wouldn't even say anyone really has ever gotten it right at scale.

It would get a lot more positive comments if it was called what it is: A low-bandwidth mesh network. It's nice, but not going to replace ISPs anytime soon. It consumes more energy, it's slower, less reliable and higher latency than regular wired/fiber networks.

These things don't scale well and all those radios can be universally jammed by a single drone over the city. Not a great backup plan if you're anti-establishment.

I'd love to partake in such a project but keeping your expectations grounded is very important.

What's so ironically funny, is that this article is also on HN's front page:

https://www.newyorker.com/culture/cultural-comment/brilliant...

HN Comments: https://news.ycombinator.com/item?id=16980503

The jealousy amongst peers for non-recognition.

"More power to you, NYC Mesh." is the correct answer. Thumbs up tlrobinson.

Shameless plug: I run a website that helps people do this kind of thing: https://startyourownisp.com (although it's free and ad free, so not much of a plug really.)

Also have a matrix chatroom on this topic: https://riot.im/app/#/room/#startyourownisp:matrix.org

The group also knows there will be growing pains as they challenge the status quo and that it's only a matter of time before the big ISPs take notice, which could bring new challenges.

I'd like to imagine this would be competition on price and service. Sadly, that won't be the case. They can expect everything from spurious CALEA requirements to rooftop lease shenanigans to unwarranted radio interference claims.

Eventually, though, this mesh model will take over the world, from both a network perspective and a commercial one. We just have to squelch our authoritarian impulse.

Of course, technical challenges may still exist, but those certainly don't discredit the model.

I don't think mesh makes sense from an economics perspective. The number of edges scales O(n^2), where n is the number of people in the network. That's not very efficient. Even if you think wireless internet is feasible in dense urban areas (I think this is still an open question) it seems like a star topology connected by wireless backhaul to fiber makes more sense. This scales O(n).

That's why we build a hybrid network with p2p and p2mp backhaul.

> The number of edges scales O(n^2), where n is the number of people in the network.

I don't see how this is true for any reasonable definition of "edge". It's certainly not the case that every node directly communicates with every other node; if they did, you wouldn't need a mesh.

A mesh network is defined as a network where the nodes connect to as many other nodes as possible. In the limit this is a complete graph with O(n^2) edges. It's true that in practice you won't have a complete graph but it's still less efficient than a more hierarchical, tree-like network structure.

Am I missing something here?

Let me approach your math with a reasonable argument:

Assume a 1 sq mile area (1mi x 1mi).

Assume each node has a 50 yard diameter reach.

If you space a node every 50 yards, you will have 35 nodes on a side, for a total of 1,225 total nodes.

Each node will only be able to physically connect to 3 other nodes on the edges, and 4 other for interior nodes.

That means the node connectivity is only .00326 for an interior node.

The longest path is diagonal across the square. Given the 35 modes on a side, and A^2 +B^2 = C^2, that means 35^2 = 35^2 = C^2. And c^2 = 2,450, meaning C = 50 (rounding up).

So I’m not sure how O^2 even applies in this example. Yes, it’s theoretically a max, but theory >< the real world.

Is this not correct?

the fact that a wifi network node only covers a very limited area is sufficient to show that your theoretical limit is never going to be reached -> by a large margin. You can probably assume that in a dense network, every node will be reachable by every other node through at most 7 hops. But thats it [1]

https://en.wikipedia.org/wiki/Small-world_network

Aren't mesh networks higher risk for security? My initial reaction is that while ISPs collect a lot of information about you a mesh network allows an unknown amount of hostile actors to collect information (unless it's set up like Tor, but that has issues too).

At least with the standard ISP model there is some accountability and privacy rules could be implemented/enforced.

Edit: I think I might have been misunderstanding what mesh network was referring to - this looks like a wireless access point?

Well, the content of your requests will be encrypted, that's what SSL is for.

Also, individuals who are concerned about this would be free to purchase a direct line via an ISP.

Website fingerprinting is still possible.

DNS over https

The radio link is still encrypted so an attacker won’t get much out of it. In fact, it’s a lot easier to tap a cable, DSL or fiber line (which operate without any encryption unless you put IPSec on top manually) than a mesh link which would be using WPA2 Enterprise with perfect forward secrecy.

Is it encrypted?

https://nycmesh.net/faq/#secure

That seems to suggest otherwise? Unless I don’t understand what network traffic within the mesh means.

Seems like anyone with access to the mesh has the keys to decrypt anyone's traffic, but it's still encrypted from an outsider's point of view.

Yes, but anyone can become an insider (and an attacker would). Probably an okay tradeoff given decentralization of network access and maybe could be made better if internal traffic was encrypted somehow, but does seem higher risk.

Mesh networks are fine for peer to peer comms inside the mesh. But if you want Netflix you need an uplink.

I really don't understand the HN reaction here. Why all the presumed negativity? This is tech, the best encyclopedia in the world is built on volunteer work and donations. A genius builds the de-facto version control system on his own in 3 months for fun. 2 guys in a garage build revolutionize search.

Certainly there are logistical differences between these things, but I don't understand why one would assert with confidence that such a project (that already seems to be working) couldn't work.

I love HN but there is a very healthy level of suspicion for developing different ways of attacking already-solved problems. May not be the best example but look at the comments on announcing Dropbox.

Doing > Wanting to do in nearly every instance. So to borrow from popular culture, Just Do It, and Haters Gonna Hate.

Yeah, I just figured that a pro-anonymity, pro-startup, pro-charity, anti-monopoly, anti-comcast community would see this as a win on all fronts.

At least that was my read on this community.

My reading of the negativity in this thread isn't "here is why I am opposed to your project" but rather "here is why your project seems like it will fail". The latter isn't anti-startup or pro-comcast at all.

...but why The Facebook? We already have MySpace.

...but why Dropbox? We already have google drive/OneDrive/WebDAV.

...but why an Airbnb? We already have BnBs, hotels, and hostels.

Which are all questions that the founders of Facebook, Dropbox, and Airbnb needed to answer to be successful. Questions are a contribution.

And those commenters have a different understanding of what success is for a project like this, then the team of volunteers working on it, and the customers who are using it. The success of this project is entirely defined by those two groups. I am inspired by this project and see a great deal of "good will" value and the potential to serve as an emergency backup internet in the increasing hurricane season prone NYC floodplain is an additional value stream for sure.

It's easy to imagine reasons something might fail, even for something that actually will be a success.

Compared to most funded startups, this idea seems extremely practical and feasible (1.6 million people in Manhattan alone). Wireless internet is already a successful business model (Monkeybrains in SF), and extending it with mesh may add latency but I'm not going to pretend to know more about it that the guy who is currently doing it...

> May not be the best example but look at the comments on announcing Dropbox.

This is the opposite direction. Mesh NYC is saying that ISPs are useless because anyone can string up some APs and get access to the Internet.

Remember the guys who had it easy learning at university? Imagine you are one of them- good at using existing rules, good at ingesting and reusing existing knowledge. But deep down know you are unable to recombine, think outside the box or just experiment childishly - to truly innovate - then those tinkerers poking the unknown could render you a obsolete dinosaur any day.

Which explains while there is always a caste of "professionals" who for the sake of "security"(job) - defend against the relentless wave of people who unleashed could undo there little safe space in life. To be fair- most of them have familys, so they are not doing that for greed- but because out of vulnerability.

Or, just maybe, they've poked an antenna up in the air in NYC and noticed the 5GHz spectrum is totally polluted and barely usable.

You want WISP competition (co-op business model or not), then lobby the FCC to free up some spectrum for it. Cheap gear is here, spectrum is the scarcity.

You are right. http://reboot.fcc.gov/spectrumdashboard/searchSpectrum.seam

So here is the plan- we jam a frequency of a local TV-station, nobody views, who is just sitting on the spectrum for license worth. After proofing for one year and one day, that nobody listened to it and nobody cared- we pronounce that this spectrum is unclaimed and should be returned to the government to be auctioned off.

Under normal circumstances this should not go through legally- but hey, there is booty, there is a government in debt and there are interested parties.

Its the perfect storm and just having legal rights - didn't help the native Americans either, didn't it?

Come to think of it- maybe planning the heist in public was - ill advised.

"It is difficult to get a man to understand something, when his salary depends upon his not understanding it!" -- Upton Sinclair

I wonder how much it's because so many Silicon Valley salaries are dependant on people assuming that replicating what they do is way harder than it really is? What happened to Britannica or BitKeeper or AltaVista might just happen to the ISP industry - or any other Start-up-de-jour...

The problem here is that those of us who have a positive presumption have almost nothing to say. The success speaks for itself. Positive comments are generally avoided on HN, unless they actually have a something to add to the discussion.

To contrast, those with negative presumptions have plenty to say, and incentive to say it (arguments are a form of discussion).

This leaves us with a skewed representation of attitude.

Because this requires brick and mortar industrial hardware to be logistically salable, lacking that it becomes vaporware.

I think an unconscious bias may be that a bad but free connection that can negatively impact the perception of web solutions from casual users, which in the end is the revenue base for everyone here involved in B2C business models.

I wonder?

Unreliable but redundant commodity grade hardware pretty much allowed "cloud computing" to eat 99.9% of "big iron" sales.

I strongly suspect what Google does internally with software defined networking looks a lot closer to what these people are doing than the traditional "buy a small number of powerful-but-expensive entrenched vendor network hardware" approach.

This mesh is kinda the network equivalent of autoscaling or autorepairing cloud computing strategies that are so common these days. Once this grows to 2 or 3 IXP connections it won't surprise me much to find they stomp the traditional ISP model reliability-wise, and if they can build that "virtually", where they don't need much more than a bunch of somewhat transient mesh participants with a few medium reliability "super nodes" which are mostly completely remotely configurable...

This stuff is cool and maybe useful in far-fetched scenarios, but it's not going to replace ISPs anytime soon. Mesh networks don't make economic sense in most circumstances.

Also, I'm guessing that users share the same IP address, which can cause problems. Plus, if one user is doing something illegal and causes the supernode to get shutdown then no one on the mesh network will have access to the internet.

Addition: quotes like "The internet doesn't really cost you anything, it's just the connection [that has a fee]... Nobody owns the internet, there's no one to pay." Are shockingly naive. Who does he expects pays for the fiber that runs across the country and along the ocean floor? The marginal cost of a new user on the internet is effectively zero, but that doesn't say anything important. The infrastructure that makes the internet work is enormously expensive.

We are replacing ISPs in NYC for our members. This is now, not the future. The main thing holding us back is scaling up the number of installs.

All networks, including us, add to this expensive infrastructure. The difference with our infrastructure is it is owned and shared by members.

Very cool. Is this a customer coop or a non-profit (or something else)?

It's a non-profit

Maybe you could get Ubiquiti to put up some money for promotion around the city to boost installs.

> The main thing holding us back is scaling up the number of installs.

So the main thing holding you back is time, money, resources, access, customers, and logistics?

Mainly logistics. We've had a huge influx of people wanting to be connected since December, so we've had to add stuff like a ticketing system, inventory tracking and also we're training people to lead installs. It's normal scaling problems that fast growing organizations have.

Sure thing. And I offer free WiFi access to all my household members, houseguests, and next-door neighbors. The main thing holding me back is scaling up the number of installs too. I'm hoping a billion dollars of donations can square that away for me and my wider-area neighbor.

I get that you're joking, but this is basically how public wifi works.

A bunch of businesses and nonprofits offer free wifi to their customers, and over time you build up a big enough blanket that pretty much anybody can walk into a coffee shop or library and check their email.

Tons of security risks, of course, but... I mean, I've used public wifi before during personal emergencies, and I was pretty grateful it existed. If you're gonna pick a comparison to be derisive with, maybe don't pick something that's widely useful and appreciated?

Comcast has even turned this into a selling point (I think somewhat unethically) by turning all of their customer access points into semi-public routers for other customers. It actually seems to scale pretty well.

Comcast doesn't do it for free. That is financed by bill-paying customers

Yes?

Comcast provides its wifi by allowing you to connect to any other customer's router. (Roughly) a mesh network provides its wifi by allowing you to connect to any of the other nodes within the mesh.

What's your point? Is the money/infrastructure less legitimate because it was donated? Is the public wifi down at the library fundamentally worse because its cost wasn't bundled into the price of a coffee?

Hardware is hardware. If it works, who cares where it came from?

Obviously you're being sarcastic, but they're not just dumping netgear routers on walls and saying "TADA! MESH!"... They're actually using industrial grade equipment to do this.

I missed the "industrial grade equipment" part in the arcticle, TBH. In fact I saw no real technical details at all. "Mesh" isn't even really defined. You'd think they'd at least mention what firmware they're running on their gear to implement "Mesh".

This is essentially what I do. I do have my housemates pay a share because they have access to a hardwire connection, but I have an open AP which my neighbors, guests, and anyone within line of sight down the street use. I have plenty of bandwidth that goes unused so I don't mind - I pay to have fast bursts of traffic, so I don't have a problem letting other people use what I don't. The only thing holding me back from having more people use it is locations nearby for repeaters. But I may put more effort into getting neighbors interested in hosting repeaters so that may change.

>"This stuff is cool and maybe useful in far-fetched scenarios, but it's not going to replace ISPs anytime soon. Mesh networks don't make economic sense in most circumstances"

Far fetched you mean like getting instantly connected to the internet for a fraction of the cost and with better bandwidth in major city?

You realize that mesh is just for the access network correct? There's still fiber between them and their transit providers. Mesh makes a lot economic sense compared to the costs of trenching and laying fiber miles.

> I'm guessing that users share the same IP address

I don’t see why that would be the case - you can give every user their own public IP directly. I would imagine they’re giving everyone an IPv6 /64 and a carrier-grade NAT for IPv4 (as it’s really hard for any new player to get usable amounts of IPv4 space).

Many WISPs have their customers share the same IPv4 address, which means when one customer gets the IP banned from some site, all of the customers are banned until the WISP resolves the issue. I don't know if that is the case here, but it is common enough that I think it's likely.

How are they at any greater risk than any other ISP?

This can work for small scale networks but it needs to be a paid service model to scale it(at least for usable performance).

A spine/leaf arch would be great for this imo. The leaves can be local cells of wifi networks. The spine can be operated by payment collecting individuals that maintain a high speed(10g+) and low latency uplink to the internet and network gear that can handle the traffic from the leaves. In spine leaf, leaves connect to multiple spine nodes as well (don't have to worry about spine operator reliability too much,could potentially load balance over spine)

That being said,I hope any modern wifi network assumes wifi encryption is not reliable and instead implements a layer2 or 3 tunnel (macsec/.11ae and wireguard respectively)

When you compare comments here a few themes emerge:

1. The internet has been captured by the telcos, who operate the internet like telcos have operated the circuit switched voice network.

2. We are all overlooking the fact that smart radios make spectrum ownership obsolete. That means that the potential of mesh wireless connectivity is orders of magnitude larger than can be realized in current unlicensed spectrum.

3. The assumption that anyone is owed a business model in internet service is going unquestioned. Of course it costs a lot of money to run the internet the way the Bell System ran phones. But there are plausible alternatives, not all of which are compatible with investor-owned near-monopoly "markets."

This is an economic tragedy and a case of local optimization holding a global optimum hostage. ISPs are a sweet business, in a way that costs the rest of the economy dearly. Kill the incumbents and reap an across the board boost in GDP.

This sounds neat. Is anyone working on something like it in the SF bay area? I know there's Webpass and Monkeybrains, which are great, but I'd love to be part of a community-owned, actual mesh network.

There's the People's Open Network: https://peoplesopen.net/ co-ordinated by the SudoRoom folks and the wider community.

Not "community owned" per say, but Facebook has deployed a Mesh network in DT San Jose called Terragraph.

Can anyone comment on the utility of mesh networks in rural areas, where the big ISPs say "Sorry, not interested", and the distances between potential nodes is high? I have read articles in the past about hardware hackers setting up systems to broadcast WiFi signals for exceptional distances. Could a system of mesh networks provide rural America (or other countries) with low cost / high speed internet service?

I have set up wireless bridges that could supposedly under optimum conditions, connect to one another over 11 miles.

So with line of sight I know you could 'beam internet' from either a plain old ISP connection OR from an internet exchange.

This was consumer grade equipment too, I am sure there is much much stronger out there.

As far as the potential of mesh in rural areas, I think one of those two options should be workable. Of course there are costs and geography that may hinder.

Point to point wireless links can definitely be used as long as you’ve got line of sight between the endpoints.

In Catalunya we’ve had such a network for many years and it is specifically used for providing connectivity to rural parts of the country. It’s called Guifi.net and has over 30,000 nodes. I was surprised it wasn’t mentioned in the article actually.

It was mentioned.

> While there are mesh networks dotting the U.S., she says the best working example of what mesh technology can do is in Spain. Guifi.net has more than 34,000 nodes covering an area of roughly 50,000 square kilometres across the Catalonia region.

Guifi.net is actually mentioned halfway through the article.

This is a great ideal. Sticking it to The Man is never a bad thing :) The question seems to be one of level of service and reliability. Collectives (if you will) can certainly work. But at this scale? And necessity?

The ultimate benefit, I would think, would be to cause downward price pressure on the establishment ISPs. But wasn't that also Google Fiber's quest? Which we know didn't last long. Profits or not, there is still a certain level of cost to maintain sustainability of the model.

I hope I don't sound cynical. I honestly just curious and will to share my thoughts out loud.

Nice. Our 200+ unit condo building in Portland just got gigabit internet via line of sight radio on the roof from another building with fiber.

This is a trend that will definitely take off and removes the huge barriers of entry that laying cable requires. 1ms latency too.

I understand you must be in USA, but I'm from Canada, and this trend is certainly very popular in most parts of Ontario. In the USA it's also catching on, have a look at Webpass. The Canadian equivalent would be Cloudwifi who offers gigabit service most condos and apartments built after 2005. http://cloudwifi.ca

Oh, on average these next generation networks have much lower latency than traditional cable and DSL lines.

I've got to know more about this. Who set it up? Local start up? Bored engineers?

Wave G. I think its a spin off a small isp.

It looks like they use a lot of Ubiquiti gear, which I'm a fan of (at least as a "prosumer", haven't used it professionally)

I'm curious about the technology they are using. Is it an actual mesh network or point-to-point wireless?

We are a hybrid of mesh, p2p and point to multi-point (sector antennas). Whatever is most appropriate to give the best connection for reasonable price.

Considering the 5100 to 5850 MHz band noise floor in metro NYC, it's a minor miracle that outdoor, roof mounted, PtMP 802.11ac based systems work at all. (Or Cambium PMP450i in the same bands, etc).

Yes the noise floor is bad, but we get about a 2 mile range which covers a huge amount of people in Manhattan. The Ubiquiti sector antennas use TDMA so it is not standard 802.11ac.

Screenshot of a UBNT p2mp at 2 miles in Manhattan please. The dashboard will show the distance. Please post.

https://nycmesh.net/panorama/1.7miles.png

That's a LiteBeam Gen1 installed in the east village on 5th St connected to 375 Pearl St downtown. The gen1 actually has a stronger signal than the gen2, so this is a bit better than our usual connection.

Seems to be an actual mesh network, based on the BMX6 mesh protocol [0].

[0] http://nycmeshnet.github.io/faq/

Here's a map of their network: https://nycmesh.net/map/ (uncheck everything except "Link" to see the network)

Down here in South Africa we have a similar network. We have had quite shite connections down here for so long that a mesh network was one of the few ways to get a good connection. Link for our map of Cape Town. https://wind.ctwug.za.net/index.php/nodes

"The network connects directly to the internet backbone, so we do not rely on an ISP."

That makes no sense. I doubt they're a tier-1 backbone themselves, they have to be purchasing transit from an ISP. It feels like lots of the docs are written by marketing-type folks. Also there's a weird diagram that insinuates NAT happens before bgp peering with the not-ISPs they buy internet from. Weird shit.

Looks like point to point in the video

I love seeing the growth of mesh network.

“We don’t have any use cases that show how nice it can be if you have enough users.”

They are going to need software. The supernodes are still a symptom of the thinking that you need the signal to go to another stage in order to communicate locally.

I think there should be more software that works on a local network, like things used to work before broadband.

This is what I’m talking about:

https://www.youtube.com/watch?v=pZ1O_gmPneI

"Anti-authority" seems like a pretty good marketing campaign that WISPs have figured out.

I mean: good for them.

Actual ISP network engineer here: I'm sure these people are having fun and all, but it's insignificant in the larger scheme of things.

(correction to original post: I did not at first see an ASN for them, or announcements of their own IP space. They have an ASN and announce two /24s, and peer at one physical location in metro NYC).

Real five nines network infrastructure takes significant capital investment.

This part is so much unadulerated bullshit:

""The internet doesn't really cost you anything, it's just the connection [that has a fee]. So however you can get plugged in — then you're on the internet. Nobody owns the internet, there's no one to pay.""

Actual, reliable internet costs real money, both in the salaries of people to engineer and architect it, salaries for 24x7x365 clued-in NOC staff, equipment, salaries and expenses for field technicians to build it. And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each (do you really want to deploy something in the year 2018 to take several full BGP tables that doesn't have a 4 million FIB capacity?), etc. The Internet is a significant construction project at OSI layer 1, whether you're putting PTP radios on rooftops, running aerial fiber, or underground fiber. Otherwise you're just piggybacking on something that another, larger entity has already built.

I give them an A for enthusiasm and effort. It just needs to be channeled the right way so that they can figure out what it actually costs to run a reliable ISP. I'm all in favor of new startup ISPs.

Our ASN is AS395853. Our peering info is here- https://nycmesh.net/peering/ We have v4 and v6 space.

Our uptime when connecting to supernode1 is significantly better than Time Warner Cable as many of our members have reported.

We are already established. This is something we have already done! It isn't impossible.

Our IXP is DE-CIX, the worlds biggest IXP. We are currently in one data-center/IXP and talking to a few more colocs.

>"Our IXP is DE-CIX, the worlds biggest IXP."

Yep, DE-CIX is great and 32 Ave of the Americas is one of the most wired buildings in NYC. Great choice. Kudos on you folks, this is a great initiative. Reading this article made me happy. And I'm glad you are getting some press!

So what's the Wifi NAME in Manhattan?

The public access point SSIDs are usually "-NYC Mesh-" or "nycmesh" followed by location.

How do you deal with malicious networks broadcasting under the same name? Do you use WPA2, or is data sent naked over the air?

How do you deal with someone making a Facebook with your first and last name?

Same answer.

Inb4 everyone renames their networks to nycmesh-something and MITM unknowing crowds

why would you trust random people more or less than other random people? That's crazy talk.

Are you using Ubiquity HW? If not, why?

They are cheap as hell and awesome.

Actually - what you should do, is have a customer pay a fee of $100 to connect to the system - and take that $100 and buy another mesh was with each customer on boarding.

https://unifi-mesh.ubnt.com/#products

I think you misunderstand the goals of this community-run mesh. The FAQ has some info on why they don't charge, who runs routers, and what they're trying to accomplish.

Digi Desert LLC, which existed in 2010? Looks like that's the AS of some entity that predates the existence of your project and current public relations effort.

You announce a whopping total of two v4 /24.

Kudos for having enough clue to know that you needed to establish a presence at a major IX, and actually doing it, because it looks like you're adding peers. But your actual network presence is minuscule.

How do you intend to compete with the six NYC based companies I can think of off the top of my head that are putting fiber fed, $9,000 to $20,000 5 to 10Gbps 71-86 GHz PTP links on rooftops to build their own backbones, when you're playing around with 1000BaseT to the roof and AF24s?

You have high uptime? Do you have any of the following, because some of your much larger competitors sure do:

sites with parallel A and B side power systems

-48VDC rectifier + battery systems sized for 24 hour runtime at load

diesel generators

propane generators

generator resupply contracts

chassis-based routers with hotswap fan, N+1 power supplies, dual redundant routing engines

pair of identical core routers

singlemode fiber to the roof

-48vdc power to the roof

ironclad rooftop lease agreements with building owners, drawn up by professional telecom/real estate lawyers that run for 5+ year terms

a 24x7x365 NOC staffed by live humans

Your original post was interesting because it raised legitimate questions about their operation. They answered those questions.

This follow up comes across as though you've drawn a conclusion and are now arguing towards that predetermined conclusion while ignoring the additional information provided. Plus your argument has kind of devolved from talking about specific concerns to throwing criticisms at the wall to see what sticks.

You don't seem to be arguing in good faith.

I'm sharing some harsh reality with them: The ISP market in NYC is highly competitive. If they want to be serious about it, it's not going to work as a nonprofit. I'm trying to tell them bluntly about what sort of infrastructure their competitors operate so that they can get an idea of the actual capital expenditure requirements involved in architecting/engineering a MAN-scale, five nines ISP composed of point to point wireless links.

Once upon a time there was a company called Microsoft. They hated free software! It costs millions to write an OS! Why is it free?? They hated the free guys so much. Then the free software got better. It got so much better than what you would have to pay for from Microsoft! Eventually Microsoft said oh well, and built the free software into their product. They even contributed back to that free software and everyone learned that you can have free and paid and as long as we all contribute the world is a better place. Thank you.

Once upon a time there was a chipmaker named Intel. Sun decided to open source its SPARC architecture. Other folks tried to design their own FOSS CPU architectures. Intel still dominates.

FOSS zealots like to point to the success of Linux over Windows (which is restricted to the server market, I might add), but there's little evidence that the FOSS philosophy is effective when it comes to physical infrastructure. Software requires bytes and labor. Bytes are cheap and labor can be donated or paid for by companies. Hardware requires fabs and factories, which are expensive.

The reality is harsh but the discussion doesn't have to be.

Well it seems to be working fine for them right now.

So I guess the points you brought up don't matter.

I don't think you get what we are doing at all! Anyway we are in 375 Pearl with a lease and it has most of your long list, diesel generators etc.

We compete on price (donation only)

Keep up the good work. Love these kind of projects!

Hello from just across the Hudson River! Keep up the great work. You are finding a way, despite what doubters and naysayers think! You will know what is possible by trying.

https://www.peeringdb.com/asn/395853 NYC Mesh

What is your problem??

You're wasting your own time and those of your "customers" unless you have at least $750,000 to establish a serious presence, in my opinion. Amateur-hour WISP stuff is fine in a rural area. You're pretending to be a highly reliable capable ISP, and will eventually either overextend yourselves, run out of rubes to fund you as a nonprofit, or once you reach a size much bigger than you are now, get stomped on and obliterated by a much larger competitor that overbuilds your entire network with carrier-grade infrastructure and takes all your revenue.

We are already self funded by individual donations. Every question I answer seems to bring up new objections.

This thread has to be one of the most overtly hostile threads I've read on HN in a while. What is your concern here? Why do you seem to feel they shouldn't even be trying?

What ISP are you affiliated with?

We bypass traditional ISPs by connecting directly at an internet exchange point (DE-CIX IXP) and peering with other networks. This is basically how the internet is formed, by networks peering with each other. We don't need this ISP layer.

I think the OP was asking what ISP walrus1 works for, given they've already identified themselves as a network engineer.

I was referring to walrus1. Sorry for the confusion. Y'all keep doing what you're doing. I have big hopes for efforts like yours.

Thanks!

I'm pretty sure the real secret is ensuring your customers have no alternatives. Spectrum may or may not have all of that gear and they still have given me outage and throttling issues for years. I don't want somebody to compete with the terrible ISPs out there today - they're fundamentally the wrong kind of organization. I'd rather have honestly crappy service than an opaque monopoly dictating my access.

> Actual ISP network engineer her

So then you must intimately understand the depth of the lies and anti-consumer business practices that spur such networks.

> Real five nines network infrastructure takes significant capital investment.

5-9s is 5 minutes of downtime a year. I've never had a commercial, let alone residential ISP hit that mark.

> Actual, reliable internet costs real money, both in the salaries of people to engineer and architect it, salaries for 24x7x365 clued-in NOC staff, equipment, salaries and expenses for field technicians to build it. ...

Going back to my first point, almost no one is unwilling to pay for their internet connection.

They're unwilling to pay less to get TV service they have no use for and internet than they are internet alone.

They're unwilling to look at the cost of network access in other countries and be happy when comparing that to the cost of internet in a major, densely populated area in the US.

They're unwilling to accept transfer caps and accept the lies they they help the network by making people share when they know that transfer is irrelevant and bandwidth is the limited commodity.

They're unwilling to accept lies of bandwidth speed that isn't attainable even under the best of conditions.

They're unwilling to accept shady and underhanded business practices such as random, "accidental" charges.

> It just needs to be channeled the right way so that they can figure out what it actually costs to run a reliable ISP. I'm all in favor of new startup ISPs.

However, as an "actual ISP network engineer", you must understand the (very arguably unfair) regulatory issues that the big players have put in place to make this impossible to do. Google failed at starting an ISP, and it's not like they are tight on either cash or lawyers.

It’s pretty hilarious to hear a cable network engineer get on a soapbox about reliability.

As an enterprise customer, I only get 99% commitment, and frequently get credits. They’ve improved it in the last 5 years, but our whole region had an outage when the single fiber connecting to their core routers was cut by a house fire.

I'm no network engineer, but when you have a large network with lots of equipment, you want the individual components to have a couple more nines than the resulting network. Maybe that's what they meant.

Google did not fail at starting an ISP. They decided that the ROI for doing individual residential home FTTH was not worth it, considering the cost per house passed. Then they went and bought Webpass, which has a totally different condo/apartment based business model, which has been wildly successful.

Starting a small ISP based out of extreme dissatisfaction with huge, evil incumbents is something that I encourage any plucky upstart to do. If you don't like Comcast, Charter, RCN, etc, if you have sufficient clue, go do it yourself. Just don't try to do it with $20,000 and some good wishes, because something with much deeper pockets like Webpass will come along and obliterate you.

The point that I am rather bluntly trying to get across to these folks is that if they're going to invest significant amounts of their personal time/engineering resources into it (unpaid labor?), that they should treat it like a serious business and not a nonprofit. Raise sufficient funds, somewhere in the $500,000+ range to start, to build a network that actually does meet five nines through sufficient diversity in topology and routing.

Compared to random YC companies that seem to raise $1m to $5m to do some vaguely defined web2.0 SAAS thing, the amount of money that a group of people need to raise to build the core of a serious ISP is not huge. Elsewhere in this thread I threw out a figure like $750,000 in beginning CapEx if they're serious. For some good reasons, however, it's easier to get $5m of series A funding from a VC for a pure software development business than a wholly-owned-facilities-based startup ISP.

Even if you completely ignore things like colocation costs, IP transit costs, IX/peering costs, office rental, office utilities and overhead, running a serious ISP with live, clued in humans watching the network takes a lot of money per year for salaries.

3 to 5 NOC persons for 24x7x365 NOC phone coverage. At a certain point if you peer with the big boy ASNs they will expect a live human to answer the phone at 3am when something unusual happens in traffic between your mutual networks. That live human needs to be somebody who has the equivalent of 'enable' on your peering router and sufficient clue to wield it.

2 network engineering/architecture positions, multiply by expected fully loaded yearly salary and benefits package for each person.

at least 2 field technicians

vehicles, vehicle insurance, tools for technicians

At NYC salaries? In a greenfield scenario, I could easily spend $900,000 a year in just payroll and benefits, just to begin to attempt to duplicate what some other serious ISPs already have in place for manpower.

And again that is before you spend one dollar on equipment.

> At NYC salaries? In a greenfield scenario, I could easily spend $900,000 a year in just payroll and benefits

I kind of don't understand what you're getting at here. They're a nonprofit. The majority of their work is coming from volunteers. They aren't paying New York salaries.

This is like going to Linus and saying "You can't possibly expect Linux to be sustainable - do you have any idea how much money Microsoft pays its developers?"

Nonprofit/open models aren't less work than commercial ventures; the difference is that people and companies donate their time/money because they want the project to succeed. If they end up with a competitive enough product, then businesses will start sponsoring more development, because it'll be a cheap alternative to dealing with Comcast.

At that point, large businesses operating in the city will pay/sponsor New York salaries to maintain the parts of the project they find useful. Exactly like every other Open Source or community run project on the planet.

I have plenty of doubts about volunteer mesh networks, but I'm not so skeptical to say they literally can't work. I think you're looking at one particular model of the market and assuming that anything deviating from that model is broken, rather than just targeting different segments or trying a different strategy or even just accomplishing different goals.

> The majority of their work is coming from volunteers.

> the difference is that people and companies donate their time/money because they want the project to succeed.

And that volunteering time and energy are probably valuable then, especially if they can do the work of a real ISP.

I think one real question to ask though is what is actually more efficient—donating volunteer time and energy to a nonprofit, or donating money and having that nonprofit hire professionals instead?

GP doesn't sound like he is saying a volunteer mesh network literally cannot work, just pointing out that a network is inherently expensive to run, whether it is 'fully accounted for' on a payroll, or merely hidden in the cost of 'free' volunteering effort. It's an open question of whether asking for people's time or their money is better in the long run. Capitalism tends to suppose the latter.

> I think one real question to ask though is what is actually more efficient—donating volunteer time and energy to a nonprofit, or donating money and having that nonprofit hire professionals instead?

That's a good question. Judging from software development (which may or may not map perfectly to hardware infrastructure) both seem to work pretty well. Depending on who you ask you'll get different answers about whether closed (purely paid) or open (either volunteer or mixed) is "winning".

But certainly they seem to both work, on the whole. Neither strategy is so radically inefficient that it's safe to dismiss a company that relies on one over the other.

The incumbents are really good at suppressing competition.

One response/workaround: Guerrilla networking.

Why, in part? Because there's nothing much to target, financially. In some ways, legally. Instead, the incumbents have to elicit suppression through regulators and law enforcement, if they can.

Google has deep pockets. But it could spend years trying to get access to poles. Incumbents don't just pay to oppose; they pay to slow things down as much as possible. Good luck deploying, when you have to wait 6 months for a court case, that's then going to get appealed.

Stay under the radar. That's a whole different approach.

And communities are getting desperate enough to try anything that might actually work.

While I cheer them on, I simultaneously hope the press coverage doesn't bring the corporate mercenaries down on their heads.

Reminds me of the Corey Doctorow/Charles Stross short story Unwirer: https://craphound.com/unwirer/

FTE can work a few extra hours a week at a non profit more easily than they can make more money at their job and donate that. They may also enjoy their time and learn valuable skills which make the time vs money argument misleading, because for most tech workers they are very different pools.

Volunteers donate their time because they enjoy it. We have so many volunteers it is hard to keep track of them all. Obviously it is more efficient to use volunteers if there are enough. We are considering using contractors for some huge projects we are planning.

> they should treat it like a serious business and not a nonprofit.

A non-profit organization can be as "serious" as any for-profit business. The only difference is the non-profit might, for example, be required by their charter to re-invest income back into the business instead of paying it out as profit to the owners/shareholders/etc.

The level of "seriousness" of an organization should be determined by their business plan (is it profitable? And sustainable?), not their profit motive or financial footprint.

> if they're going to invest significant amounts of their personal time/engineering resources

First, it's their time and resources to spend as they wish.

Additionally, a suggestion I've heard is that pursuing anything that requires a serious investment of your time and resources (e.g. a job/internship, school, non-paid opportunities for "exposure") is worth pursuing if and only if it at lest two of these are true: [pays well/profitable, will improve future opportunities, you really enjoy it]. (If you love doing something and you're learning a lot or gaining experience that looks great on a CV, it can be worth a serious pay cut)

> build a network that actually does meet five nines through sufficient diversity in topology and routing

That is a good goal, but it isn't strictly necessary at first. Limit growth to what is possible at the time, and grow into a larger organization if it's successful.

Google fiber had over 450,000 customers and was thus several times Webpass size when they acquired them. They have continually grown over time and by any reasonable measure very successful.

> Google Fiber to cut jobs and halt expansion of US internet service

> Ambitious – and expensive – high-speed internet program will stop plans to expand to new cities as business reportedly under pressure to cut costs

https://www.theguardian.com/technology/2016/oct/26/google-fi...

Although it looks like they've started expanding again as per https://www.techrepublic.com/article/the-rise-and-fall-and-r.... So perhaps I counted them out too early. However, even with the backing of Google their success isn't guaranteed and they've had many non-technical, not-purely-financial problems along the way.

I still think it's a good example of why a "plucky upstart [with VC funding]" isn't able to waltz in and build a large, traditional ISP in the current environment.

There are zero nationwide high speed ISP's. So, it's kind of a question of how do you measure success.

Yea, google paused expansion for ~3 months but building a nationwide network is a ~100 billion dollar investment. The only option to get there quickly is a LEO satellite constellation, otherwise it's going to be a really long slog with lot's of local governments getting involved.

I hadn't heard they started expanding again, hence my not changed original stance.

> Google did not fail at starting an ISP. They decided that the ROI for doing individual residential home FTTH was not worth it, considering the cost per house passed.

~Kind of sounds like it failed.~ Apparently Google Fiber is starting to expand again. I still think it serves as a good case against why the current climate makes it near-impossible for a "plucky upstart [with vc funding]" to start a large, traditional ISP.

> Starting a small ISP based out of extreme dissatisfaction with huge, evil incumbents is something that I encourage any plucky upstart to do. ... Just don't try to do it with $20,000 and some good wishes, because something with much deeper pockets like Webpass will come along and obliterate you.

You can't be a plucky upstart while simultaneously having pockets deep enough to fend against the current major ISP players.

I'm not going to sit here and reply line-by-line to your post. The issue isn't that people want a free network. The issue is that people are fed up with the large ISPs and that the large ISPs have made it all-but-impossible to start an ISP.

You seem to think that VC-sized money is the only thing people need to enter the market. You're entirely wrong. Entrenched players have made it next-to-impossible to compete in terms of capital and regulatory hurdles.

Google is just a terrible example. On net, Google got incredibly preferential regulatory treatment. https://arstechnica.com/tech-policy/2012/09/how-kansas-city-...

They just realized it makes no sense for them to be in a low ROI industry: https://www.mckinsey.com/~/media/McKinsey/Business%20Functio...

> The issue is that people are fed up with the large ISPs and that the large ISPs have made it all-but-impossible to start an ISP.

that's the thing, they haven't. Not in the service territory this new nonprofit is operating in. There's something like 14 different ISPs which overlap service area with them in their part of New York City, none of which are big nationwide Comcast or Centurylink sized entities.

What they're trying to do on a shoestring is on super hard mode, they've picked the single most competitive market in the entire USA to try to compete with independent, non-LEC, non-major-telco companies that do have a few million dollars to throw at their own facilities-based network infrastructure.

NYC, and potentially a few other urban areas, have the capability to run fiber easily.

I'm also just not sure why you're acting like this. You're being simply dismissive of the project. There are many successful meshnets.

You've yet to give a specific complaint about this project and have simply ranted about how they're just small, underfunded, and not a real ISP without ever actually substantiating any of that or specifically what they're doing wrong.

NYC is actually the textbook example of most expensive places to run new fiber, due to needs for nearly 100% underground, and extensive pre existing utilities. I'm going to guess you've never been involved with an outside plant fiber project.

With your knowledge of the subject, how would you suggest they overcome this challenge?

Do they need to run their own fiber - is there a way they could rent fiber space from an existing line? At least for a little while?

Don't be worried that a new player has entered the space; be worried that the existing ones haven't improved the space. The total market space can be increased by creating new connections, and someone has to do it.

There are about 10 fiber providers in the city. We've spoken to most of them. It's pretty much impossible for us to lay fiber, so we have to lease it. We'd like to eventually lease a fiber ring across the city so we can cut costs when connecting multiple buildings. If you don't have a ring then you have to pay for the run all the way back to a coloc.

That's a good start to having "your own" network, and then you can choose to have packets traverse your network or the peering networks. You could shift egress points around, based on cost and other factors as well. Each link you add can multiply your routing options.

Get large enough, and you can start carrying other traffic too, and eventually start buying/laying fiber. But you know this, it's probably part of your business plan.

African Internet is really slow. Being a noob and knowing we can access the undersea cable. How would you go about building such a mesh network for an medium densely populate african city i.e 3+ million people ?

Do you have any experience with either ISP level network engineering or regulation? Because you speak awfully authoritatively.

I think maybe the whole point is that not everybody wants all that shit, or they want to experiment with doing without some of it. That last 5% of reliability is on the order of 50% of the cost, or more, isn't it? Is it worth paying double for the small 5% improvement? Not for everyone, obviously.

And that's true even assuming just bare costs without markups. Obviously when you take into account how much profit & graft is attached to every single level of that stack, it really starts to add up. Throw in the absolutely, inexcusably bad, awful experience nearly everyone has with the big telecom companies and their employees, and that's enough ill will to propel many an alternative DIY network.

Metaphorically speaking, you just told a bicycle owner what a terrible car he has. Or rather, warned him how much the car is going to cost that he doesn't want in the first place. Is it possible the bicyclist can actually tell the difference between a bike and a car? Maybe he even sees some advantage you don't? (As opposed to the implausible but apparently favored hypothesis by car-owners that no, he just likes punishing himself while unsuccessfully pretending the bike is a car. When we don't know a stranger's rationale we glibly tend to assume there is none, and they're just cray-cray.)

To make yet another metaphor, when all you know is hammers, the whole world looks like it's full of 1) nails, 2) hammers, and 3) things that suck at being hammers.

A phone network that objectively sucked compared to AT&T and Verizon but was boatloads cheaper seems to have worked out just fine for T-Mobile. I can see the same thing working for this. $20/mo for an internet connection that works most of the time would have plenty of customers. I'm not familiar with the wireless tech involved so no clue how well this scales outside of high density cities but within some place like NYC there's plenty of people that would pay for this over $50+/mo (min!) to the local cable companies.

I have to wonder at the level of condescension in your comments. I'm wondering if it's because you actually feel threatened by this.

>"Actual ISP network engineer here: I'm sure these people are having fun and all, but it's insignificant in the larger scheme of things"

No it is actually very significant. In a past life I was also an "Actual ISP network engineer" at a large ISP. And what I can say is that large ISPs aren't exactly wellsprings of innovation. However these folks are hacking on things, solving an actual problem and seem to be having fun in the process. It's exactly this ethos that usually ends up moving the needle. Sometimes by accident. It's curious that you are so keen to poo-pooh them.

>"Real five nines network infrastructure takes significant capital investment"

There is no ISP that provides customers 5 9's of uptime. Not even for business customers. With a Time Warner or Comcast you are lucky if you get 3 9's from their network. This is what you get from "the duopoly."

>"And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each (do you really want to deploy something in the year 2018 to take several full BGP tables that doesn't have a 4 million FIB capacity?), etc."

There a clue in the article about which IXP they're - 32 Ave of the Americas(DE-CIX.) It's one of the 3 of the most "wired" buildings in NYC. You can peer with everyone there. These people seem to have done their homework. And you don't need to buy $15K Cisco ASR any more. You can take full tables on commodity hardware running Linux and ExaBGP or BIRD. There are many CDNs and IXP's doing just that.

The whole 5 nines aspect is interesting. I wonder why there isn't more cheaper more unreliable service available for customers built on cheaper hardware using less reliable infrastructure. Laser Links to cross distances etc.

I imagine consumers would be fine with 95% uptime of a gigabit connection compared to 99,99999% uptime of 16mbit.

>I imagine consumers would be fine with 95% uptime of a gigabit connection compared to 99,99999% uptime of 16mbit.

I can guarantee that nobody would be okay with 95% uptime. That means that your Internet access is down for over an hour every day. Given that people go berserk when their Internet is down for even a few minutes, 95% uptime will be unacceptable to consumers.

Depends which hour it's down. Plenty of people would be happy to have their internet down from 3-4 am every morning if it meant they paid $10/month instead of $70/month.

FWIW, many of the people selling five nines would just claim those 3-4am windows as "scheduled maintenance" and go down for an hour every night anyway whether they need to or not...

(Actually, if I were an evil ISP lawyer/marketer, I'd try to claim a "Maintenance window of 1hr downtime sometime between 4am and 3am", then ignore any downtime of less than an hour during the day and if you complained about not getting yur expensive five nines I'd laugh and say "published schedule maintenance not included!"...)

First guess is it would be most likely to be down when traffic is highest.

You could switch to mobile data on downtime? And make that switch automatic/unobtrusive?

That means that you either have to provide an LTE connection for every customer, which would entirely defeat the purpose of your main connection, or your customers would need to have their own LTE connection, which would entirely defeat the purpose of your service in general.

God, this reply couldn't be worse:

"Real five nines network infrastructure takes significant capital investment."

No, it doesn't. Overly complex, corporatized infrastructures designed by committees in meetings - that takes significant capital investment. Simple infrastructure designed by hackers can achieve bullshit metrics like "five nines" without breaking the bank.

"And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each"

Establishing colo at a "major IX point" is a few emails and phonecalls away. Your salesperson will fall all over him or herself and waive the NRCs to get you in there. It doesn't cost much, even for a full rack with 20 or 30 amps. If you're just doing networking, you won't even need that. Then you get a fiber cross connect to he.net for 10 or 20 or 40 gbps and you're in business.

You can buy very, very nice used routers and since you're a hacker and not Mr. Enterprisey IT guy, you can run it and support it yourself.

We had many iterations of this exact critique in our early days - explaining how we couldn't possibly deploy this volume of storage without fancy bullshit enterprise solutions from EMC or Compaq and how we couldn't possibly run a global storage network without fancy routers (we currently own zero routers) and fantastically complex clustering and failover ... and 17 years later we're still making it all work just fine.

Fuck those people. Fuck those school trained, passionless, non-hacker, no-talent-pussies.

Good luck with your mesh network!

This comment sounds very much like the AT&T engineers mentioned in http://www.shirky.com/weblog/2010/04/the-collapse-of-complex... (a post which appears on HN quite regularly).

---

> In the mid-90s, I got a call from some friends at ATT, asking me to help them research the nascent web-hosting business. They thought ATT’s famous “five 9’s” reliability (services that work 99.999% of the time) would be valuable, but they couldn’t figure out how $20 a month, then the going rate, could cover the costs for good web hosting, much less leave a profit.

...

> The ATT guys had correctly understood that the income from $20-a-month customers wouldn’t pay for good web hosting. What they hadn’t understood, were in fact professionally incapable of understanding, was that the industry solution, circa 1996, was to offer hosting that wasn’t very good.

...

> For a century, ATT’s culture had prized—insisted on—quality of service; they ran their own power grid to keep the dial-tone humming during blackouts. ATT, like most organizations, could not be good at the thing it was good at and good at the opposite thing at the same time. The web hosting business, because it followed the “Simplicity first, quality later” model, didn’t just present a new market, it required new cultural imperatives.

Agree the article is BS. Mesh networking is incompatible with QoS. So they can surely deliver crappy Internet cheaply. More than that? Nope. That'd need serious infrastructure such as licensed P2P, 60GHz short haul links, leased fiber. $$$$$

Oh, and they'd need peering: more $$$$$

Did you read any of our comments?

You have any opinion on if internet should be treated as a utility?

I wish I could upvote this many times. Great reply. Agree 100% and thanks for speaking for us that work behind the scenes to make things happen.

you are upvoting [things that are not true about us]

> you are upvoting nonsense

I don't think this is a helpful reply.

Since it sounds like you're associated with the project, can you address some of the specific concerns mentioned, like reliability? Do you disagree that hundreds of thousands of dollars are required for five nines connectivity? Do you disagree that that level of reliability is required for your customers etc.

No you don't need hundreds of thousands. You do need tens of thousands. We've had no downtime this year at supernode1. Yes reliability is a big priority, but it is not a big priority for the big ISPs in NYC. Time Warner Cable (now Spectrum) has barely two nines. In Manhattan it would go down in the East Village after every large storm. We are already providing faster more reliable connections than this.

Thanks. This seems like a cool project.

Do you know what the cost per user is currently?

What frequency do the point-to-point links run at? Have you considered using some sort of lower frequency packet radio for longer distance links, perhaps to other nearby nodes?

Our recurring costs are only for the lease at supernode1, which is around $1000/month. Divide that by our member node count (158) multiplied by users of each node ~4, 1000/(158*4)gives you under $2/month per user. Members usually pay for their own routers so that isn't included. To set up the supernode was about a $10K one-off charge for install fees, servers and antennas.

DE-CIX, our IXP, donated bandwidth to us, and we have transit also donated from Packet Host and WebAir. We actually pay nothing for bandwidth. Probably one day we will pay but it is not that expensive if you do this at an IXP and use peering.

We have an AirFiber pair that is 24Ghz, but all of our sectors are 5Ghz wifi.

> DE-CIX, our IXP, donated bandwidth to us, and we have transit also donated from Packet Host and WebAir.

What is the incentive for them to donate bandwidth/transit to you? Your non-profit status? Do they get something else in return?

We get a lot of support from the NYC networking community, and we have a lot of friends at the NYNOG meetups. Everyone is very helpful and they also want the big ISPs to have some competition, so we get donations from quite a few people. Some also join our network and get nodes on their own roofs and help with other installs!

DE-CIX is the biggest IXP and they measure their network in terabits/sec. It is not a big deal for them to donate us a 1 gig connection. (it's a big deal for us!) Eventually we may upgrade it to 10 gig.

So you're sharing a 1G connection between 600-ish users - all presumably on 802.11ac 450Mbps (or 1.3Gbps if they've got three-stream?) connections?

How often do you see full saturation on that link? I'm guessing if all 600 of your users all tried to stream one stream of $TV-show-de-jour at once they'd barely get 16Mbps each? Can one user's home office full of Apple gear flood the bandwidth downloading a dozen Apple software updates simultaneously?

That is a pretty low ratio in terms of over-subscription, on many cable networks you'd have 24 downstream Docsis channels @ 38Mbps usable per channel, for 912Mbps usable across all houses on said node. A single node often supports 500 homes, and that 912Mbps of capacity carries switched digital video, voice & data traffic.

Cable companies consider these high ratios to be quite embarrassing actually, as it shows how crummy their networks are: https://www.dslreports.com/forum/r31258251-Speed-Issues-Char...

Netflix standard streaming is 3Mbps, so even with your math (which isn't how bandwidth works) everyone is happy.

We've never come anywhere near saturating our gig connection. We monitor it. Each building is limited to their own rooftop connection which is an average 100Mbps, so they can't do more than that if they tried. The thing is not everyone is downloading a file at the same time, and streaming uses much less bandwidth than downloading. From Netflix site-

0.5 Megabits per second - Required broadband connection speed

1.5 Megabits per second - Recommended broadband connection speed

3.0 Megabits per second - Recommended for SD quality

5.0 Megabits per second - Recommended for HD quality

25 Megabits per second - Recommended for Ultra HD quality

Ah, so you get free bandwidth from people that want to balance their traffic for better peering contracts. That's clever. Probably not a helpful model for other areas that aren't bandwidth centers like NYC, and should probably be more out there when you're evangelizing so people understand you're getting a large portion of your operating expenses donated...

I have to assume another cost that's not noted is roof rights. That's not free - most people either have to pay or give the property management a kickback.

NYC is an interesting place for this - if you live in Manhattan, you clearly can afford the $80 for 1Gb/s FiOS or $40 for 100Mb/s FiOS or whatever Spectrum is charging. It would be way more interesting to plop this in a rust-belt city where people are on a paycheck-to-paycheck salary...

FWIW, I have many many companies as customers for whom I've offered pricing for three, four, and five nines of reliability (for mobile app backends, not residential ISP service, but lets run with it any way).

Not a single one of them has ever signed up for anything more expensive than three nines. Only a few have even discussed the differences between 3 and 4 nines solutions.

I'm reasonably sure I'd choose the same for my home internet connection - if offered representative pricing based on costs of providing 3, 4, or 5 nines, or possibly even 2 nines - I'd choose the least expensive because losing a few minutes or even occasionally 15mins a day of connectivity at home (or maybe more realistically an ~8 hour outage per month) really isn't going to bug me greatly - not if it's two or more orders of magnitude cheaper than a four of five nines connection.

Well, it seems this idea has touched a few nerves. In this case, it may indicate you're on the right track. Keep it up!

Förderverein Freie Netzwerke e. V. has been doing this, rather successfully, in Germany since 2003 [0]

Similar projects exist in Austria (FunkFeuer) and Switzerland (Openwireless).

[0] https://en.wikipedia.org/wiki/Freifunk

There are several distributions for mesh-networks:

- Gluon is the most popular: https://gluon.readthedocs.io - Libremesh is more flexible: https://libremesh.org/

Like NYC Mesh both are based on OpenWRT.org a wonderful distribtion for wireless devices from 4mb flash / 32mb memory up to 64bit x86 support.

'Anti-authority', even with scare quotes around it.

God forbids somebody uses the word "anarchist"...

Synonyms are fine. Those with knowledge know what's going on. Those with prejudice can remain in the dark for a time.

Is there an inexpensive point to point technology that isn't hampered by rain?

2.4 and 5 GHz aren't too affected by rain fade. And they're cheap.

I thought wifi couldn't go through water? (Or is rain just so sparse/diluted that it doesn't really affect it?)

I'm not sure about water, but rain doesn't have much effect at those frequencies. At higher frequencies (e.g. 24GHz) rain can make the link useless.

Yes. Whatever TowerStream uses. But it is definitely line of sight and you have to be on Empire State or Trump Tower or a few other buildings to have any reasonable visibility

[Ron Paul voice] It's happeniiiiinnnnng

NYC public housing has had rock bottom cost, high speed Internet for years.

Have to now do something with it.

I'm in NYC. I know a few people who played with it. It barely works.

I'm all for this kind of tech solutions, but for crying out loud pretending that this is a replacement for ISPs in NYC really discredits techies.

Why all the negativity? Isn't this tech in the early stages? Isn't there hope that these issues can be worked out? and in what way was it "barely working"?

I had at&t in one of SF's typically old buildings and I can tell you, the internet there barely worked - i had to reset the router about once a week, plus hour long downtimes not uncommon. Even now, when I'm in a brand new 3 year new townhouse, I still have a reset the router once or twice a year and I get half the speed I'm paying for on a DSL broadband plan, not to mention that I get stuck in a 12 month contract anytime i want to switch plans.

Considering that there are currently no other options than the att/comcast douopoly, I would think people would give a little more credit for providing alternatives, even if it's not perfect yet.

No, this tech is not in early stages. This tech has been around for years. We have been running this stuff between building since mid nineties. The cards were called "wireless T1" and they were full size ISA cards. It was pre-DSL days. I had a T1 terminating on 4th floor of a brownstone and a couple of my friend who lived across the street got connectivity from me using those cards because Linux supported them. No one else on a street had anything other than a dial up. Eventually we ran Ethernet to the other neighbors because we did not need to run across the street and those neighbors agreed to share a cost of the circuit.

The reason why I'm negative about it is that it gives people incorrect perception. The perception is that my neighbor does not need to pay Optimum or Time Warner or Verizon for internet and he or she can watch Netflix. And that's simply not the case because as soon as five of my neighbors get this WiFi chokes because in a regular Brooklyn block it is going to add another 40-50 wifi access points. Negative perception from folks who own their houses is a death to community ISPs.

This kind of wireless is not a solution. A solution is to wire as much as possible, including wiring between the buildings of my neighbors who own their houses and if needed terminating a single high bandwidth wireless or fiber link into a node that carries entire block.

P.S. I reached out to these guys about a year ago offering to help them when I read another piece written about them. Maybe it even was on HN. I did not get any response.

Our average speeds connecting to our supernodes is around 100Mbps. I'm not sure what you are referring to.

What's the throughput?

What's the latency?

What is the median speed?

How does it work for torrenting?

Does VoIP work?

What if I want to run a server at home?

I manage 3 nodes in brooklyn and the speeds are ~150mbps. It's so reliable that no spot has a backup connection so it is acting just as that - an ISP. I don't see how the ambition is a discredit to techies (lol).

So it works, then? Good news. If you're saying that what they need is more internet bandwidth and better coverage, I'm pretty sure that's what they said themselves in the article.

Why doesn't a startup run their own fibre? Are building owners that protective?

Actually scratch that, I tried getting a 1 gig cross manhattan link. Cost more than 1g to London.

Building owners? Before you get anywhere near the buildings, you have to dig up the streets to run cabling around the city. Any idea how much that costs?

Speaking of London, talking to building owners is exactly what Hyperoptic did to build out a fibre network in London.

Unfortunately, the building owner where I used to live wasn't interested -- it turned out most of the building was empty, people keeping apartments for occasional visits or as an investment :-(

https://www.hyperoptic.com/how-hyperoptic-works/ or https://www.hyperoptic.com/map/

Pilot is doing this cheaper than others, look into how they get their fiber in the ground...

That's what you outsource to the latest incarnation of Metromedia Fiber Network - Zayo.

Don't you have ducts? How about sewers?

The city's network of utilities long predates any sort of central authority managing them. Each utility maintains its own network of infrastructure, independent of any other; most of them are not mapped, and where mapping has been done it's not shared with the municipalities or other utilities, let alone publicly available. There are no ducts set up for general usage.

And the sewers a) aren't human-travelable like the old ones in Paris or parts of London; and b) overflow in heavy rains. You probably don't want to run electrical wiring through them.

It depends on the location. If you are talking about downtown Manhattan you can get dark fiber between any major buildings for about $5k/mo. If you are not in a mojor building then Zayo would do it for about $20k build out worst case scenario + same $5k/mo, since Zayo touches a boatload of manholes. There was also Metro Optical but I dont know how large is their footprint. They are quite creative.

But in general, go to a decent building and order Cogent. That gets your building on the net. Cogent eats fiber extension charges left and right. Plus you actually get 1Gbit/sec for $500/mo. Now you take that 1Gbit/sec and wire to neighboring buildings by running ethernet or standard yellow fiber. That's how ghetto ISPs get stuff done. It was done for years because it works. There's incentive for those who own building to form a community ISP or provide an access to a community ISP because those that own builds want the same product and they know they can share some of the costs.

Yes, that sounds about right -- we pay about $4k for 1gig midtown-midtown

However GTT offered me 1gig from New York office to London Docklands for $2400! Under $2k for p2p to office in Washington DC.

If local internet is $500pcm , get two lines into the building for $1k/month, charge 50 people $30 a month and give them 1 gig and 25:1 contention ratio, plus the cost of dropping the fibre and a router, leaving $6k a year. It's not going to give you a full time job, but will give you some decent internet and a bit of pocket money to look after it.

There are actually a lot more interesting things that you can do:

Order an SDI video circuit from VZ to one of the interconnect buildings. That's $1.5K/mo but it is delivered as a point to point single mode fiber with VZ gear being plugged in. After they install it and leave unplug VZ gear. Plug in your 10G. You just got yourself a 10G transport for $1.5K/mo.

Remember, because it is SDI there's no way for VZ gear to know that it is actually unplugged.

$4K sounds incredibly overpriced.

Hate to sound stupid but how does DNS work on a mesh?

We originally used the qMp/OpenWRT package that included mdns, which is a version of zeroconf for the internal network. It isn't scaling so we now use our own DNS server to resolve internal addresses on our 10/ network.

Mesh networks are going to be incredible when they're widespread!

It's truly superior to centralized control by ISPs/Governments and the world will be grateful for this after this whole net neutrality debacle recently.

You guys are brilliant and doing an amazing job and I'm grateful people like you guys exist!

Thank you for the response!

Who's responsible when someone torrents copyrighted material through that Internet exchange point?

"NYC Mesh will comply with all federal laws in the countries it operates, however, as policy, no data is collected and therefore no data exists to provide requestors."

This is cute, but if you aren't hunting down people abusing the mesh, the whole connection to the internet will go away for a ToS violation.

I've worked for an ISP for about six and a half years. In that time, I have received I-don't-know-how-many e-mails about people torrenting and so on. I've had one conversation with a Special Agent from DHS investigating credit card fraud.

Over this same period, I have given out a customer's identifying information zero times (if more than 24 hours have passed, there is a very small chance that I am able to identify the customer). In every instance, it was "sorry, I am not able to identify the customer".

Zero is also the number of times I have heard anything from any of my upstreams regarding any ToS violations or similar.

> if more than 24 hours have passed, there is a very > small chance that I am able to identify the customer

Hmm ??? Could u please explain.

I always assumed that all ISPs will store all DHCP (PPPoE? PPTP? IANANE) allocations by IP & time.

I'm sure that most do. Those records are kept for systems I manage as well, but not for any longer than I need them.

CG-NAT, NAT, etc?

I imagine it's not as bad as running a Tor exit node, which organizations like Noisebridge [1] seem to do without much issue (aside from the occasional visit from the FBI... [2])

1. https://www.noisebridge.net/wiki/Noisebridge_Tor

2. https://www.noisebridge.net/wiki/Noisebridge_Tor/FBI

What is the agent's thought process on a visit like that? Do they not know what Tor is?

Didn't one of the major ISPs get their ass handed to them in court for trying to say that they were somehow exempt from keeping exactly this sort of information? I mean, you can SAY it, but it would be sort of insane to make a statement like that without a lawyer signing off on it. The law doesn't have a "this is too hard for us" or "we didn't build our network for this" exemption, does it?

The user, just some one user torrenting material using Comcast doesn't endanger Comcast themselves.

That's just not how IXes and peering agreements work.

The beauty of it is as it becomes popular people will start torrenting WITHIN the mesh network.

Brings back memories of using DC++ hubs on the university network

I imagine tor exit points have the exact same issues?

Mesh networks are too complicated to be reliable over a long term. They are useful in disaster situations or places with no infrastructure. Project Byzantium (http://project-byzantium.org/faqs/) is a great way to quickly stand up a mesh network as a temporary thing.

> Mesh networks are too complicated to be reliable over a long term

I mean, this isn't true. I'm not even sure to begin. Perhaps specific points instead of nebulous nonsense would help us better understand you.

Sure. The protocols, the routing, the interference, backend and gear maintenance, regulatory requirements, expense and complexity of multiple WAN links, and ad hoc setup of said links in an uneven physical environment. To say nothing of managing uneven customer load.

Running fiber is expensive but less complicated and less error prone. And unless I missed something, there aren't any initiatives by big companies (that obviously don't want extra expense and headache) to go mesh. Big ISPs use their customer nodes as wifi hotspots but that is obviously not mesh.

> Sure. The protocols, the routing, the interference, backend and gear maintenance, regulatory requirements, expense and complexity of multiple WAN links, and ad hoc setup of said links in an uneven physical environment. To say nothing of managing uneven customer load.

I think you have an overly narrow view of what meshnets can be.

What are you talking about? Most wireless ISPs running OSPF could be classified as “meshnets”, plus the whole internet backbone is routed by BGP which works in the same way.

Reply on news.ycombinator.com