Starting an ISP: Deploying Fiber | Hacker News Reader

Starting an ISP: Deploying Fiber

chrishacken.com

276 pointschrishacken63 comments

This is the real spirit of entrepreneurship. Hard work, learn quick, make mistakes, but get it done. With skin in the game rather than burning through investor money.

I hope to read more of these posts, but I know Chris cannot afford to spend a lot of time writing them. Nevertheless I'm looking forward to reading part 3 when it is posted in 18 months.

PS slight grammar nitpick: overtime != over time

Fixed the grammar :p

Also fix "I'm clearly a little bias against wireless"

Edit: and "genre's"

I enjoyed reading the first part of this last year, and was thinking of it recently.

As someone who built a bootstrapped company from $40k/y revenue through $3.5M/year over 14 years, before we were shot in the head by our bank, I was thrilled to hear that you were doing something like this, but concerned over the many potholes (pun, I guess, intended) you were going to encounter. One of the biggest ones that was an eye opener for me early in starting and running Scalable was the rent seeking behavior of your suppliers. You wrote this as "Everyone wants a piece of the cake."

Yeah. That. It doesn't get better.

My comments for now are

1) beware of banks and debt instruments. If you can avoid personal guarantees on loans, then sure. Chances are they will say "no guarantee, no money". That money is extremely expensive to you. Avoid it like the plague.

2) beware "partners" and people/companies that claim to want to help you. Do due diligence, deeply, on any organization, including lawyers, accountants, etc. you might consider using. Most will waste your time.

3) hiring ... is ... critical. This is hard as a small co, as you generally can't find great people unless you have a pile of cash. You can try to turn who you can get into great people. But this only works if they are willing and able to learn.

4) Venture capital. Don't get me started.

I'm rooting for you, and like you, I had every ounce of skin in the game. My retirement, my daughters college. Everything.

Understand that things don't always work out the way you intend. Starting over when you are my age is very hard. Consider me as a cautionary tale.

I had a number of folks offer to buy the company. I was too stupid to realize that one should hit a number of singles and doubles to be able to get enough of a base to hit homers and grand slams.

Best of luck, and I've subscribed to the site now.

>I'm rooting for you, and like you, I had every ounce of skin in the game. My retirement, my daughters college. Everything.

>Understand that things don't always work out the way you intend. Starting over when you are my age is very hard. Consider me as a cautionary tale.

I'm pulling for him too, but I have the awful feeling that I'm reading the story of a guy blowing his savings, ruining his credit, and maybe taking mom down with him.

I started out in the ISP industry, back in the mid-90's. This was back in the dialup / T1 days. I can't imagine doing it now...

We're a profitable company doing over a quarter million a year ATM. We aren't running fiber on a whim.

We were too. Until we needed to expand to meet a customers needs. Then we had to take on debt.

As I said, a cautionary tale. Call me the ghost of startups past :/

Do you have a write-up available about the circumstances that led to your bank issues? I'm very interested in that read.

Great write-up. Great entrepreneurial tenacity. Would love more insight on the wireless challenges and how you tackle them? I wonder if a software monitoring app could detect increased noise and automate channel hopping.

Surely that must exist -- on the other hand, you mentioned the vendors do everything to lower the BOM, so perhaps they are lazy and this software is needed. Could be a good OSS project.

So Mimosa, one of the hardware vendors we use, does actually have this built-in to their firmware. The issue is, from what I've seen, is that it hops around too much. It also moves to congested channels sometimes. I'm sure there's a way to perfect it, we just don't have the time at the moment (I also don't believe they provide full API access yet, so I'm not even sure if we can change it without creating an automated UI tool that clicks around on a browser.)

How has your Mimosa gear worked out? I tested some B5s and they choked when you got them near even a tiny bit of interference. The B11s we have won't even hit MCS9 on a 1.5 mile link. I hear people swear by them, but in my testing, Airfibers & Rocket M5/AC spanked them every time, particularly when there was even a little bit of interference.

And FWIW I've never had auto anything actually work and pick the right channel. If you're trying to GPS sync, forget auto. But you know that already.

A founder of Mimosa (CTO Jaime) is a friend of mine. I'll try to direct him to this thread.

I really wanted them to work.

I've tried to get their attention before and it winds up being a constant loop with Support. The B11s work good enough that it's not worth my time to chase the last couple hundred Mb/s. I just gave up on the B5.

10 We need you to connect these to the cloud so we can monitor them.

20 Okay we see your monitoring data, we will circle back in a month

30 Hey try this latest firmware update.

40 Are you sure the antennas are aimed right?

50 Try replacing the cables

60 Oh, your chains are unbalanced by 3db, that's the problem.

70 GOTO 20

By about the 3rd iteration we replaced the B5s with Airfiber using the same antenna and haven't touched it since. I've just accepted that the B11 link won't hit its advertised speed and quit spending time/money on it since we are putting in fiber soon.

I pulled the B5s back off the shelf a few months ago just to see if maybe newer firmware would fix, and once again a tiny bit of interference dropped a link that was doing 1.2Gb/s to 7mb/s or less.

I'm testing some B24s in a real-world link in a couple of weeks. I hope they do what they're supposed to.

Feel free to contact me via email -- my username at gmail

Jaime from Mimosa here...happy to speak to the software radio related stuff, thanks Chris for mentioning. So we've done a huge amount of experimentation on auto-channel approaches now that we've had a few years to collect data and analyze. Simple answer is it's really different in different environments, and if you're collocating with yourself vs. with other WISPs and gear.

We definitely allow the B5 series to be more aggressive on auto-changes since it's dual-chain, and is in a noisy band, so moving 1 impacted chain while we keep the other up, if it means better throughput, is a nice option IF that's what you want.

To your comment Chris, we definitely will begin to open channel programming via software API for y'all, and more rich monitoring in the API/SNMP info relatively soon (hoping before year end).

When it comes to 5 GHz, I always suggest "exclude before you auto" - we were unique in offering single page spectrum setup and live spectrum analysis to make it easy for you to program where you want the radios to operate. So smarter exclusion based auto can be really nice if you know how to use it well. As you know, we built this all in the radio, but hadn't opened it up externally much other than the UI/browser.

We actually also built in some L2 and beacon discovery smarts to locally detect other radios on the same sync settings to make sure to coordinate so to speak. Frankly speaking, even if the radio wants to make a change, it has no way of knowing if it might impact something else, it just knows what's best for it's own operation without some higher level spectrum management coordination, so some coordination intelligence is a must if you have some spectrum that needs protecting when collocated.

Really the need for auto behaviors is relatively limited to backhaul and unlicensed - most don't want their sector/APs to move unless it's a major event given all the clients connected. On new products like the B24, we really just analyze narrowing channels if things get bad enough, but there's really no other channels to move to, and on licensed products, you have to stick to your licensed channel of course.

On multipoint, for example, the APs collect and analyze spectrum masks from each client, and the primary thing they'll do is allow an individual client to narrow (e.g. from the 80 MHz channel to 20 MHz) in case they see more noise in a part of the channel, poor mans OFDMA, but that functionality becomes very advanced in the 11ax based chip for us in the future.

Lots of new stuff announcing and shipping this week and in Vegas next month, so stay tuned.

From a WISP perspective you rarely want sector or horn antenna APs hopping to randomly chosen frequencies. Particularly if you have a well designed AB-AB back to back synced reuse or ABC frequency assignment pattern by azimuth. Any of this gear has a fairly full featured SNMP MIB and OID documentation so there are a lot of different backend operational support syetems you can use to monitor it.

As a one-time dabbler in the local ISP business I found this fascinating but I also wanted to scream "Nooooo...stop" as I read! There are contractors who spend every day digging up streets, laying conduit. FFS use one of them to do the job rather than messing around looking for a Bobcat with the right Diamond saw attachment. Another thing I'd worry about is that there may be fiber in the ground or on poles already in this area -- currently not being used because there isn't a big enough market for service. In our town I am asked from time to time if the city should pay to "drop a fiber conduit in a trench they already dug for some other purpose, because this town has no fast Internet". I take them outside, point up at the fiber bundles running all around town on poles...

I’m already well aware there’s fiber on poles here. The difference is, their network isn’t built to support FTTH, it’s built to support a few larger enterprises that are willing to pay $3,000/m for a circuit. Name one person that doesn’t want fiber internet. Saying there isn’t a big enough market for service when literally every single household and business purchases internet is ridiculous.

We already have 60% of the street we just cut up signed up for service.

>Saying there isn’t a big enough market for service when literally every single household and business purchases internet is ridiculous.

I certainly wish you good luck but I suspect that you are missing something if you think you're going to compete with national providers who have access to billions in capex and infinite marketing budget, selling a product where almost no customer understands the fine details of QoS and customer service (which are the only ways in which you can differentiate your offering).

fwiw we have dedicated fiber loop (not ftth) providers here (middle of nowhere, USA) who will deliver service for $600/mo with low or zero install charge so I question your $3k/mo figure.

Absolutely true. I have property about 30 miles north of WB and Frontier told me they can't get a technician out to replace my service until January.

It would be faster for me to crawl on the ground to the nearest PoP and plugin to a GPON port then it would be for me to wait for them.

Yes (I have my own version of this story that involves drilling holes in a mountain top for a tower, pouring concrete, deploying my own solar power system, microwave links, ...) however note that Frontier are making money by not providing you service. There is no mythical better ISP who will spool fiber 30 miles out to you, and will then sit back making money. They will go broke.

I was recently at BornHack2018, a small security camp for the 3rd summer this year held on the island of Bornholm.

The organisers also run BornFiber who are connecting the entire island with optical broadband.

BornFiber is a good example of how anyone with the right skill can start a fiber provider here in the nordics.

I'm glad to see this philosophy being adopted by the US.

However. Once they have fiber they still need some ISP and that's where the national ISPs come in. Thankfully we have plenty of options here but I hear things aren't as good in the US.

If only they were on the other side of Denmark. I wanted to bring some fiber to a small German island called Insel Fohr.

So what's stopping you? :)

Just get some neighbors together and go for it.

When deploying fiber, always lay down 10x what you need. Or calculate one fiber duct for each property on the way. So when more subscribers want to get connected you just have to dig at one point, cut one of the unused ducts and join it.

As someone whose only entrepreneurial idea is to open an ISP, this is a great read. I'd like to know how the councilman and the director of operations helped out. Did they expedite permits, waive fees, or something else?

It's really encouraging to see an effort like this happening in the Scranton/Wilkes-Barre area.

Having lived in Northeastern PA at various points in my life, I can attest that Internet speeds there have always kind of lagged behind. I knew people in more rural parts of the area who had basically zero wired internet options at all. It kind of just becomes something you learn to deal with.

Are there any grants or any kind of city money for work like what you're doing? I would think towns like Wilkes-Barre and Scranton would be bending over backwards to try and attract the sorts of businesses that would rely on infrastructure like what you're putting down. That you have to do all this with your own money and credit seems really wrong.

You would think that. Unfortunately many/most people care more about their precious roads than they do about infrastructure.

are roads not infrastructure? that transports food, and medicine?

In the article, he's referring to people complaining about a 2-minute increase in their commute time for something that can absolutely benefit the community.

The short-sightedness is astounding.

Inspiring story. Will be interesting to see what Chris is capable of if / when he's able to get access to more capital.

Are there VCs who invest in this sort of enterprise? The odds of getting a positive return seem favorable compared to many of the currently trendy software-only startups out there.

Also, I like Chris' tagline:

    Create value, not excuses.

No, the returns aren't there to make infrastructure investments attractive to VCs.

A software startup can exit at 10x, an infrastructure investment won't.

I think the key difference is that infrastructure is more of a long term investment. We can return 10x’s but it’ll likely take a little longer than a software company would. However, i can make the case that we have a higher guarantee of delivering than a software company can.

"A little longer" == 25 years if your returns are 10% and grow 10% annually.

Other than than all true. Telecoms infrastructure is an utility play, steady long term returns. Something pension funds like, but they on the other hand don't like risky startups, so it's a challenge to get funded until you are established and have a solid track record.

200 subscribers with 90% coverage of a town-sized city (40k population)? sounds absolutely horrible.

but I really came here to comment on what I think is a financial error: rent vs buy cost. yeah, it seems more expensive to rent if you consider long term ownership of equipment, but this is very misguided. maintaining, repairing, insuring and just storing heavy equipment that you use rarely is just not the way to go. it's cheaper to rent.

> nothing to show for the nearly $10,000 in rental fees

you have no long term expenses to show for it. that's a wonderful tradeoff.

>We just launched in our 2nd city; Scranton, PA.

That's not... the Scranton, PA is it?

That it is.

The electric city

So what is his competitive advantage over the incumbent providers - the cable company and the phone provider?

Is he offering service in areas they aren’t? https://www.broadbandsearch.net/service/pennsylvania/scranto...

If it's anything like the smaller ISP's in my area, his advantages are:

- Vastly superior customer service

- Simpler contracts; straight fee-for-service without any bullshit

- Likely better price overall

From my real-world testing, you shouldn't really expect to get more than 30-40 Mbps down and 5 Mbps up in a typical scenario. If you're competing with DSL,

For the most part, he isn’t competing with DSL. He is competing with cable. It looks like Comcast does service the area.

Another thing that stands out was that the article mentioned he was paying $1000 for a “< 1Gpbs” Fiber line.

From the link on what’s available in Scranton, it looks like Comcast also offers 1Gbps downloads.

He also said that he has 200 customers - mostly business.

I don’t see how this makes sense from an investment standpoint to offer service in an area that doesn’t appear to be underserved.

I’m definitely no fan of Comcast, but their business internet service is straightforward. If I’m running a business, the cost difference between Comcast and Loop is neglible.

Personally, I’m glad to have the much better, cheaper, gigabit service from AT&T, but I would grin and bare Comcast again if I had too.

I don't have an answer for your question, but I believe you've misquoted the article. It was <$1000 for a 1Gbps circuit, not $1000 for a <1Gbps circuit.

I've met Chris and I told him that if I didn't live so far outside of the city I would be happy to pay him for service to my home. I have decent speeds through Comcast but their customer service is atrocious. Telcos have a monopoly in most areas and they know it and act accordingly.

Yes Comcast is the definition of poor customer service. We were forced to have them when we lived in an apartment. I haven’t had any issues with the customer service I received from AT&T and before that Bellsouth. But I also use OTT providers for TV service.

AT&T Fiber in my experience is a lot more reliable than Comcast ever was. Even thier old DSL was reliable.

My mom has had DSL for years in the small town I grew up in. She got cable internet when I stayed there for the summer because I needed more speed to work remotely. But she decided to keep both. The cable for better speeds and she said after 50 years of dealing with the phone company and 40 years dealing with cable, she trusts the phone company’s reliability more.

If a smaller ISP could offer the reliability of the incumbents, I might switch just to support them, but they would have to do a lot to gain my trust.

He's doing fiber optic. They're doing DSL and cable.

But his speeds aren’t any better than cable. And yes I am dying a little inside for actually recommending Comcast....

Have you ever tried WeFunder or similar crowdfunded mechanisms to raise money? I would invest a few thousand at generous terms to support you guys and the mission, and I’m sure many others would too.

Can't imagine it would be worth the investor liability/risk for a couple $k.

percentages are percentages.

It’s crowdfunding. He would raise hundreds of thousands or millions.

Title III crowdfunding is limited to $1M.

Thanks for sharing this! As someone in a fiberless city (Amsterdam), with a monopolistic local provider that's unwilling to expand, this is inspiring.

What happened to Citynet? They were supposed to wire the whole of Amsterdam with open access fiber. Did they stop after the Reggefiber/KPN acquisition?

The cost for micro trenching can be upwards of $200k a building in Manhattan. I would assume dealing with sunken homes, collapsed PoE's and NIMBY folks would make it prohibitively expensive.

But I have no idea why they stopped.

Manhattan?

Great work so far Chris! Keep up the great work!

how do you deal with the legal aspect of running an ISP? What can the small players do?

> [Wireless] Speeds are competitive with cable (for now).

Latency, or just bandwidth?

Both. Our latency is better than cable in most cases.

I know of a few WISPs that have real network engineers behind their systems and have built reliable networks. One of them has several thousand users and multi-gigabit services.

People are jumping every day from cable to them for both speed and reliability.

Most WISPs are started by the local computer repair shop by someone who things that configuring a Linksys router is network engineering. They are plagued with reliability issues mostly of their own creation. I'm not knocking the people, they are filling a need and doing it 'good enough'.

If you build it right, they will come.

My backup WISP link is pretty terrible, and does up to 10 Mbps, barely 5 Mbps most of the time, even thought there is clear line of sight to the base station antenna, and using Ubiquiti gear, probably because the people are either inexperienced or overselling. 4G gets me 300 Mbps (now up to 300 GB for $9/mo, or unlimited data for $25) in a village of 904 people. (though I use two different antennas and connect to two base stations to obtain maximum speed from LTE with carrier and link aggregation.

I love reading these kinds of stories.

Keep them coming Chris, if you can :)

Reply on news.ycombinator.com