ex_ubiquiti

> You assume anyone except the hacker knew anything about AWS.

There were a lot of smart and experienced people at Ubiquiti when I worked there.

Nick Sharp manipulated his way into total control over everything and wouldn't let anyone outside of his isolated team touch it. Nick was hired out of his job at Amazon because he was supposed to be the AWS expert. He used that to lock out anyone but himself and a trusted team.

It didn't matter that others knew better. Nick controlled it.

> Also, even though they may have had read access, not many knew it existed. But it wasn't super hard to find (I stumbled across it basically).

We didn't have read access until Nick Sharp and his team took over GitHub permissions and gave everyone access. Wonderful security work.

> Oh and then there the whole metrics collection debacle, where the controller basically phoned home about the topology of every network that it managed. Even if you opted out. Opting out just meant they fuzzed your ID so any given record couldn't be linked back to PII. Which may or may not be legal, IANAL.

Nick Sharp was at the core of this too! He built the 'trace' system to collect all of these metrics and had all of these ideas about how to secretly collect the data in ways that would be hard for people to detect.

He pretended to be a principled person who stood for security and privacy, but whenever he saw an opportunity for political gain he abandoned all principles. He was the only person I knew at the company who was enthusiastic about collecting all of that data.

> For reference, I spent a year (mid-2018 to mid-2019) running the UniFi Network team and worked with Nick during that time.

Nick's whole strategy was to find a problem, exaggerate it as much as he could get away with, and then offer himself as the hero who would fix it all.

He exaggerated or lied about everything he wanted to use for political advantage, right up to the end where he fabricated a hack and used Krebs to exaggerate it as much as possible for his own personal profit.

You have to realize he did the same thing during his time at Ubiquiti: Found problems he could use for political advantage, exaggerated them as much as he could get away with, and then amplified his lies until they were gospel. A lot of what you're saying has some roots in truth, but I can tell you have the exaggerated Nick Sharp version of events.

> There was Robert.... and then nobody knows. I asked repeatedly why we didn't have a CTO, or a COO, or a CFO, or CMO or ANYTHING and I got nothing but shrugs and "idunno" as a response for the whole year I was there.

This wasn't some big mystery. Everyone knew that Robert ran everything as CEO and the legal, marketing, and other teams operated out of the New York office.

> Nick came in and started putting "proper" AWS structure and security in place, primarily by scaring Robert (the CEO) into giving him the keys to the castle

Nick was hired specifically to run AWS. That was his job from the beginning. The old cloud team quit and Nick was recruited from his job at Amazon because supposedly he was an AWS expert.

The incident where he scared the CEO was the first of his political games to exaggerate or fabricate security incidents for political gain.

> So why wasn't anybody else notified? Simple. Because he was basically "god". If anybody was gonna be notified, it would've been Nick. He was the top of the totem pole company-wide when it came to AWS.

Yes, this. All of these news stories are missing the point that Nick was the cloud lead. You don't have to believe anonymous commenters. His LinkedIn profile will confirm it. He was recruited out of Amazon to lead the cloud efforts, but he was in over his head and had severe personal issues.

> at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to.

This is another Nick exaggeration. It's true that older devices had hardware signing keys stored in a Git repo before the system was updated and keys rotated. However, those old keys were only accessible by a few people until Nick and his team took over GitHub and restructured permissions with the web portal they built themselves. In the process they made too many repos accessible to too many people.

> To keep with the metaphor, Ubiquiti couldn't even get Pre-school level security in place, much less 101. I have no idea how something even more massive hasn't happened yet. Must be dumb luck.

Ubiquiti's overall structure is far from perfect, but you were only there during the Nick Sharp era. Ubiquiti had a lot of people who took security and proper practices very seriously before Nick Sharp took over everything, but it was also a distributed company with a lot of isolated divisions. Nick Sharp got into power by taking the worst and oldest parts of the company and convincing people that everything was equally bad and that only he could fix it. If you got your security information from Nick Sharp, you'd think that Nick is the only person who can do anything properly at the company.

> Speaking of, by the time I left the company, the team that was handling the door entry-way systems (UniFi "Access" I guess) had been caught with numerous security issues, not the least of which was logging user credentials in plain text (not just storing, but logging, in response to authentication events). They were also based in China and subject to Chinese laws around government access, so take that how you will.

I also heard that, but I think it was just incompetence on their part. Nick was pushing the conspiracy that they were doing something with the Chinese government, but it doesn't follow that they'd do it by sending the data to AWS servers under his control. I think they just made a sloppy prototype to impress the CEO and got caught doing dumb stuff. I do blame the company for not cutting that team off, though. They had no idea what they were doing other than their ability to put together quick prototypes to impress the CEO.

Ex-Ubiquiti employee here. Nick Sharp wasn't just a senior software engineer. He was the Cloud Lead and ran the whole cloud team. His LinkedIn profile will confirm it. This is why he had access to everything.

Nick had his hands in everything from GitHub to Slack and we could never understand why or how. He rose to power in the company by claiming to find a vulnerability that let him access the CEO's personal system, but nobody I spoke to ever knew what the vulnerability was. I discussed this with another ex-Ubiquiti person in an old thread [1] Now I'm positive he faked the security issue as a power move, just as he faked this attack for extortion purposes.

He would also harass people and use his control over Slack and GitHub against the people he didn't like. Many people left around this time partially because Nick made everything so difficult at the company. What a terribly depressing series of events.

[1] https://news.ycombinator.com/item?id=26694945

I worked there and I didn't even understand why we had to force cloud logins on Dream Machine. In the early days we were all about letting people run their own controller hardware and not requiring cloud logins. No one could ever tell us why we had to force everyone to the cloud. It was a mandate from above

Several of us from the UniFi team went to competitors but we're focused more on enterprise.

We always thought MikroTik was one of the biggest competitors for low cost equipment. Our main advantage was the UBNT community and having famous supporters like Troy Hunt which MikroTik didn't have. The community fell apart after the redesign killed it and I don't think people like Troy Hunt will endorse Ubiquiti now so it should be interesting to watch what comes next

There was a lot of infighting and turf wars when I finally quit. I'm not even surprised that this latest turf war spilled into the news.

The gear is locked down to UniFi firmware. Some of us wanted to open it up to alternatives like OpenWRT but that wasn't an option for us.

Is that going to the trace service?

There was a falling out between teams while I was there because the cloud team wanted to collect stats from APs even when users disabled analytics in the UI. It was so bad that some of the developers and one of the leads quit because they didn't want to be a part of it.

Someone on Reddit started reverse engineering it https://www.reddit.com/r/Ubiquiti/comments/lwr4ud/update_ubi... The APs are reporting things like connected clients and client stats according to recent dumps. Do you have analytics disabled in the UI and this is still happening?

He was in charge of cloud when he "found" a way to forge Ubiquiti SSO logins for any user using his root access to the SSO signing secrets.

In the Krebs article the whistleblower calls out forging SSO logins as one of the things that was compromised. If the attacker is really an ex-employee like Ubiquiti says, then it's scary that the SSO signing keys aren't even being rotated after the account forgery stunt.

> Adam says the attacker(s) had access to privileged credentials that were previously stored in the LastPass account of a Ubiquiti IT employee, and gained root administrator access to all Ubiquiti AWS accounts, including all S3 data buckets, all application logs, all databases, all user database credentials, and secrets required to forge single sign-on (SSO) cookies.

> they’re definitely moving in a little bit of a different direction then where many of us would hope

it pains me to say this because I was there for the UniFi glory days: The old Ubiquiti is dead and gone. Almost everyone I know quit.

I hope they can land on their feet and return to the glory days but I don't have much hope. The company got toxic fast at the end

> There was no company wide communication, and all communication channels were made private

I couldn't understand why the ex-Amazon cloud lead was also in charge of Slack. When he made all channels private and put a Slackbot in every channel to monitor conversations, I knew it was all over. I'm worried his Slackbot logs are part of the leak. Guy had his hands in everything :(

Same guy who took over GitHub and forced everyone into his self hosted source control because he couldn't trust Github. That decision didn't pay off.

Most of the US leadership and many of the US employees quit in recent years. The CEO wanted to focus on international offices where employees were cheaper. It was backfiring while I was there and I heard it only got worse after I left.

Sad situation. I knew a lot of good people there who cared about making good products during the UniFi glory days. Everything collapsed fast. I knew we were in trouble when the CEO's early employee friends were disappearing and their offices were being closed without warning.

I remember the cloud lead they hired out of Amazon was as toxic as they come. If he's still in charge I can see him blaming his own team members.

The culture at Ubiquiti collapsed in my last year there. The company was unrecognizable because everyone was quitting so fast.

The early days at Ubiquiti were good. I worked with a lot of good engineers and we shipped good work. The decline is a recent problem.

> How the brand hasn't become toxic already is a mystery to me, yet look at the stock price tracker. It's been trending up for years and it has well over doubled in the past six months alone.

This is your answer. No incentive to change. All of the bad engineering decisions have been rewarded by increasing stock price and continued sales.

Most of the original engineers have quit by now. I lost track of how many UniFi engineering leads joined and then quit after it started falling apart. Before I quit, I heard rumors that the CEO was making two separate teams work on the Dream Machine project separately, competing against each other. That made more people quit. I think they were trying to reboot engineering in foreign countries when I left because it felt like we were forgotten in the US offices.

I worked at Ubiquiti while you were there. I can confirm that the company was going downhill fast.

The US offices were starting to feel empty because so many people were leaving the company. Only place I've ever worked where engineers would quit before they got another job.

Saddest part was all the wasted potential. There were good engineers making good products at Ubiquiti only a few years ago. Once UniFi exploded in popularity the CEO started trying to micromanage everything and it all started falling apart.

The directors and VPs quit, also.

Yes! The move to low cost countries is confusing because the company has more than enough profits to pay high wages. I think the company wanted offices in cheaper countries because it was easier to convince the employees to accept the abuse because they had fewer options for employers who could pay well.

In the US, good engineers don't have to tolerate abusive companies in exchange for good wages. Not all countries are so fortunate.

Some of the best engineers I worked with were in non-US offices. They were also smart enough to leave.

You cut my quote short of the important part which was "and employees complained less about constant crunch mode". It is easier to "trap" engineers in countries where average salaries are lower by offering them 20% over market rate and then threatening to fire them if they don't work constant crunch hours. We were promised very large bonuses that never arrived.

> That said I don't hear the best about the company (long working hours, not the highest pay)

It is a sad situation. The pay was very good and the working hours were reasonable when I started, but that changed for the worse. That is why all of my peers left the company.

Ubiquiti had a steady exodus of engineers in the past few years. It's a very different company now compared to the glory days of UniFi.

As a former Ubiquiti employee, I'm sad to watch the slow decline of the company. There was a steady exodus of engineering talent through 2020. The CEO was focused on moving to countries where engineering was cheaper and employees complained less about constant crunch mode. If you search around, you can find interviews where he brags about closing the San Jose office because he thought everyone there was too entitled.

The saddest part is that we had many good engineers who could have continued to do amazing things with the UniFi momentum. So much time was wasted on dead end products like FrontRow. Most everyone I know left for jobs where we were treated better and paid more.