HNHacker News
TopNewBestAskShowJobs

kevin_nisbet

1,669 karma · joined January 30, 2013

submissionscomments
kevin_nisbet··on Ottawa wants the power to create secret backdoors in networks for surveillance
Radio Access Network - basically think the cell phone towers and the equipment for those towers.
kevin_nisbet··on Ottawa wants the power to create secret backdoors in networks for surveillance
I was going to add the same thing, this isn't only something the telecom providers do, the equipment providers include lawful intercept features as part of the equipment, the telecom just has to setup access. I don't have a chance to check right now, but I think this is even part of the published standards.
kevin_nisbet··on Ask HN: Founders who offer free/OS and paid SaaS, how do you manage your code?
One company I worked for did the enterprise version through a git submodule. So main repo was the OSS version (with it's own main), and the git submodule was a different program entrypoint that contained / activated the enterprise code. So devs would have both the OSS/Enterprise module checked out and could work on both together.

In this example as I recall most of the enterprise code was just activation code, so a properly motivated individual could probably figure out how to turn on some of the enterprise features, but most people won't bother to figure out how to do their own builds, and it's more work then finding the secret env variables in an open repo.

It's a little bit painful at times because changes that cover both code bases need two commits, and it isn't uncommon for someone to forget or commit the wrong submodule pointer. But depending on the team, it can work.

Not for everyone, just including it as an option I haven't seen mentioned in other comments.

kevin_nisbet··on Tech Jobs Leaving California
Seems like a misleading chart since there are both a left side and right side scale. I can’t tell which is which but my guess is it’s to make Texas stand out more.
kevin_nisbet··on Why use ECC? (2015)
Early in my career I had to deal with similar issues, but with cellular networking equipment. I do think it's one of those things where if you have a way to detect the bit flips it happens way more than you think... but most people can't detect them and mostly get lucky enough that they never see an impact when it does happen.

But because we had some equipment that would checksum itself we were able to see the bit flips happen, probably every 1-3 weeks or so I'd find a bit flip.

kevin_nisbet··on Multiple Planes Cross Runway While Another Is Taking Off [video]
I have the same question in my mind... seems like there is one of these reported every month or two and it's just asking for a more serious mistake to be made.
kevin_nisbet··on The Airship to Orbit Project
I don't think ion thrusters would work, for the balloon to maintain orbit you'd still be somewhere with enough atmospheric drag to be a problem. Ion thrusters only produce millinewtons worth of thrust, so they're very much relying on a no drag environment and the ability to run for continuous use for weeks to years.
kevin_nisbet··on Unikraft Launches KraftCloud: Never Pay for Idle Again
This looks awesome, congrats on the launch.

I've been pondering this type of issue for awhile, not from an application standpoint, but from a dev environment where I want more resources than my laptop has, but it'll be very bursty workloads and I want them to just pause when I'm not using them. So I'll be excited to see how this tech develops... good luck!

kevin_nisbet··on Avoid blundering: 80% of a winning strategy
Also luck is probably a much larger factor than most people realize. It's not just bad luck leading to failure, but luck leading to customers/revenue.

As an example, if no one is competing with you, it doesn't matter too much how good you are.

kevin_nisbet··on Ask HN: Doom and Gloom but financial markets doing well?
Please don't consider me an expert, but here's sort of the birds eye factors I've been reading or seeing over the last few months / my opinion on what seems to be happening.

Right now there does seem to be some interesting disconnects in some measures of the economy. I'm talking things like business revenues in financial results, compared to the various surveys that are done. One of doom and gloom measures is a survey that asks how people think the economy is doing. But where it get's interesting, is surveys that ask about personal finances appear to be showing fairly strong results. So there seems to be a broad sense that people as individuals are doing pretty good, but perceive the economy as bad for others or in general.

I think another angle and personal bias of mine, is Hacker News is a specific tech community with lots of exposure to only a portion of the economy. So we're seeing layoffs and difficulty in the labor market ourselves, and expect that to be happening across the broad economy, especially compared to 2 years ago. But I think there are a few factors that impact us differently. I think a primary one is going to be interest rates. For startups, which are going to light a pile of money on fire and hope they grow, there are other areas of the economy that are much safer to get those returns with current interest rates. Why fund a speculative startup when I can get 9% on a private loan to a business. Also you look at medium and big tech companies, those bonds (debt) are way more expensive competing for those same interest rates, so the investment into growth and expansion is way down compared to 1 or 2 years ago. And we get to perceive that through the big layoff announcements that seem to keep popping up, as interest rates have risen. So the B2B space, which eventually serves other non-tech companies and consumers, is soft on growth and expansion, which is often rooted back to funding through debt instruments.

But if you look at the broader economy, for the average worker in North America, wages are going up at or faster than the rate of inflation, the unemployment rate is low. We're probably sitting around full employment, which actually can be it's own problem. The inflation took a shock but is trending back towards norms. Consumer spending even with current inflation rates hasn't really stopped, purchases don't seem to be getting deferred all that much. So companies where there was an expectation of a pullback in spending haven't really seen it in their financial results, maintaining their value on the open markets.

Another angle for people to perceive doom and gloom is also affordability factors in some segments of the economy. Where I am housing is a disaster for many reasons, so I'm sure many people see being unable to buy the same quality of living as their parents as part of the doom and gloom. But the overall financial markets aren't reacting to this, because enough people still have those mortgages, and with the rise in interest rates, defaults don't appear to have meaningfully risen. Overall debts are getting paid, even if they're more expensive.

And another angle I want to tread carefully into, is the politics and policy angle. With an election going on, as I understand it one of the most correlated factors to a change in governance is the health of the economy. So there may be some motive to color financial data or results through a lens to try and produce a political outcome. I have no idea what overall impact this might have, but some of those surveys I talked about above when divided by political lines or where people receive their information get very interesting. I'm not trying to have the political discussion or take any side, I'm just saying watch out for motives in your sources of information.

And there are just overall expectations, inflation has been difficult, interest rates have risen, so we just expect things to be bad. Also keep in mind one of the disconnects in the financial markets is where would someone put that wealth instead, what in the current situation would be driving people to exit the market and try and park cash for example?

So this is probably a very long way to say, from what I've seen, when you try and measure peoples perception of the economy, those measures appear to show the economy in poor shape. But when you look at other measures that influence the overall markets, like sales, defaults on debt, consumer spending, those measures don't seem to paint the same picture.

kevin_nisbet··on Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users
> I just always imagined the GSM spec was so resilient that you could always make a 9-1-1 call if a working network was available but this outage proved that wrong.

As I recall it is slightly more nuanced than this and was particular to the failure mode, and has a couple of different things aligning to create the failure mode.

If you're phone is just blank, no sim card. To make an emergency call, it has to just start scanning all the supported frequencies. This is very slow, tune radio, wait for the scheduled information block that described the network on the radio protocol. See if it has the emergency services bit enabled. If not, tune to next frequency and try again. I used to remember all the timers, but almost a decade later I can't remember all the network timers for the information blocks.

The sim card interaction, is say you're at home and you boot up your phone with 100% clean state. You don't want to wait for this scan to complete, so the SIM card gives the phone hints about which frequencies the carrier uses, so start on frequency x to find the network. But if you roam internationally, it can take alot longer to find a partner network, and there are some other techs around steering to preferred partners, but I don't know that those come into play here. I don't know but would be surprised if there is a SIM option to try and pin the emergency calls to a network, I think it's more likely the interaction is this hint on where to start the scan.

The way the rogers network failed, it appears to me it caused the towers to stay in a state where they advertised in their radio block the network was there, and the 911 bit was enabled so the network could be used for emergency calls. This is where I don't really have the details since they haven't been public about it, how much of their network was still available internally. Maybe the cell towers could all see each other, that network layer was OK, and the signalling equipment was all talking to each other as well. That's the part I don't really know and have to speculate, as well as the tower side since I was a core person. So because the towers had enough service to never wilt themselves, they kept advertising the network, along with the 911 support. But then when you try to activate an emergency call, somewhere in the signalling path, as you get from tower to signalling system, to the voip equipment, to the circuits to the emergency center the outage knocked something out. Oh and for all these pieces of 911 equipment, there are two of everything for redundancy... two network paths, two pieces of equipment, etc.

And because they lost admin access to their management network, no one could go in manually and tell the towers to wilt themselves either.

If the towers had just stopped advertising 911 services, the phone would fall back into the network search mode as I described when you have no sim card. It just starts scanning the frequencies until it see's an information block for a network it can talk with the emergency support advertised to and does an emergency attach to the network that the carriers will all accept (An unauthenticated attach for the sole purpose of contacting an emergency center).

So my suspicion is because carriers are so used to we have two of everything, and all emergency calls are marked for priority handling at all layers of the equipment (they get high priority bits on all the network packets and priority CPU scheduling in all the equipment), this particular failure mode where there was a fault somewhere down the line, and they lost control of the towers to tell them to stop advertising 911 services all sort of played together to create the failure mode.

kevin_nisbet··on Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users
> Also fun where the towers remained just powered on enough for phones to stick to them but not be able to do anything, so 9-1-1 calls would just fail, instead of failing-over to other networks. Seems like a deficiency in the GSM spec (or Rogers SIM programming?) that I don’t think was actioned on.

Actually, I think this is going to change after the Rogers outage, it's just slowly happening behind the scenes so it's not getting much attention these days. The government has mandated a lot of industry response to failover between providers... we'll see where they land after all the lobbying happens. I do think implementations are changing a bit around this, mostly in the phones so that they give up and go into a network scan if the emergency call is failing.

I worked mostly on core network stuff, so I was a layer removed from the towers, but if they hadn't lost management access they would've been able to tell the tower to stop advertising the network and 911 service. I do understand the question of from a vendor implementation perspective of how automatic this should be though... because automation in this regard does have some of it's own risks and could complicate some types of outages or inadvertently trigger and confuse recovery of problems.

I'm with you though there should be an automatic mechanism to fail over to other network operators, I just haven't thought through all the risks with it and I hope the industry is taking their time to think through the implications.

kevin_nisbet··on Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users
To everyone trying to speculate on the root cause, I haven't seen enough information in any of the comments to really draw any conclusions. Having worked on several nationwide cellular issues in Canada when I worked in telecom, we saw nationwide impacts based on any number of causes.

- A new route injected in the network caused the routing engines on a type of cellular specific equipment to crash nationwide. This took down internet access only from cell devices nationwide. But most people didn't notice because it happened at 2AM maintenance window and was fortunately discovered and reversed before business hours why the routing engine was in a crash loop.

- A tech plugged in some new routers, and the existing core routers crashed and rebooted. While the news worthy impact was just a regional outage for something like 20 minutes, we discovered bugs and side effects from the Pacific to Atlantic coasts over the next 12 hours. So when you say you're impacted at location x, that data point could be everyone is down in the area, many people are having issues, or only one or two people have issues spilled over to some other region. This is why seeing it does or doesn't work in location x is limited value, as almost every outage I've investigated could result in some people still having service for various reasons. The question is in a particular area is it 100% impact, 50% impact, or 0.001% impact.

- A messaging relay ran into it's configured rate limit. Retries in the protocol increased the messaging rate, so we effectively had a congestion collapse in that particular protocol. Because this was a congestion issue on passing state around, there were nationwide impacts, but you still had x% chance of completing your message flows and getting service.

And then there was the famous Rogers outage where I don't remember them admitting to the full root cause. It's speculated that they did an upgrade on change on their routing network, which also had the side effect of the problem booting all the technicians from the network. Then recovery was difficult because the issue took out the nationwide network and broke the ability for employees to coordinate (you know because they use the same network as all the customers who also can't get service). All the CRTC filings I reviewed had all the useful information redacted though, so there isn't much we can learn from it.

So it's fun to speculate, but here's hoping at the end of the day ATT is more transparent then we are in Canada, so the rest of the industry can learn from the experience.

kevin_nisbet··on Automated Unit Test Improvement Using Large Language Models at Meta
Yea, I've seen this get carried away even by just individual team members.

My personal favorite was tests that a team member introduced for an object that had all the default runtime parameters in it. Think like how long should a timeout be by default, before getting overridden by instance specific settings. This team member introduced a test that checked that each value was the same as it was configured to.

So if I wanted to update or add a default, I had to write it in two places, the actual default value, and the unit test that checked if the defaults were the same as the test required.

kevin_nisbet··on Freenginx: Core Nginx developer announces fork
I'm very out of date so take my opinion with a grain of salt. The customer support I received from F5 when they acquired a telco product was about the worst support I've ever seen. Now this wasn't the general LB equipment that F5 has the reputation around, it's some specific equipment for LTE networks.

We'd get completely bogus explanations for bugs, escalate up the chain to VPs and leadership because there was an obvious training, understanding, and support for complex issues problem, and get the VPs trying to gaslight us into believing their explanations were valid. We're talking things like on our IPv4 only network, the reason we're having issues is due to bugs in the equipment receiving IPv6 packets.

So it's one of those things where I've personally been burned so hard by F5 that I'd probably to an unreasonable level look for other vendors. The only thing is, this was awhile ago, and the rumor's I've heard are that no one involved is still employed by F5.

kevin_nisbet··on What it was like working for Gitlab
> Thankfully many newer companies seem to be going towards 10 year exercise windows, but they are still a minority

I'm actually a bit on the fence for the 10 year exercise window, as I'm not sure it's always a benefit. As I understand it, startup exits timelines are on an upward trend, so in 10 years will the average exit still be below 10 years, and even then are you working for a company that meets the average timeline. Then, from a taxation perspective, if you do want to exercise before the 10 year expiry, if it's cost prohibitive now, it should be bankrupting after a few more investment rounds and increases to the 409A valuation.

So I'm curious if there is a side effect here that the 10 year exercise window actually leads to many more shares becoming even more cost prohibitive to exercise right before the company reaches a liquidity event and as such not exercised (if you're in the early stage employee group).

kevin_nisbet··on What it was like working for Gitlab
I'm not a lawyer so take the comment with a grain of salt, but employee's should likely insist on using company devices as well. If the company were to be sued it may open up the personal devices to all sorts of subpoena for the contents of the device.
kevin_nisbet··on What it was like working for Gitlab
I largely agree, especially with the sentiment that there are two sides to any story.

I think one angle I'll nitpick though is this doesn't have to be related to whether or not the employee was a high performer or had particular leadership needs or whether they're adaptable to the current maturity level of the org. There can and likely are many other elements such as whether the manager/employee even get along, or agree on direction, strategy, problems, etc.

And the difficulty is, these types of issues can often show up as performance issues. In lots of cases, is the performance issue the employees fault or the leaders.

In my anecdotal experience where a job went from the best role I ever had to one of the worst happened over a very short week or two with the change in a manager. And I think to the point you're making, when I resigned I outright said... each of us is going to blame the other for my departure. The hard truth is the answer is likely if either of us had been different people I'd still be in that role doing work I really enjoyed, but in that particular leader/follower relationship we simply didn't get along well at all.

If I ever did a what it was like or why I left post, I'd probably have plenty of arguments for my side, because I think to your point, I'll be the hero in my own story.

kevin_nisbet··on Cutting down AWS cost by $150k per year simply by shutting things off
I've for a long time set my cloud VMs to shutdown on idle. I usually use it to also justify running a much larger VM to cut down on build and test times.

Just set a cron to run the shutdown command with a grace period. And then if you're working late, you just cancel the shutdown and the shutdown will be retried in a couple of hours. And have a script or command to just run the cloud API calls to boot the VM in the morning / when needed, and the environment boots in a minute or two.

For other stuff I've been tempted to do a more complicated setup, with something like a micro-vm as a proxy, that will do the shutdown / activation on TCP connection, but haven't gotten around to it.

kevin_nisbet··on Ask HN: 9-yo son wants to build a game, I'm lost. What can I do?
It might be difficult for a 9 year old but there was an interesting series on YouTube or twitch called handmade hero. I didn’t follow it all the way through but the person hosting it did a weekly livestream and just started writing a game and engine from scratch in c. I don’t know if it’s still active but there are 600 videos in a YouTube playlist.

Even if not sticking with games I think it’s possible to learn a lot from the series.

kevin_nisbet··on Apple built iCloud to store billions of databases
I could be wrong, but I sort of think when I learned Oracle back when I was in school (mid-2000s) supported dropping a database on a raw block device. So it's been around a long time, but would be uncommon in some tech circles.
kevin_nisbet··on Slashing data transfer costs in AWS
Managing on-prem hardware may not necessarily be hard, but it can be extremely time consuming. To me, the nice thing about dropping a bunch of hardware in a colo is you get to take a lot of shortcuts and take risks that you cannot buy from the public cloud providers.

I worked for a company and would do it again that did the colo route, and it gave immense cost savings compared to public cloud, taking on risks that you can't do elsewhere. Before they started investing in having folks take care of the infra as a raw startup, it was just some servers and some desktop unmanaged switches. But that gave the company breathing room to survive as the business model probably didn't work without it. But also had a reputation for unreliable service.

I've also built the five nines infra at telcos, and yes you can do it with average engineers, but it's going to be time consuming, slow, and expensive in costs and labor. To allow 26 seconds of unplanned outage a month, you're going to be testing every firmware update for every piece of equipment on an ongoing basis, and practicing every operation and change as best as possible. And you need the scale that you get that 26s by having most outages only impact a subset of your customer base, otherwise you're going to blow that outage budget fast.

kevin_nisbet··on A supply chain attack on PyTorch
Absolutely. The real difficulty is tests on PR are by definition remote code execution by an untrusted source, so a full risk analysis and hardening needs to be done.

Here's a similar mistake on an OSS repo a company I worked for made: https://goteleport.com/blog/hack-via-pull-request/

kevin_nisbet··on Gas pumps freeze at Calgary gas stations [video]
> Not good enough to trust your life to if you’re an Albertan.

I believe the heat pumps intended for these climates provided alternatives for those cold days. If you want to stay all electric, it would include a resistive electric heater, but could be paired with some other fuel source.

Personally, I'd seriously consider going ground source where temperatures like this are the norm, but that doesn't work for everyone either.

kevin_nisbet··on Malware abuses Google OAuth endpoint to 'revive' cookies, hijack accounts
Sorry about the imprecise language, this system rotated both together.
kevin_nisbet··on Malware abuses Google OAuth endpoint to 'revive' cookies, hijack accounts
Yea, I equate it to part of security theater.

I worked on a product that rotated the TLS certificate frequently. And it actually showed up a number of times in questions from customers or vendor security questionnaires about whether we rotated the certificates and how that happened.

But what we were never asked was whether old certificates were cancelled... which in that system they were not. So it didn't matter how many times we rotated our secrets, any old or leaked secret in a backup or elsewhere was still completely valid. But we had met the security theater that those rotations happened.

So I expect what you do, is that changing a password would cancel all sessions using that credential. But that's kind of hard to do, so we'll just leave that side buggy and untested, because we did the important part of the theater that said we can change passwords.

kevin_nisbet··on Malware abuses Google OAuth endpoint to 'revive' cookies, hijack accounts
I doubt google needs to use this as a feature for tracking.

Even if we argued that this was for tracking purposes, google could keep the cookie for tracking and just deny access to the services until a login flow was completed.

kevin_nisbet··on Kubernetes Needs an LTS
> It'd be a big lift (if not impossible) to get projects like cert-manager cilium whatever to adopt the longer release cycle as well.

It's a great point which probably should be part of the discussion. Say even if Kubernetes project offered LTS, how would that play into every other project that is pulled together.

> Is it normal for a distribution to also package upstream projects that customers want?

I suspect it differs by distribution. The distribution I worked on included a bunch of other projects, but it was also pretty niche.

kevin_nisbet··on Kubernetes Needs an LTS
From my perspective as a former developer on a kubernetes distribution that no longer exists.

The model seems to largely be, the CNCF/Kubernetes authors have done a good job of writing clear expectations for the lifetime for their releases. But there are customers who for various reasons want extended support windows.

This doesn't prevent the distribution from offering or selling extended support windows, so the customers of those distributions can put the pressure on those distribution authors. This is something we offered as a reason to use our distribution, that we can backport security fixes or other significant fixes to older versions of kubernetes. This was especially prevalent for the customers we focussed on, which were lots of clusters installed in places without remote access.

This created a lot of work for us though, as whenever a big security announcement came out, I'd need to triage on whether we needed a backport. Even our extended support windows were in tension with customers, who wanted even longer windows, or would open support cases on releases out of support for more than a year.

So I think the question really should be, should LTS be left to the distributions, many of which will select not to offer longer releases than upstream, but allow for some more commercial or narrow offerings where it's important enough to a customer to pay for it. Or whether it should be the responsibility of the Kubernetes authors and in that case what do you give up in project velocity with more work to do on offering and supporting LTS.

I personally resonate with the argument that this can be left with the distributors, and if it's important enough customers to seek out, they can pay for it through their selected distribution, or switching distributions.

But many customers lose out, because they're selecting distributions that don't offer this service, because it is time consuming and difficult to do.

kevin_nisbet··on Rivian software update bricks infotainment system, fix not obvious
Yes. And also things like rolling out the update in batches, and then also things like golden images, where if there are two crashes or failures in the first 24 hours of the update, change to the last known good software version.
← PreviousPage 4 of 16Next →