1,669 karma · joined January 30, 2013
Sorry for being unclear, the vendor was attacking our organization only, and any other company was expressly forbidden in the contract. As I recall it was a fake SSO sign-in page to collect credentials that they would try and social engineer our employees with.
The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.
sync.Once - Run exactly once, only the first call is invoked.
singleflight - Merge concurrent requests into one request and return the response to all the callers.
Where I tend to use both, is sync.Once tends to get used for lazy init code, the first caller does any client initializations, and subsequent callers wait, and then the lazy init is done and never done again in the lifetime of the application.
For singleflight, it tends to be on merging relatively expensive requests together. Like retrieving and parsing a JSON object from a server in concurrent requests. Merging those requests together, doing the expensive work once and distributing the results to each concurrent caller type of idea. With the results becoming invalid over time so a later set of requests need to do it all over again.
I don't think I can claim credit because I'm sure I wasn't the only one, but it took 3 or 4 emails to a couple of legislator offices to get some policy changes. In my case it might've just been small enough (no news coverage, basically only a small number of people were aware of a regulatory memo), the first time or two they just kicked the can down the road deferring the implementation, until ultimately they reversed course. And my part was just laying out a very strong case for why the particular situation was unfair, how many people would be impacted (voters), etc. Nothing confrontational, just laying out the argument.
How that maps into the hardware I don't know.
Sorry if I was unclear about that.
I started my career in Wireless Telecom, and some of my work was directly on Diameter Routing and working with roaming partners on technical problems.
The SIM card stuff I have no expertise on, so I won't comment on that.
There are a couple of important things to note which I think the report misses.
The first is what a cellular network does for tracking a user. It's not returning a set of GPS coordinates. A cellular network has a problem to solve, which is on which radios do we transmit that the device should wake up from idle about an incoming call (or SMS or packet). We could not track this at all, and then the entire bandwidth is spent notifying the entire country about every call. On the other side we could track this down to every radio, but then you have the problem of your entire network is just signaling traffic about changes in the best radio to reach a device. A tree swaying in the wind causes constant updates type of idea.
So we break the network up into areas, and if the device moves and see's a beacon from a tower that it's in a new area, it tells the network to update it's location records. There is a slightly more precise record which is the Cell ID, but the device doesn't need to keep it up to date. So in the report when you see the references to the Cell ID/E-UTRAN Global Cell Identity, and LAC/TAI this is the concept those identifier tie to.
There are databases that can map the Cell identities to GPS locations, and you can think of that as the assignment to the tower, although there are some deviations that occur (remote locations for a radio, etc). So most often you're not getting the GPS location of the device, you're getting the tower. This is still a privacy implication, and get's more precise over time as the radio networks get more and more dense to support higher speeds. But still might be dozens of KMs away from the actual device location.
I ended up writing a novel, but need to cut it down. So a bunch of the evidence cited on the Diameter protocol and DNS behaviour on GPRs DNS I don't think is as strong as one might conclude when reading the report.
What particular struck me was the DNS NXDomain as an indicator of trade craft to conceal the source, that they refer to several times. To me this is an expected behaviour on roaming DNS if the source network used to make the query does not have a roaming agreement with the other provider. The DNS specifications by the GSMA are a bit awkward, and I've been bitten by them several times, but they don't carry the same DNS related expectations as you would expect on the internet. On the open internet, you expect to see a delegation from com to ycombinator, and to be able to follow that to ycombinator. This isn't the case on the roaming exchange networks, firewalls and answers are only opened up when there is an agreement. So if this provider is a small fry, there might not be many agreements in place, and it's not weird to get back an NXDomain or timeout. This does depend on whether it was the roots or the provider certs that provided the response that they don't go into detail.
Some of the Diameter related statements also struck me as not having a complete understanding of the technology, and suggesting a direct link to trade craft. Things like the Origin-Host and Route-Record 1 being the same, while perhaps a technical violation of the standard, have no impact, and can just as easily be explained by a network operator not wanting to advertise internal details to the rest of the world. Similar, with the IPX provider not detecting the mis-match between realm and host, I'm not even sure that's expected or how they would do it, although from an analysis perspective it is a clear screw up from the adversary that revealed it. But I don't remember sufficient information getting exchanged between providers that would actually allow an enforcement over those fields, but I could be wrong. Also keep in mind, it wasn't just the roaming exchange that didn't enforce it, all the networks also failed to enforce it. And now the adversary might just see this report and fix their bug, so it's not like that enforcement would've completely changed the situation. But they do have a point that if enforced, it might've been detectable earlier that there was a bad actor present.
There are also alternative and resonable explanations for some of the other claims, like was 019Mobile the one actually relaying the messages, or was the second hop tricked into accepting messages, and the adversary was just impersonating 019Mobile. That shifts things around a bit, they talk at length about 019Mobile being the source of these messages, but while likely, there are other plausible explanations for the origins of those messages.
There are also some technical details they got wrong, but probably not in material ways.
So it's an interesting report, it seems like there is a real operation going on, just that I take issue with much of the evidence cited that I think many readers may draw a strong conclusion from then they should.
If your site is for B2B and impacts customers own operations or revenue, you'll likely be wanting to chase the 99.9%, customers won't tolerate the 1.5 hours per week of downtime and will churn.
However, if the value you're site creates is tolerant to those sorts of disruptions, someone is just inconvenienced and can come back later, a large investment to move from 99% to 99.9% wouldn't be justified. There is literally no impact from the investment. The harder part will be the reality, most investments will be somewhere in the middle with ambiguity on the impact. IIRC, SRE principles do talk about this when setting SLOs in different terms.
I've heard some companies refer to the concept as economical thinking, which is I think a great way to think about it. Doesn't mean you'll always get it right, more so that we embed being conscious about the ROI in our work.
I also believe this is an area that I've observed several engineers really struggle with, especially when moving from big tech to startups, where it's really easy to import culture from another company, and in earlier stages of startup life... if you don't have product-market-fit, it doesn't matter how good you're availability is. Attention is a resource, make sure it's allocated to what creates value for the customer.
However, with the tables going back to 2022, it's almost 100% increase of businesses and 50% increase for Non-business filings, which I find more interesting.
Would be great to see the totals as population adjusted against historical context though.
To me it's like anything else in engineering, are the costs, risks, and benefits fully understood, and worth the tradeoff in the particular context.
I worked for a startup doing internet of things, the consumer would buy a device and get lifetime service baked in. And that company was a step further, just renting space in a colo was incredibly cost efficient, which supported the sales model and competitive landscape of that product. But it was also very costly to attention, one of the most valuable resources. But it can also get costly in non-intuitive ways, an example that comes to mind is we started to get interviews where a generation of candidates no longer had experience with metal, it was a foreign world to them.
With more experience, I find it's really the costs that get severely underestimated, both for and against the suggestion.
What triggered me to go back was I never got a really clean mental model for how to keep ontop of Github PRs, bring in changes from origin/main, and ended up really badly mangling a feature branch that multiple contributors were working on when we did want to pull it in. I'll probably try it again at some point, but working in a team through Github PRs that was my main barrier to entry.
It might be OK to place some bets on an initiative or feature, but if we all understand we're placing a bet, this is an area to load up on debt and really minimize the investment. This also requires an org that is mature about cutting the feature if the bet doesn't materialize, and if the market signal is generated will reinvest in paying down the debt. And also has the mega-danger territory of a weak market signal, where it's not clear if there is market signal or not, so the company doubles down into the weak signal.
Also these bets shouldn't be done in isolation in my view, well executed product and market discovery should also provide lots of relevant context on the ROI.
I suspect this is most apparent on things like meeting culture. Something happens and all of a sudden there is another recurring meeting on the calendar, with 15 attendee's, costing x dollars in wages, that produces no value for the customers because the lesson was already learned.
Or when reacting to an incident of some sort, it's so easy to have a long list of action items that may theoretically improve the situation, but in reality are incredibly expensive for the value they produce (or the risks they reduce). It's too easy to say, we'll totally redesign the system to avoid said problem. And what worries me, is often those very expansive actions, then cause you to overlook realistic but small investments that move the needle more than you would think.
And as a hot topic I also think the costs are an input into taking on tech debt. I know we all hate tech debt with a passion, but honestly, I think of it as a tool that can be wielded responsibly or irresponsibly. But if we don't know what our attention costs, we're going to have difficulty making the responsible choices about when and where to take on this debt. And then if we're not conscious about the debt, when it comes do it stings so much harder to pay down.
I think the most fun I had with the page cache was when one of the vendors "fixed" a bug where the memory calculation has previously excluded the page cache. These boxes were all network services, they didn't rely on the disk for anything more than holding the binaries, configuration, and logs. Where the fun comes in is the first time you grep through the logs, which fills the page cache, and sets off the alarms that the cellular network is about to die. ;)
Anyways, important concept to know and understand when it comes to how software performs when interacting with a host.
This happens with managed services and I understand the frustration, but vendors are just as fallible as the rest of us and are going to have wonky behaviour and outages, regardless of the stability they advertise. This is always part of build vs buy, buy doesn't always guarentee a friction free result.
It happens with the big cloud providers as well, I've spent hours with AWS chasing why some VMs are missing routing table entries inside the VPC, or on GCP we had to just ban a class of VMs because the packet processing was so bad we couldn't even get a file copy to complete between VMs.
I think this is the issue, is these datasets are out there and at least big tech companies have them since they're used to assist with GPS. I was about to post the same thing as above but saw vessenes beat me to it.
Without thinking about it too hard, the two directions I see are either making observations of the environment in real-time that is only relevant at that time (IE sniffing actual wireless frames, even if they're encrypted and making observations on them, however, most devices won't let you go into promiscuous mode and do this) or encrypting the messages in flight so only participants can decrypt them (IE a model like the signal protocol with E2E message encryption).
Anyways, this is a cool approach, but that risk occurred to me as well about the ability to just brute force the entire dataset to decode every location.
I ponder how effective this would be against an adversary sufficiently motivated to look like they're not using a VPN. And then does it result in a false sense of trust, since a user thinks the system more reliably detects a VPN then it does. Or an adversary who has bypassed the system to then point to it to build additional trust.
But from the article, turning off fsync and expecting to only lose a few ms of updates. I've tried to recover etcd on volumes that lied about fsync and experienced a power outage, and I don't think we managed to recover it. There might be more options now to recover and ignore corrupted WAL entries, but at that time it was very difficult and I think we ended up just reinstalling from scratch. For clusters where this doesn't matter or the SLOs for recovery account for this, I'm totally onboard, but only if you know what you're doing.
And similar the point from the article that "full control plane data loss isn’t catastrophic in some environments" is correct, in the sense of what the author means by some environments. Because I don't think it's limited to those that are management by gitops as suggested, but where there is enough resiliency and time to redeploy and do all the cleanup.
Anyways, like much advice on the internet, it's not good or bad, just highly situational, and some of the suggestions should only be applied if the implications are fully understood.
So I get a contract and am told it's been vetted and I should sign it. What I found was outrageous.
- If we cancelled for any reason, including if they just didn't do any of there terms in the contract, we owed the full price of the remaining contract immediately.
- The way they structured it was also as a rental, so we were paying full price for purchase of the equipment embedded into the term of the contract, but it was the vendors equipment, so if we cancelled we still paid them full price for the equipment, and they got to keep it.
- If there were any legal disputes, no matter which party was at fault, my side would pay for all the lawyers.
I said nope, can't do it. And my staff were pissed at me for like a year because everyone just signs those things.
Vanguard asset allocation ETFs are at like $1.3T [2]. 4 Of Canada's Big banks appear to add up to just over 2T Assets under management based on what Google just gave me as summary. So while I think this is a great outcome for a startup (even with Power backing them), to me it seems in a similar space as the above article that we're still talking a relatively small market share, and likely still closer to early adopter status.
[1] - https://en.wikipedia.org/wiki/Wealthsimple#:~:text=As%20of%2... [2] - https://www.vanguard.ca/en/product/investment-capabilities/a...
I'd say in addition to entrepreneurs, it's an important book for product teams / product engineers to understand what the Mom Test teaches, and tune the filter on asking the right questions to get the highest signal, and ensure the solution closely matches the value prop for the customer. Then sales and marketing get a whole lot easier when you've asked the right questions and solved the right problems.
If you are specifically getting targeted, there might be a slight delay by having the adversary try and exploit the honeypot ports, but if you're running a vulnerable service you still get exploited.
Also if you're a vendor, when prospective customers security teams scan you, you'll have some very annoying security questionnaires to answer.
To just expand a bit, I believe some of the confusion around printing of money comes from the way some economics reports are built. As a micro example, Assume a 10% required reserve, If Alice deposits $100 and the bank lends $90 to Bob. Alice ($100 deposits) + Bob ($90 cash) think they have $190 in total.
This is mainly useful for economists to understand, study, and report on. However, when the reports get distributed to the public, it looks like the banks printed their own money, as we now see $190 on the report when there is only $100 of cash in our example system.
Whether the system should work on a fractional reserve is it's own debate, but we need to know what it is to debate the merits and risks of the system.