HNHacker News
TopNewBestAskShowJobs

yebyen

2,755 karma · joined October 9, 2012

[ my public key: https://keybase.io/yebyen; my proof: https://keybase.io/yebyen/sigs/Q4xYtntxlQX51bnJaCqx5ERBtsA-G0clfn1VJVEY2F8 ]
submissionscomments
yebyen··on How a Jenkins Job Broke our Jenkins UI
I did a technical review of our Jenkins solution with a friendly neighborhood InfoSec agent once. He was interested in a few things:

1) how is the stored credential being encrypted (is it encrypted at rest),

2) is it real encryption (like, does it use an actual encryption algorithm or is someone confusing base64 or rot13 for encryption)

3) where is the encrypted data stored, where are the keys stored, is it properly on a machine that is not situated directly on the internet (users must access through VPN)

4) can I review the code that decrypts the secret (yes you can, there are plenty of walkthroughs and gists published by many different Jenkins fans and users that show where Jenkins stores its encryption keys, and how you can borrow them with POC code you can use to decrypt individual creds or even the whole store, assuming you have admin access to Jenkins)

He didn't really ask or care about sandboxing at all. There are mitigations you can apply for any threat model, but the crux of the matter is that even InfoSec assumed that any Jenkins users would probably have admin access anyway, and that they should all be allowed to decrypt those secrets.

Which happens to be exactly what they tell you in articles about how to manage your Jenkins server anyway:

https://www.codurance.com/publications/2019/05/30/accessing-...

> 7. Treat all credentials stored in Jenkins as plain text

> Treat everyone with access to Jenkins as an admin user, and you will be fine. Once you give someone access, even read-only, to a Jenkins it's game over. All developers on a project should know all secrets anyway.

The sandbox is just a mitigation that has been built in so that people who run untrusted code can do so with some degree of safety. Jenkins servers that may be exposed to untrusted contributors are a completely different animal than internal Jenkins servers. Different use modes experience different threat models differently.

The fact that you seem to believe there is a technical mitigation possible which would save this is potentially part of the problem, too. You're obviously a well-educated person. Organizations of sufficiently large size that settle on Jenkins can fight an uphill battle here, since many mitigations can only reduce risks, but cannot eliminate them. Leadership roles may not be inclined to understand the threat model issue, there is a dichotomy experienced when Jenkins is used to build software in the open vs closed developer models. Running builds for any PR from a random fork of a public repo is not the same threat model as running builds for any branch or tag in a private repo, from an IC with write access. And yet the mechanical actions are quite the same, and there are only about two checkboxes in a typical configuration separating these models from each other.

I hope we are rapidly reaching a place where companies can participate in both Open and Closed style of development, but if they are supported by Jenkins, I hope it's at least two separate Jenkins servers, because the threat models and risk mitigation that are needed for closed-access teams and open-source style public benefit repos are both completely different, even if their activities are not different.

Maybe I am overstating the risks, but I have been a Jenkins admin before, when it was not in my job description, and it was not supported by leadership, who did not believe in the benefits of open development, and made their posture closed by default (as companies are often wont to do.)

And as a result, I'm afraid, before I left that job behind, we dug the hole so deep in the wrong direction... if I could go back and do it again, I would certainly draw some bright lines in different places than we did, and I would not allow customer data in the dev instances. (You can do that kind of testing in a staging environment, where similar controls as production can be applied and reliably maintained. But you should do this as far away from Jenkins as humanly possible!)

yebyen··on How a Jenkins Job Broke our Jenkins UI
Because you can embed credentials in a Jenkins installation (and when you use Jenkins, we do.)

It inevitably gets connected to resources which can cost money, and since it's an internal infrastructure system, it will inevitably be connected with resources which contain replicas of private information. "Because why not, it's an internal system which means it's perfectly safe, and we should really be testing with real user data, you know, for realism."

And once you cross that bridge, you have wholly and truly gone completely radioactive. This is the intersection of financial risk/attack surface and private customer data, and execution of unproven code. If we send new Groovy scripts through an approval process, we can at least narrow the risk of accident or intentional disclosure by manually vetting scripts before they are run for the first time after changes, (or effective sandboxing so they cannot escape and access for example, any secrets that were not intended for them.) But, roundly, one can argue there's not much to be done, as it is an internal system, and each control we place in the way can easily be seen as an obstacle to getting the job done straightforwardly.

Then think of this also: maintaining Jenkins is known to be an operational burden, to say the least. If you have different departments with their own independent need for Jenkins or something like it, neither of these departments is going to want to own the Jenkins instance if it means being generally responsible for uptime and upkeep. You can bet that someone higher up is going to see this as a tremendous opportunity to consolidate and save money. They're going to wind up running on the same environment together, operated by someone who has no idea what either of these departments is really up to.

If you're lucky it's only two tenants, and that shared-service admin department responsible for the Jenkins instance is going to actively pursue the common interest of keeping things secure on behalf of everyone. What's more likely is, shared services platform team only ever hears from their Jenkins customers when something has ceased to function and their Admin access is needed, and for those people it's basically hair-on-fire can't-work-until the admin guy can be reached. After ten or fifteen times and some political pressure, their boss says this is taking too much of our department's time and we're not adding value, so they decide to shirk the admin duties, and give admin credentials to a "designee" from each team.

Now through diffusion of responsibility, nobody is in charge, and everyone is too afraid of breaking anyone else's stuff to ever upgrade even the most basic stuff.

That's right, now you have potentially private customer data, from multiple departments, with access to secrets that may control resources which can be scaled up to cost money, in an environment where code runs for the first time (potentially even code from third party contributors, but not likely if it's an internal instance... right? ...) code running before it has been tested by anyone, in a multi-tenant environment that nobody wants to pay or spend time to maintain, with competing czars that don't really have much incentive to talk to each other, where nobody has enough visibility to safely upgrade anything for fear it will break something and step on somebody else's toes, and if it goes down or stops functioning for basically any possible reason because of any of those teams, the maintenance of business is completely halted until we take care of it; really it's basically even worse than having cats and dogs living in the same house.

Sure, you can implement policy to solve any one of these things individually, but if you write enough policy to make it truly safe, then people will hate you for all the red tape required to work with it, "why can't we do the easy thing which is technically possible" and as you can see, there is so much wrong possible that even the best policy is going to have to concede some things, to make this tremendous expense worthwhile "otherwise why are we even using (footgun) if I can't easily shoot myself in the foot whenever I need to?"

yebyen··on The NFT market bubble has popped?
Thanks, that was probably my point. Consensus is hard. Ethereum is one of the global systems that currently exists for tackling the consensus problem. Shouting at clouds is on the other hand, not tackling the consensus problem.

Someone or something has to be at the helm of the command economy. I'm definitely not saying we should "go back to communal living and producing stuff based on actual needs" but I am saying that without a rigorous system for distributed consensus, you really don't have a chance of achieving this.

yebyen··on The NFT market bubble has popped?
> Spending billions of KWh on blockchains so rich people can speculate on buying their own number (correct me if i'm wrong but to me a NFT is just a number) is just part of this.

You say this as if numbers have no utility. Is an IP address "just a number"? What about a /8 cidr block? How much energy should we be allowed to spend to secure global address space and provide a secure implementation of identity for all people? This is one of the actual concrete uses of NFT tech, (see Urbit Azimuth.)

> We have to stop this madness, abolish private property and the capitalist industry entirely and go back to communal living and producing stuff based on actual needs, not dumb single-use Internet of Shit products and other planned-obsolete non-flashable-single-purpose computers.

Good luck making a non-disposable world where nobody has domain over their own numbers or is allowed to have their own distinct and unique identity secured by a leading consensus protocol. I for one am glad that you and others of a like mind are not empowered to tell me or anyone else that I can't have my own number if I want it, or to secure it with as many megawatts and global collaborators as I can muster.

Because you know what happens when private property is abolished? Spoiler, only the well connected and powerful will hold private property, and overall the scope and impact of inequality even broadens.

Edit: I'm getting downvotes but not sure which part you disagree with! Have you ever lived in an area where private property was abolished? (They tried this in Lithuania and USSR and that's exactly how it turned out! When nobody is allowed to own resources, who decides how they are allocated?)

"To each according to his need" sounds great until you get to see what an actual functional implementation of that looks like, and it turns out that Stalin is the one who is in charge of it all.

yebyen··on The anus is an evolutionary marvel
Our dog had an ingrown corkscrew tail (English Bulldog) which was amputated (laser surgery!) and anyway, I'm telling you this because our groomer does anal expression and said at first, it wasn't working for her because there was no tail to grab onto.

There is definitely a technique, but for $5 or $10 extra at the grooming appointment I think I can afford to go a while longer yet before I try to figure it out for myself - I mean, I know you can go on YouTube and learn pretty much anything but I just can't see myself...

yebyen··on Tech audit of Colonial Pipeline found ‘glaring’ problems
That's outstanding. They avoided the loss! If there was ever a better example of cutting the nose in order to spite the face, I never saw it.

If we've automated ourselves out of the capability to operate for a few days with paper and pen, when the alternative is a nationwide crisis where untold million humans were impacted by resource shortfalls...

How many pennies did we save by shutting down the pipeline, and did we successfully externalize the costs of the worst of the effects of the hack... if so then job well done! LOL

Never change, Colonial Pipeline! Never change a thing.

yebyen··on DarkSide ransomware gang quits after servers, Bitcoin stash seized
Yeah yeah, and if you are the one on that phone call at that time, then you are welcome to call the bluff. Gulp

Putin already denied responsibility and Biden apparently accepted that, so we wouldn't be attacking the Russian government. Wink

yebyen··on DarkSide ransomware gang quits after servers, Bitcoin stash seized
> If a business externalizes the cost, does it matter to them?

I mean, yes? Maybe not before next quarter's revenue statement, but eventually it will have to start to matter?

If your dog goes and craps in the yard every day, you eventually have to clean it up or you will get flies in the yard, and if you have to open the door or leave the house at all then sooner or later you will have flies in the house, it matters, yes. It's really not any more complicated than that.

If you are responsible for dumping toxic waste out the back door of your factory, it's only a matter of time before it's in your drinking water at your house, a couple of miles down the road. Externalizing a problem doesn't really get rid of it, just makes it someone else's problem (for now at least.) Those other people are real people, and they will find you.

yebyen··on DarkSide ransomware gang quits after servers, Bitcoin stash seized
It's easy to say "basically zero chance" when we're armchair quarterbacks and not the ones in the hot seat.

I'm inclined to agree that our cyber-security apparatus is not up to the task, but it's also true that nobody has perfect OpSec, (and I'd guess there are few out there have deeper pockets to track down and make sure the perpetrators regret this, than the combination of US government + oil companies.)

yebyen··on DarkSide ransomware gang quits after servers, Bitcoin stash seized
They paid $5 million, if "it was cheaper for them," that's solid math that ignores some really important stuff though, LOL. What is the externalized cost of this crisis on the entire country? The $5 million dollar ransom is a worse deal if you can convince your board to consider that externality.

The criminal penalties for executives in leadership and board positions (and I'm not saying this is my preferred approach) would certainly go a long way toward changing the calculus of this exchange.

yebyen··on DarkSide ransomware gang quits after servers, Bitcoin stash seized
I imagine it went something like this

"OK, now that you have our attention, and the eyes of the entire international media apparatus are on us, here's how we're going to do this. We're going to send some integer number of million money dollars down this pipe, and you're going to turn that gas pipe back on like you said you would.

Then here's what happens next... we're going to give you an integer number of minutes running head start before the drone strikes start raining down on these 12 sites we've identified as likely candidates for your location, ... now how many millions was it that you were asking for from us again?"

Doesn't really matter how much it was, either, if it has really been seized already in less than 24 hours. Was it enough to convince the boss guy or gal to take the bait and risk revealing themselves? (Probably not, but IMHO that wasn't likely to happen anyway, at least not since the heat started getting turned up on them all.)

yebyen··on DarkSide ransomware gang quits after servers, Bitcoin stash seized
When I heard that this pipeline company started advertising a job opening for CyberSecurity Advisor in the last few days, and heard today the ransom of about $5 million was paid, my first reaction was to say "I bet the salary for that position is a lot less than $5 million, and I bet the budget for that department will be less, too..."
yebyen··on Minio Changes License to AGPL
https://github.com/cncf/foundation/blob/master/allowed-third...

I haven't read the details, or ever seen this policy before (I'm new to both projects) but it was summarized by one of our counterparts at the Linux Foundation here:

https://twitter.com/cra/status/1384859663615864833

Tl;dr: licenses must be approved for use, and the CNCF has this list of allowed licenses, AGPL is not on it. The CNCF is in the business of distributing permissively-licensed software is the short version I guess. I don't understand, I don't work on the legal side, I am a dev and I support end users.

It seems if your Apache 2.0 licensed project needs to modify and distribute as modified an AGPL project, (which for Grafana it seems likely we will need to do at some point, the Linkerd project already has needed to do this if I understood correctly) then you cannot distribute them together, or something about this becomes much more complicated. Chris says they are going to try to work something out, but when a component has made a decision to re-license with a restrictive-copyleft license such as AGPL,

I don't know what there is that can be done to fix it. I hope they come up with something.

Maybe the CNCF adopts AGPL too, (which would mean that then all those "viral-GPL" FUD-spreaders will have been right...) that seems counter-productive if that is the outcome. (So I hope they come up with something else than that!)

Everyone is within their own rights to do whatever they want with the output of their own labor, I just wanted to help dispel the notion that only exploitative companies are affected by this change.

If the "why" rather than the mechanical/legal "what" is what you're after, this is the CNCF position paper on the topic: https://www.cncf.io/blog/2017/02/01/cncf-recommends-aslv2/

> Proponents of copyleft licenses have argued that these licenses prevent companies from exploiting open source projects by building proprietary products on top of them. Instead, we have found that successful projects can help companies’ products be successful and that the resulting profits can be fed back into those projects by having the companies employ many of the key developers, creating a positive feedback loop.

yebyen··on Minio Changes License to AGPL
The entire CNCF apparently has a no AGPL policy. We just found out this week that we cannot update and redistribute Grafana anymore with our CNCF projects, (or if we do we will be on very shaky legal footing.)
yebyen··on We need to know the origin of Covid-19
Yes. We confuse blameless postmortem with sweeping things under the rug and helping everyone involved in the mishap to save face at our own peril, they are not the same idea!
yebyen··on Whistleblower: Ubiquiti Breach “Catastrophic”
The triangle says Confidentiality, Availability, Integrity.

While your concerns are 100% valid, we need to remember too that setting up access in restricted ways and inviting users to understand the protection and remove the correct barriers, or implement the concerns necessary to interact with those for themselves, always runs the risk that some users will find your protections cumbersome and instead find a (totally incorrect) way to baffle them, or otherwise even route around them entirely mooting any efforts to secure a platform.

And every time I hear this played out in conversation, the answer is "that's on them!" But it's clearly a balancing act, it's a trade off; tautologically, when you make the service less accessible then... it is, well, ... made less accessible.

Besides facilitation of the secure access also sales conversion ratios will depend on that accessibility. The crux of your argument stands, the defaults are too open, and we need to do more to ensure that naive users aren't handed a loaded gun to aim at their own feet.

yebyen··on Git email flow vs. GitHub flow
They aren't random at all, they are practically in order. It was probably a footnote reference to the bottom of the source article, with a link to each RFC at the footnote.
yebyen··on Open source Heroku Like Platform on premises
Well, they built Helm which is reasonably successful and carries on their legacy with a large community following. They proved themselves to be among the top experts in the space and got some of the most solid financial backing you can imagine as a result of their efforts; the remaining Deis / Azure folks moved up the value chain, to build AKS, which is also pretty successful and has a large client base.

They built Workflow which was IMHO incredibly popular with those that used it, but, from discussion I had with the folks that moved on leaving Workflow behind, it wasn't "pressure from above," or anything like that; they simply went where the market forces were pulling, and opted to work on what they thought would have the most impact.

Team Hephy has never really had the aspirations to be a business, just to preserve the capability to go on using Workflow on newer generations of K8s. We develop and maintain it because we like using it, we don't need it to change in major ways, and so we aren't really concerned about its commercial viability (which Deis certainly was.)

So, without the SaaS offering, what is the best outcome for a company that develops Workflow? You build up a base of customers who were all too cheap to use Heroku, and once in a while a breaking change from K8s upstream means they depend on your support, but only if they are conscientious about upgrading...

I don't think that means there isn't or wouldn't be any market for Heroku-compatible competitors, not by a longshot. I have no idea if it's a viable market, but I still hear a lot of groaning about how hard it is to use Kubernetes often, and at least twice a year I see someone trying to invent Deis PaaS again. The "Kubernetes On-Ramping" market is still very ripe.

How does it go? "I'm convinced the majority of people managing infrastructure just want a PaaS. The only requirement: it has to be built by them." https://twitter.com/kelseyhightower/status/85193508753294540...

I haven't tried this new one out yet, but I definitely will. Have to scope out the "competition" and see what the grass is like on the other side. I hope it is greener there!

yebyen··on Open source Heroku Like Platform on premises
Not a problem for me, congratulations on the launch!
yebyen··on Open source Heroku Like Platform on premises
Deis Workflow is not abandoned! Just changed names, and teams, (but it's still my grandfather's axe!)

https://web.teamhephy.com

If porter-dev is selling managed hosting platform services, they already have a stronger business model than Deis did (which was, I guess, to get acquired by Microsoft and subsequently never provide any form of managed version of Workflow)

I don't know what Deis Labs business model is/was to be honest but it seems to have been a success, :-) there are worse footsteps to follow in

yebyen··on Open source Heroku Like Platform on premises
Are you aware of Hephy Workflow by any chance? (Neé Deis Workflow)

If so, are you aware of the Porter project which is built on CNAB, (by I believe many of the same folks?)

https://deislabs.io/

I am with Team Hephy, and for the record I think this is super cool what you've done. I just think it's an awfully great coincidence that the name is so close to something else so nearby in the same space!

yebyen··on Is that ship still stuck?
Well the front's not supposed to fall off. The front fell off in this case, but it's very unusual.
yebyen··on Removed gem breaks Rails ActiveStorage
In fact they even sell insurance for this, and companies that want you to use their software can offer indemnity protection with the same effect.

"What if somebody sues me because my use of your software constitutes a violation of their intellectual property rights?" – "Don't worry, we will protect you. Since you pay so much money and are a valued customer of XYZcorp, we don't want you to worry about such things. You'll be covered by our umbrella policy."

This conversation certainly happens, (although it almost certainly wouldn't have happened between any of "Rails" customers and the Rails core team.)

yebyen··on Let's Encrypt's performance is currently degraded due to a DDoS attack
> This is a very morbid thought, but

This is such a well-understood problem in fact that it has a name and Wikipedia entry, called "bus factor". According to:

> The "bus factor" is the minimum number of team members that have to suddenly disappear from a project before the project stalls due to lack of knowledgeable or competent personnel

As for proving that you own a domain, I think the DNS-01 challenge that is used to grant Star-certificates does a pretty good approximation, if you can create and update TXT records in the root zone, you have at least functionally "owned" a domain even if you don't legally own the domain.

yebyen··on Google admits Kubernetes container tech is too complex
To a person who isn't using Erlang or ASP.net, suggesting that we should use either of those language packages and it will solve any of our problems without creating a thousand new ones sounds equally non-starterish to me.

To add a counter-example, I have lots of Ruby experience and I've just joined a Go team. I won't tell them to use Ruby, I will just do it where it makes sense and saves us time. (And then we'll have two problems... enter "limiting blast radius")

Point of my counter-example is, I'm extremely skeptical that all the world's problems can be solved by adopting a new mono-culture, whatever it is. There are 100% always gonna be some problems that are better solved in a different language. PHP is the best way to run Wordpress, for example (ok, so it's the only way to run Wordpress, but you get the idea... "Wordpress is the best way to..."), but I've been in high-functioning IT organizations that won't touch that with a ten foot pole, because "it's another language to support, and PHP is icky."

We also got rid of a perfectly fine Wiki in favor of centralized Knowledge-base software for similar reason. "Better to just have one KB. We don't need to be hosting another thing." So the chances of moving everything over to BEAM VM are next to nil, unless you are a product-focused company with just one product, or happen to have an absolute champion leading the effort to migrate all the things. For all the other things, you need to have a consistent answer too.

No tool is one-size-fits-all. Where Kubernetes shines most is under any environment that isn't running a single monolith or building a software monoculture and/or can't manage that for whatever reason (because those are all basic use cases that are frankly easy enough to manage without adding on top the additional complexity of Kubernetes; don't need it, don't use it!) IMHO, diversity in infrastructure is a plus though, and Kubernetes is a technology that it turns out enables this.

yebyen··on Epic will pay off class-action loot-box settlement with in-game currency
...but I'm sure the lawyers who brought the case aren't being paid with in-game currency, amiright?
yebyen··on Average Rent in San Francisco Has Dropped $1k This Year
Same in Indiana. But the proposed idea was to tax more heavily all unoccupied properties, not merely "all properties other than one's primary residence."
yebyen··on CentOS 8 vs. Amazon Linux 2
How generous of you. Since we were comparing it to something that has only one year of support to come, I thought it was fair to count any more than that as "years"

Amazon has a historically good record and reputation for supporting things longer rather than axing them early. Look at what they've done with Amazon Linux 1, which was supposed to be EOL this summer: (they made a judgement call that people still needed the support, and extended security updates for at least another year.)

Regardless of the dates that are on paper, I think you're the one that's being pedantic now. What's the last time that Amazon ended support for anything that was in GA? It's absolutely fair to say years to come, when making this comparison.

yebyen··on After raid on Covid whistleblower, it’s revealed password was on FL DoH website
That depends on which news you watch. I'm with you, I don't want to acknowledge that there is any chance. But I've just spent the week with my folks who leave Fox News on in the background for most of the day, and I'd say now my sentiment is much closer to "nevertheless, they persisted" – people really buy into this stuff.

And it's not a decisive minority of people either, not in either of the places I've lived in my lifetime. Certainly the vote reflects that it was a minority, nationwide, but "have you ever actually seen a million dollars"?

Even of those that know it shouldn't be possible, regardless there are many who are still hopeful it gets overturned. (Still, thank you for saying what I'm thinking.)

yebyen··on CentOS 8 vs. Amazon Linux 2
Here's another version, that pre-dates the official docs (when they announced that Amazon Linux 2 was now possible to install on your own metal, it wasn't exactly well-documented.)

The Amazon docs probably cover the same stuff now.

https://superuser.com/questions/1048091/can-i-install-ec2-am...

← PreviousPage 10 of 34Next →