I'm a computer science graduate with experience in embedded systems seeking to get a career in software engineering. In my free time I love learning all kinds of things, working on complex projects in various programming languages such as C++ and Rust, etc.
I'm not really sure what your getting at. The entire reason that masked LEs are opposed is because (1) it makes it impossible to hold them to account and (2) it makes it trivial to impersonate any LE (ICE hasn't exactly tried all that hard to make special gear for themselves, just buying what's on EBay and similar, so literally anyone could just buy the same gear they're using and impersonate an ICE officer, and since ICE doesn't like identifying themselves, you would have no immediate way of distinguishing between a random citizen who wants to ruff you up or murder you versus a LEO)
I mean, if it uses the exact same identifiers and maybe even comments, then sure, that's pretty good evidence. However, 99.99999999999 percent of the time this will not be the case. As in, the super-super-super-majority of LLM-generated code is customized for the specific use case it's being employed in. As such, claiming any kind of infringement, let alone proving it, will be insanely difficult.
> The prospect of being sued because the LLM outputted training code you're not allowed to use, or to relicense, is exciting?
The problem with this is that if someone sues you for this they have to somehow prove that you actually infringed. They can't just point to the code and say "See, mah, infringement!" The legal system does recognize that it is very possible for two people, completely independently, to arrive at a nearly identical solution.
The biggest problem with having an LLM do this, from what I've observed of at least frontier models, is that the way they are trained/forced to behave makes them do one of two things, neither of which is educational or helpful:
1. They ask you a "hard question" but then immediately give you some kind of "hint" which makes it obvious what the "answer" is (or, even worse, they just tell you what it is).
2. They get pedantic (and I mean maximum emphasis on the word), to the point that they will nitpick the exact, precise wording you use, even if the context makes it blatantly obvious what you meant. GPT is currently the biggest offender in this area (it is ridiculously pedantic even though I haven't told it to be so).
And yes, in case anyone is wondering, I do actually read the output of tools like Claude and I have tried, repeatedly, to get them to do something like this, and one of the two aforementioned things always happens for some reason. I'm not sure if this is a problem with how they were trained, an issue with their system instructions, or what.
Am I the only one who wonders if there is a way to get the kind of performance we want out of non-soldered components/SOC designs? Like from what I've heard, it's a "physics says no" problem, but I'm not sure if that's actually true (not being an expert in this technology) or if it's just the excuse many companies use to justify soldering everything to make serviceability impossible. And it would be really neat if this is actually being looked into by labs right now (possibly with AI's help), since areas like that are where I think (non) LLM AIs might be very useful. Same thing for medicine, materials, etc.
I think it's more the image formats PDF supports? I could be wrong, but there was that forcedentry vulnerability (which was insanely cool, even if really bad)
I'm pretty sure not even the safe languages would help you here. PDF allows you to pretty much run arbitrary code since it supports javascript, images, and a bunch of other things I'm forgetting, and you can't not implement javascript support or otherwise be selective about what you support with it because then all the end-users will complain when they get that one PDF that they need to fill out but it needs JS because... Nobody knows.
I mean maybe but... There's nothing forcing you to use Claude or what have you to play around is there? I mean an employer might force you to use it but you don't have to in your free time. That's one way I comfort myself lol
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
Also important for physical modelling and geometric acoustics -- you need a geometric mesh (made of triangles) to work with. Which parts of could be sent to the GPU for processing as well.
> ... and by doing exactly what you're complaining about:
How exactly does requiring people to identify themselves actually protect anyone or slow them down? Their identity system doesn't even work reliably (it's just your classic "verify your ID" scheme), and it failed to verify my ID no matter how I did it, so in the end I gave up. Even assuming it was reliable, I don't see how it stops bad actors from doing it, since we all know how easy it is for most sufficiently resourceful actors to forge IDs, and I suspect these digital ID verification services have all kinds of vulnerabilities that would make them easier to bypass compared to the tools the state has to prove an ID is legit. For the higher tiers of CVP there may be more requirements, but do you seriously think these companies aren't just using AI to verify those tiers too?
This doesn't work. Even if you use agents that are 3 months behind, they will still find the vulnerabilities Mythos or Claude Opus would find, it would just take them a bit longer to do it. As such there is little reason for these cybersecurity filters to exist. They don't actually prevent people from finding vulnerabilities; it just slows people in finding them because they have to use less capable models, and by extension that makes cybersecurity postures worse. You can't ever get rid of the arms race between defender and attacker. Anthropic acting all nanny-ish doesn't at all help the situation -- if anything, I would argue, it just makes it significantly worse. Especially when the filters will have significant false positives (e.g., they will trigger for local things that are completely unrelated to hacking at all, because somehow that's super duper dangerous).
I would believe Anthropic about how Claude reduces attack surfaces if they weren't simultaneously nannying everyone about how they supposedly don't want their models used for cybersecurity because... Reasons. And if you do want that access, you have to verify your identity, and do a bunch of other things, all of which goes against "reducing" the "attack surface". If they really gave a damn about cybersecurity, this cybersecurity filter wouldn't exist, period.
As one example I can think of, there are a ton of devices that only work on Windows (or Windows and MacOS). The audio interface I have is one example: its "official drivers" are Windows and MacOS only. I know that someone is trying to make Linux drivers for it, using LLMs, because if there's one thing LLMs are very good at, it's RE and figuring out how hardware works. One of my other friends made an accessible equivalent to the Razer app for his Razer headphones because the official app is complete shit.
I would love to use Lazarus, but the UI toolkit it uses has some very bad accessibility bugs that make it practically unusable with my screen reader. FreePascal's love of syntax diagrams is also a turn-off for me (I've always preferred things like EBNF or similar for grammar notation).
The (only) part of this that I find rather ironic (and kind of funny) is that these idiotic companies never seem to learn a thing. Nearly every single time they take something like this down, it just draws even more attention to it (the Streisand effect).
I've had a lot of bad experiences with current chatgpt (mainly, it tends to just get sidetracked and hyper-fixated on literally anything, so if I give it a rodmap of tasks to do it will fixate on the first one and keep adding caveats as to why something isn't complete) and a lot more success with Claude (not) doing that. But I don't use Codex, but chat chatgpt (I don't have the max plans so Codex gobbles up usage limits in like 20 minutes even on the simplest of tasks) so Idk, maybe that's why?
To counter the bad though: right now Claude and I have been doing some very interesting work on acoustics and advancing steam audio. I've been researching this for several years trying on and off again to make my hopes a reality but could never take all the math and make something coherent out of it by hand, and Claude has helped with that (well, and Steam Audio). The trouble mainly comes from being blind and trying to build geometry from code and not being able to mentally visualize the geometry in question (which is also why I struggle with physics engines). It's been really fun just experimenting with this and seeing how good it gets. Who knows, I might make it a library that I publish someday.
Am I the only one who noticed just how entitled this Žilka person is? If they are so good at maintenance maybe they should take it over and learn the hard way that things like this are not at all remotely simple to pull off and that maintainers are (usually) unpaid volunteers who have their own lives (and in some cases, they maintain many other packages).
Edit: I should note that it definitely looks like entitlement based on how the article presents it, and doesn't at all seem like this person is taking into consideration all the maintenance policies/rules/whatnot.
Yep, this control server thingy I was trying to get it to implement was entirely local. In theory it could be used over the network, or tailscale... But I have no idea why anyone would do that, and I may consider just removing the address part entirely since the entire point is to debug the testbed, verify measurements and physical modelling, etc., and not to be some secure control endpoint or something.
Agreed. I really don't see how these filters prevent anything. Claude stopped me the other day because... Get ready for it... I told it to implement a control mechanism in a testbed tool I'm using to allow it to control the testbed to give it real-time feedback so it could debug problems in real-time. The "cyber" filter really didn't like that it was implementing a TCP/HTTPS-like server.
Like guys, come on. Really? All the "filter" has to do is read my transcript to figure out that what I'm doing is not actually cybersecurity adjacent at all. I had to go have GPT implement the server because Claude refused. Truly, Claude really did prevent something terrible from happening! /s
I think the contributing factor is people who make software but have no idea what they're doing. They want to make photoshop, but they have no knowledge of photo manipulation or how photoshop works, or any software engineering/development practices. And then we wonder why slop still exists. I am of the opinion that it will always exist: people just have no idea what they want to make let alone how they want to work. "Reimplement photoshop" is a nice end-goal, but you have to actually know to a decent degree how photoshop actually works! Not to mention the reverse engineering knowledge you need
All these "accidents" with agents just makes me want to use them less, not more. Then again, I don't use agents because they consume tokens at an absurd rate, so...
I might be missing something but... How exactly is this better than telling Claude for example to "install and set up a full RE environment including Ghidra" on my local system and get to work? Like what does this do that my current RE methodology doesn't?
I mean, being pedantic a little, we don't actually know if c is constant, since measuring c is rather difficult. If c is not in fact constant in some medium or environment, then a huge number of things get very weird very fast.
So, yes, we could've measured c wrong. We just would have no idea if we did.
Source: Veritasium did a very fascinating video explaining this problem.
This doesn't actually change what I said. The imagined future is that people just use agents for shopping. Assuming the capitalistic economy we have constructed remains (and I've little reason to think it will vanish anytime soon), poor people, or people with low incomes, are certainly not going to be using high-end or even midrange models/agent providers. They will go for whatever sounds the best and which is the cheapest, regardless of it's quality or how gullible it is. So attempts like tfa to figure this problem out now are very useful indeed.
I think it's more that you don't understand just how poor many people are. Most people will go for cheaper models (assuming they even know what a model is to begin with) just because the more expensive ones are ones they literally can't afford because all of their money is spent on bills and necessities. Supposedly the cost of using models will just keep falling forever but I have my doubts (since that's not exactly how an economy actually works, nothing is truly free unless the government is paying for the entire thing). If we take the mindset of the technical illiterate average person, they will go for whatever sounds the best (and whatever costs the least) because gas is $5-$7 per gallon, the price of everything continues to rise, and on and on. I think it's probably smart not to be so dismissive of the struggles of the super-majority of people who are barely able to survive because money is in fact an obstacle, and the only way you'd get them to use a mode like GPT-6 Sol or Astra is by making it pretty much cost nothing, and that isn't how the real world works: someone is always paying for it, and the question is who.