3,011 karma · joined December 10, 2021
I do keep an OpenRouter account topped up for things that Ollama doesn’t have. 99% of my usage there is embeddings, the other 1% is wanting to test some new model Ollama doesn’t have.
I wrote an agent launcher sort of bash script. Pass in the command to start the agent, the script checks if there’s a Jinja file in a special directory matching that name, and builds it to AGENTS.md. Then it launches the agent.
I was trying to use it as a sort of janky RAG. I had a bunch of snippets (one for DB architecture, one for how load balancing works, etc), and my Jinja files were mostly a list of snippets to pull in. Voila, a bunch of agents that share little pieces of info but have a single source of truth.
I never got a ton of value tbh, it was very good at just grepping the snippets.
That feels like a bog-standard search feature on a marketplace. I can only assume the omission is intentional.
Eg I haven’t finished reading it, but it cites Heller on the first page and enshrines that AI is/can be used for self defense. That feels like a huge can of worms for the court, because now every 2A case in our history is relevant to any AI bill.
I also don’t know that it will accomplish what the author intends. Even if we hypothetically establish that AI use is covered by the first and second amendments, the government can still regulate it if they can pass strict scrutiny, ie if they have a worrisome enough complaint.
What the author wants, at the level of certainty they want it, would require a constitutional amendment and dismantling the strict scrutiny system. No idea what the impact of that would be, but my wild guess is “probably bad”.
I went with Docker because history has taught me that new isolation strategies _will_ have escape bugs at some point, and I’m distrustful that the LLM can’t find one if it wants to.
> Probably the biggest (and least hardened) feature that I spent time on in mine was trying to figure out how to allow arbitrary GUI apps so that I could run agents in it via Zed.
I actually have code for this if you want to use/fork/borrow it. TLDR, mine pretends to be an ACP agent so you run Zed on the host, but under the hood that binary is just creating a Docker container with your image and agent, copying files, etc, and then proxying ACP messages via web socket back and forth. Except for the built-in ACP read/write file and shell endpoints. Those get executed inside the container by the proxy by default, though there’s a config option to pass either or both through to the host.
Repo is https://github.com/SethCurry/abyss and the code you’d want would be in ‘internal/websockets/wsacp’.
It does wrap them in protobuf and there’s some router-like stuff so you can add your own non-ACP messages between the two ends of the proxy.
main also has experimental wasm plugin support in the proxies so you can block prompts/tool calls/whatever in an agent-independent way, or add RAG that works for every agent in the world, or whatever.
It works well, though there is some leakiness around paths. I opted to make it place/mount files at the same path as on the host so paths are the same (as opposed to manipulating the ACP messages to modify paths on the fly, that felt messy and buggy).
Configurable networking is on my list for the future, but I haven't decided whether to start with IP-level firewalls or if it's better to start with a proxy and firewall rules to force traffic to it. IP firewalls suck for APIs that might have semi-dynamic IPs.
Those two things really aren't comparably difficult. Eg in the dumbest case for pads, you could make the pads slightly smaller than axle diameter (to prevent dead shorts) and have the car randomly jostle around the space until it detects current flow. I've seen some amateur systems that work vaguely like this. Some signal that charging is nearby, and then a semi-random search to find exposed contacts.
That's not going to cut it on a recessed outlet. Your odds of hitting it randomly are tiny, and it will probably the damage the port with enough repetitions of ramming it into the floor.
> Planes refueling mid flight is hardly an apt metaphor given the charging station isn’t moving at all and the car can move as slowly as needed.
Change your reference point to the plane carrying the fuel and they get an awful lot closer. The planes are moving a lot relative to the ground, they're moving very little relative to each other. I'm not sure if you've watched them doing it, but relative to each other it's actually a very slow process.
You can kind of just leave them exposed and live.
None of that applies to a 220V, 50A (or whatever amperage) charging. That can absolutely kill somebody; jet fuel may not melt steel beams, but 11,000W sure will.
You’d end up needing some kind of recess probably, and then getting cars to connect to it like planes refueling mid-flight is a nightmare. You could try some sort of USB-C like signaling before power is sent, but people are never going to enjoy “exposed 220V contacts that may or may not be live depending on if things malfunctioned”.
It only really makes sense to tarball repos into cold storage on S3 or Glacier, but short of GitLab cloud I suspect there aren’t enough repos cold enough to be worth the dev costs.
S3 that doesn’t have wild costs for git objects sounds like it’s just NFS
Postgres and Nginx make sense in system programming languages; they’re extremely performance sensitive and that granular level of control offers them features. Interpreters are sort of the same, they interact with the OS a ton, it makes sense to work in the same language as the OS.
I do generally agree for the app tier of a web app. I wouldn’t build a CMS in Rust, but I also wouldn’t build a reverse proxy in Python.
The slow closing prevents water hammer, at least in water systems. In air systems it might just prevent a blast of back pressure blowing all the dust out of your vents onto your carpet.
Still a bad bug that lingered entirely too long.
> when a file is changed asynchronously, which for me, most happens due to a pull or LLM edits!
If you’re using Pi (I am), that’s partially because Pi doesn’t have “real” ACP and violates the spec as a result. File read/writes are supposed to happen via ACP RPC calls from the agent to the client, so Zed would see them and know to refresh. Pi wasn’t built for ACP so it embeds its own read/write tools and ignores the ACP ones, and Zed never gets the “refresh this file” signal it expects.
The research seems to agree with you, though. The paper calls out that for “bash capable” models, adding tools to do things bash can already do doesn’t improve performance.
Vaguely the same result as RAG. Unless you’re in specific domains, you won’t beat handing the agent a shell and grep.
I would be shocked if you could publish an iOS app without Apple being able to tell the government who you are. Less because Apple cares and more because Apple requires you to pay, which is very hard to do anonymously for something like this (I’d bet the options they offer are effectively “credit card only”).
We’re culturally very far from even being able to attempt that.
I have more faith in companies with a more targeted approach. Eg gzip does fine, but video codecs beat compressing raw video by a ton.
> As an aside, I wonder how many days are we away from Codex or Claude
That sounds like SourceGraph but twice as expensive, although it does have “AI” so probably lol
Eg absurd idea, but you could write something that minifies a codebase (by token, rather than byte) and then translates edits back into the expanded code. Probably an insane use of fuse lol. Partially minifying on each tool call sounds like a huge pain with a lot of state to track.
There’s also a lot of common situations where humans prefer solutions that take more tokens because it’s easier for us to read (eg for loop vs map vs list comprehension), which may have some gains.
I strongly suspect there is some form of token compression that works, but I don’t think it will be as simple as “pipe arbitrary text with no context into this tool”.
Jetbrains feels like a place this might come from. “Take this code, parse it to an AST, find the fewest token representation of it” feels like something they’d do, or maybe Astral (specifically in Python land, type checkers feel sort of adjacent as well).
Eg if I ask Llama 3 to find me the best headphones, I think we can agree its response isn’t an ad.
If Claude responds with a banner that Sennheiser designed and paid Claude to show, that’s an ad.
They’ll live in the probabilistic grey area between those. If Sennheiser pays Anthropic to add “Sennheiser makes really good headphones” to the system prompt, is it an ad if Claude recommends Sennheiser headphones? What if they pay to ensure their site is included in the web search tool results for any queries about headphones?
It’s sort of akin to paying Google to rank higher in searches if that were a thing they did.
I do think they should have to declare those, but they’re not traditional advertising to me.
Mine pretends to be a local ACP agent, but it’s actually managing a Docker container and proxying the ACP connection into the container over websocket. You can specify a bunch of utility stuff in the YAML definition like directories to bind-mount, directories to copy from the host, scripts to run when the container starts, etc. You can also toggle whether ACP-native tools like read/write file and shells execute on your host or in the container (in container by default).
Works well, I forget whether I’m on that or pi-ACP directly until it spits out a time in the wrong timezone or I forget that it can’t check my DNS settings or something.
Really cuts down on the damage it can do. Mine is basically down to “it can delete my ~/.pi and the repo it’s working on” and that’s about it unless it can escape the container.
Clouds are appealing (note, not technically better but more appealing) in the enterprise for numerous reasons. We already have a contract, I don’t have to spend a month with procurement. Their support is “respected” so “I asked AWS and they told me to pound sand” is an acceptable response to a lot of requests. No one ever got fired for picking AWS. The list goes on.
None of those friction points apply to hobby hosting, really. IAM is annoying because the alternative isn’t “fill out 7 forms and host 12 meetings to get a new vendor in”.
You also get less of the benefits. Low volume SQS can be trivially replaced, but if you use enough that you’re debating making a whole team to manage Kafka then paying the cloud tax can look appealing because of the predictability (your Kafka team might fail, SQS probably handles larger workloads currently).
At some scale, not having to wait 12 months for the message queuing team to support TLS for some regulation becomes worth a cloud tax.
My theory is mostly that "gambling" seems like it inherently means "buying stocks based on something other than their concrete value". More gambling means stocks drift further from their "true value", which means a higher payout for correcting them back to what their price should actually be.
The whole thing does get very fuzzy because of the "market can stay irrational longer than you can stay solvent" aspect. It's not enough to know what the correct price is, you have to know when other people will realize that as well, or else convince people that your price is "correct".
At a sufficiently small company, you can just trust people will generally do the right thing. I ask X to accomplish some compliance goal, I just trust that they'll do it right.
As the company scales, you actually know a smaller and smaller portion of the people you work with. You no longer just trust that they'll do the right thing, because you don't know them. You fix the lack of trust by adding processes that ensure the "right thing" happens, but each of those processes is a friction point. Eventually you have so many of those processes that they become the majority of the effort for a project.
That gets exacerbated by the usual turnover. People pissed off by all the processes they have to comply with will leave, and people who love having their own fiefdom will stay and double down.
That all ends in the evergreen stupidity of "spend $1,800 to have 12 people who make $150/hr argue about whether this app really needs a $5/month Aurora database".
That's a personal pet peeve, but is emblematic of the issue imo. Trust has broken down so severely that the company is willing to spend more than the lifetime cost of compute for the app to vet whether it's "necessary".
A similar vein is the "platform team" where all they do is take an open source project and wrap it in a custom DSL, so you get all the complexity and none of the searchability of the original project, plus the features are almost always a subset. I can't tell you how many times I've looked for solutions to a problem, found a snippet that will fix it, and then had to reverse engineer how to make a stupid DSL output that config.
Theoretically, more gamblers should mean better price discovery because the payouts for correctly taking the opposing side of the trade are higher.
The market solution would be that the gambling will eventually solve itself. They’ll either learn enough to be trading on knowledge rather than vibes, fueling price discovery, or they’ll exit the market when they’ve lost too much or everything.
My sticking point is that a lot of brokers offer leverage to people they really shouldn’t. I could have sworn you had to be a qualified investor to get leverage, but if that isn’t law it should be. Show the brokerage your certification, or a pile of cash large enough to convince them you can afford to lose the whole thing.
Doubling your meds on a bad pain day can put you way beyond the safe limits. People think it’s safe because basically every other OTC has a huge therapeutic band, and double dosing is not recommended but not really dangerous.
CDC estimates it at 56,000 ER visits a year, 26,000 hospitalizations, 458 deaths, about a hundred unintentional deaths per year. As a point of reference, it’s about 3 accidental overdose deaths per child that dies from being locked in a hot car.
They filter their results through their AI, though, so you get a sort of meta-summary of the top few results. It did well with geopolitical news stuff, but I've not tried a hard science sort of query.
You do pay for the tokens, but in theory on a smaller model each token is cheaper.
They run a proxy for LLM providers, which inevitably creates a marketplace. They’re amazon.com for tokens.
The flip side that others are pointing out is that Amazon is sticky because there’s a sprawling, physical logistics apparatus underneath. You can’t replicate amazon.com’s business without billions of dollars and a decade of building warehouses.
I don’t see where OpenRouter has that. As far as software goes, theirs doesn’t seem particularly tricky. LiteLLM does vaguely similar things, at least for organizations where managing accounts isn’t absurd overhead.
They were first, and there’s money to be made there, but what’s stopping someone else from building LibreRouter that charges a 3% or 4% fee instead of 5%? That’s where I get dubious of their valuation.
I still remember GDPR coming out and I read til my head hurt, decided the lawyers would have to figure it out. Then legal shows up and says they don’t really know either, we’re going to do X and hope someone else gets sued first so they can see what the court thinks the spirit of the law is.
Similar issues happened with opiates. They get overprescribed, DEA cracks down and says they’ll publish prescribing guidelines, then never does so everyone is left running on vague “as much as is necessary and justifiable” type verbiage. Can you keep raising levels to keep pace with a rising tolerance? Does that only apply to terminal patients where addiction is less of a worry? What’s the bar for them to be justifiable? Discomfort, pain, debilitating pain?
Both cases end in people who are genuinely trying to comply with the law being unsure of what compliance even is.
And the courts generally won’t take a hypothetical “is it a crime or not under this law if I did X?”. You have to just do it and accept it for the Schrodingers Cat it is. It’s both illegal and legal until the judiciary opens the box and decides it was one or the other for sure.
The alternative is complying with the spirit of the law, which is an eternal guessing game. Who knows whether it's legal or not, we have to wait for the Supreme Court to decide what Congress _actually_ meant. It implies that the law means something beyond what anybody bothered to actually write down, and nobody has any idea what that is until the Judiciary interprets it into "actual law".
--- Start Quote
(2) intentionally accesses a computer without authorization or exceeds authorized access, and thereby obtains—
(A) information contained in a financial record of a financial institution, or of a card issuer as defined in section 1602 (n) [1] of title 15, or contained in a file of a consumer reporting agency on a consumer, as such terms are defined in the Fair Credit Reporting Act (15 U.S.C. 1681 et seq.);
(B) information from any department or agency of the United States; or
(C) information from any protected computer;
OpenAI's nonchalance is forced. If they are found to be even partially responsible for the CFAA violation then they have an _enormous_ problem. They _need_ for whoever prompted the LLM to be responsible, because the alternative is having to have an efficacious process for identifying hacking attempts. They don't have that (and no one does).
> The community here at the same time cheers for fully releasing the open weight models without any hacking limits and at the same time criticizes a proper response.
No, at least I personally criticize because closed weight models incur a rent. I can only make sure their model can't find vulnerabilities in my software if I pay them to check. I can pay basically whoever to do the same thing on open weight models.
It creates a fundamental conflict of interest. OpenAI/Anthropic/al _should_ stop bad actors, but it fuels their sales if there are X bad actors and as a result X*10 (or 100, or 1,000) good actors have to burn tokens checking if those bad actors will actually find a vulnerability. You can see their line-toeing where they talk about how safe it is, but also how dangerous it is to have code you _aren't_ auditing with their LLM.
As a result, I do not trust them because their goals are not aligned with mine. The open weights might not filter out hackers, but I'm also free to check the results on my own hardware, or OpenRouters', or whoever else. The line between "my LLM can find vulnerabilities" and "you have to pay me" is a lot more blurry. It's a lot easier to claim an LLM can find vulnerabilities than it is to be the cheapest inference provider. Anyone can bullshit on Twitter about how scary a vulnerability is (see CVE scoring), a lot fewer people can build the most cost-efficient inference in the world. They would rather be buzz-worthy than competent or open.
I find their position morally abhorrent. It's a mob-style shakedown. "Pay us to check your software or we're not responsible for what happens" is nothing short of a shake down. They need to either fix their systems for detecting hacks or offer some way to immunize against the hacks their software would propose, otherwise they're just as culpable as anyone selling a 0-day.