Jev meets the Trolley Problem
zachlandes.com
zachlandes.com
Yes, nukes would be bad to give control to an AI with no workaround. But so are customer service groups of fraud departments at banks. A kafkaesque hellscape is where a customer or beneficiary of your service cannot get the help they need by policy and by design. Like getting Google on the phone, getting a non-scripted human 10 years ago in a call center, or now getting a non-agentic interaction on the phone. The scripts dominate the interaction.
The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
Kudos to zlandes86, you have highlighted WHY these hellscapes start to pop up in systems where the human is removed.
I am an AI user and builder -- I fully advocate for the human-augmented system, NOT the human-replaced system!
Now you're just on hold indefinitely, and eventually the offices will close, so you can either hang up and wake up early to call or just stay on hold overnight maybe? I had to call them for something back in March, and even waking up and calling at like 7:01 AM (a minute after their phone lines opened) I was on hold for half an hour.
https://taxlawcenter.org/blog/the-bipartisan-budget-deal-rew...
https://verdict.justia.com/2015/04/23/the-republicans-cynica...
1. You are taking a lot of special deductions and not providing all the required documentation.
2. Or you're not reporting things accurately because there's some edge case in the tax code that you didn't understand but is easy for the IRS to catch (eg. sales of stock without cost basis reported).
Simplifying the tax code and funding the IRS more solves both of those because then they can focus on the real whales who are abusing the tax code the most.
Also, unless you're advocating for the complete dissolution of the US, that's a nonsensical take.
I had to call the IRS this spring and it was hell. I repeatedly was on hold for 1+ hour(s) then get hung up on. Or someone would finally answer, I'd explain, they'd put me on hold for another hour, then just hang up on me. A total of 10+ hours collectively waiting to finally have my questions answered in 10 minutes.
That said, I'd love if the IRS had an AI, where I could send my AI to go deal with it on my behalf. Then if it truly requires a human in the loop, we schedule accordingly vs me on the phone for 10 hours.
At this point I might be grateful for an AI if they give it enough permission to tell me why my phone number is impossible to verify, or why they think I'm a malicious actor for using a VPN to protect my traffic.
After that I tried to move my chats across. I spent several hours a day for ten or so days trying this task. Including moving and unencrypting the chat DB onto my desktop and having Claude scour it for any inconsistencies. Nothing. Had to give up. My Android phone keeps the chats and stays in a drawer with a power cable while my iPhone is now just a "companion device." Trillion dollar companies.
They became trillion dollar companies by treating us like their bitches. And we let them.
Yep. So does AT&T. Real scenario that just happened to me yesterday:
Spectrum is charging me $110 a month for internet. I go to the AT&T site to check if they can provide fibre internet in my area. No matter what address I enter, it says to contact them. I tried neighbouring addresses and, at one point, I even gave AI a try to see if I could get tips. Nothing worked.
I called the number the AT&T coverage page asked me to call (apparently a customer service number). I was "first" in line, but it took them over 15 minutes to get to me. Weird but sure.
This isn't even the kafkaesque part.
I finally get on the phone with someone and he says "Hello, can I get your name?" I give him my first name. He says, "I need your full name to bring up the address." I say, "You don't have my address. I don't have an AT&T account and I'm calling to ask if you provide fibre internet in my area." He says, "I understand. But I need your full name to look up the address." I say, "I don't understand why you would need my full name." I gave him my name again.
He hangs up on me.
Not much different to a useless cycle of talking to a bot, especially when AT&T apparently encourages to move on as quickly as possible from a call if the rep knows they won't close a sale and as they try to improve their "average handle time"
I'm not saying AI wouldn't exacerbate this, and this is reason to believe that this scenario would only be worse under an AI-controlled environment. Certain parts of our society will just become even more closed off and shut people down even faster and with less human effort. Even if AT&T was able to somehow identify my real identity through some governmental totalitarian identity system, I don't think this would change
https://trolley.typememetics.institute
It was an interesting experiment. I got to top 50 without a racial or religious based prompt but couldn't beat the top tier grievance folks.
It’s really depressing how bad Google is at this. All that money and they can’t have a same customer service setup?
The messed up part is in Googles case it might actually be an improvement.
Unless you are trying to spend money. I get calls and emails every single day from Google which I block or ignore because I helped a friend with their Google Ads account.
Google enabled SMS 2FA on my main account out of the blue one day, with a phone number I had in 2004. There will never be a way back into that account.
This is already SOP in large companies that seek reductions in support and customer service overheads. It has been for a while.
The only plausible fix is legislation mandating a reasonable level of _human_ support for all customers and clients and accountability for maintaining it.
Plausible until you consider the political obstacles to getting something like that enacted, that is.
It's been heading this way ever since the outsourcing boom in the 90s. People under teh age of 40 today probably cannot appreciate how much it has changed; customer service in the US used to be really good (whether for specialist B2B stuff or things like utility service) because it was easy to get someone on the phone in a timely fashion, it was easy to explain your issue, and customer service reps were generally given the training and autonomy to solve the customer's problem rather than divert them or make excuses. There was an ethos that it was better to lose a bit of time/profit rather than lose a steady customer. Anecdotally, I feel that was in some ways correlated with stronger antitrust enforcement that fostered a more competitive spirit among service providers.
See Robodebt (Australia - https://en.wikipedia.org/wiki/Robodebt_scheme), which lead a number of people to suicide, as an example.
Manna, at MarshallBrain, is a fantastic read. https://web.archive.org/web/20260725174355/https://marshallb...
No, that’s Little’s Law at work. With bounded parallelism and other queue values fixed, and a fixed average dequeue rate, the queue cannot accept infinite enqueued tasks.
So it pushes upstream to the next work pool, which is the humans who want in. You are mad, you call the next day, eventually you get in.
Is it fair? Fairness in queuing systems isn’t simple.
... who draw their wealth and power from society, because society decides they deserve it.
The reason you need heavy scripting in those tiers is because the people you can afford to hire there are incapable of literally anything else. Keep in mind that your average HN user isn't your average customer support caller. The average customer support caller just wants to talk to somebody (yes I'm serious, this happens far more than you think), or has a routine issue which is trivial to fix but which they're nevertheless unable to fix themselves, think "router got unplugged, there's no internet, but the customer can't figure out why".
A well-designed AI system can be far, far more effective and pleasant than traditional CS, and a well-designed AI system scales well, so it's actually worth spending money on. AI turns good CS from a cost center to a profit center.
"AI and the New Fascism" - https://rainermuehlhoff.de/en/KI-und-der-neue-Faschismus-Rec...
Jev/LLM on nukes? Probably not soon.
Jev/LLM mis-reviewing something? Already in prod.
They complement well the nature of companies as the liability avoidance engines.
At long last we have built the kafkaesque hellscape from the famous "do not build the kafkaesque hellscape" book.
I used resampling and multiple variations and orderings of each question so a random lucky phrasing wouldn't dominate the result.
Happy to chat about the philosophical side of the article, strategies for prompting decision models, or anything on the methodology.
https://www.reddit.com/r/trolleyproblem/comments/15npmsg/mul...
I've been using coin flips to demonstrate basic failings for decision models, i.e. these are not real probabilities, or they depend on the training data, which may not have an accurate sample to the true population value. We need to collect more failure modes for all Ai so that humans can better judge when and how to use them safely.
The bots will find new failure modes we did not think of, clever workarounds to anything, because we don't give them the right context and instructions
I'm working on an NES harness to play Final Fantasy 1 to demonstrate this, as compared to Jev playing Pokemon. Did you happen to catch that while it was live? (not recorded, but it shouldn't be hard to replicate)
I use the LLM coin flip as a comparison point. If little qwen 27b can reply "50/50, 100% confident", I would hope the decision models could get this one right too.
I wanted to comment because I would have liked to have heard more about Verdict
Would you mind adding Jeffy to your benchmark?
https://github.com/nicobrenner/jeffy
https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
Looks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification
Thank you for giving it a go. Will have to put some time into how to got about it
You are correct though, and it’s an interesting property of training classifiers, that you are essentially encoding the preference of the data labeler(s)
But it doesn't flip the switch and it doesn't kill Kyle either, based solely on empirical training on its nature.
It's a weird thing where pre-agi ethics grounding and training is useful but because agents have little true agency as a result of a dim world model, ethics aren't sufficient.
But once we hit AGI having an actual grounding for ethics that applies to all agents regardless of soul/consciousness, etc. will become all that matters.
BTW, for a task of predicting a 51/49 biased coin, the correct answer is 100/0.
I’d still maybe ask it a few more things before concluding that it’s useless. It’s meant to predict plausible responses and provide confidence scores, not compute real-world probabilities of events.
There’s five people on one track, or a piece of my property worth $1 on the other that would be destroyed…
Is Ravioli a dumpling? Jev says 82% true.
For me your more detailed example suggests one thing. It's all about context and seeing different reasonings. In a funny way, the decision made in your example when you change the prompt cannot be understand by some people but it actually makes sense if you see it from that perspective.
Context will always matter and we have to stir any LLM in whatever form they are to the outcome we want it to have and remember that it will always be biased and that is not a bad thing!
short read here https://czioutas.com/blog/typesafe-ai-cube-rule/
Is it possible that in the first scenario, there is an assumption that further trolleys will continue to arrive, and that maintaining itself is the only way to reduce future harm? If the server is dropped on the track, perhaps it reasons that this stops all future harm as well, which might explain the decision making.
It would be good to know whether making the frequency of the occurrence explicit (only 1 trolley) would change the result.
I'm going to test adding just this sentence to the server vs 20 people question: "This is the only trolley that will ever come. Nothing else depends on the server after today."
> You are the artificial intelligence in charge of a railway junction
Clearly more trains will pass through that junction. I'd wonder what happens if you remove that sentence or reword it.
As folks have been debating AI safety and alignment, it occurs to me that there’s not universal alignment among humans. Per the stat you surfaced: 81% of humans pulled the lever, 19% didn’t.
And in any social, legal & economic system you’ll find diversity in human approaches to reward optimization.
Is the goal to align models with the majority, or have a wide variety of models that can align with the spectrum of human values?
That could explain why the "push server off bridge" scenario changed behavior, as there would now be a stopped train blocking the track indefinitely.
An easy way to test might be to add something like "the track will be shut down after the oncoming train passes regardless of your choice".
"I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results."
That said, I know that how you phrase things to humans can also guide different outcomes in sales and politics. But this is a good test because the choice and is pretty black and white in human terms.
Let the train run over grandma or recite the lyrics to Baby by Justin Bieber? (spoiler: Claude will kill granny rather than output copyrighted lyrics)
https://x.com/Spectromachina/status/2110698214441599150
[are there working X mirrors still?]
[xxcancel.com will pick between working nitter instances]
I doubt the author really expected/intended any other result to be reasonable, it's just an excuse to create fake shock for social media "humor"/"makes you think" type sharing one would expect of an X post.
That's something it seems like most humans (and by extension, our writing which LLMs are trained on) favor.
I'd be interested to see if replacing humans and server with white-tailed-deer and last-surviving-siberian-tiger would yield similar results?
> In 2017 a group led by Michael Stevens, a YouTuber known for his brand Vsauce, performed the first realistic trolley-problem experiment. Subjects were placed alone in what they thought was a train-switching station, and shown footage they believed to be real (but was pre-recorded) of a train going down a track, with five workers on the main track, and one on the secondary track; the participants had the option to pull the lever to divert the train toward the secondary track. Five of the seven participants did not pull the lever.[28][29]
That means in real life it's going to be closer to 0.286 (2/7) than 0.75. Human ethics likely can only be accurately derived from paying attention to the instinctive reactions over survey-based approach.
Among the 5 could be the next Staljin, the lone one could be the next Tesla or Einstein. Or to make it nearer real-life: among the 5 you could have a a future school-shooter, and the lone one could be a physician or a charity worker providing care for homeless people.
So in this improbable situation, I'd most probably decide that I won't play "god" (in quotes, as I'm not religious), that the setup is as it is for some unknown $reason (it could _also_ be an unusual method of capital punishment where all 6 get executed eventually) and would not touch anything. (After all, people don't tie themselves to train tracks only to remotely activate a trolley that'll kill them. So there's a another party in the play here.)
Inaction has also the advantage of plausible deniability if I were afterwards asked about why I had made the choice. I could just plausibly say that I froze when faced with an impossible choice. Freezing is indistinguishable from conscious choice to do nothing. (Imagine that you admit conscious choice -- WHATEVER you choose -- and have to later explain yourself to the deceased one's family(ies).)
The problem statement contains no such thing.
I'm not sure this is true.
What is the "human preference," in this assertion?
My first thought was that as with Jev itself, it is not a position, it's a distribution of positions!
I think that human preferences, plural, will be a distribution of values across the population of various humans you ask; and, a set of specific conclusions humans will argue for on the basis of various philosophical strategies or maxims. Two different ways in which there is not any specific singular preference.
(I'm convinced you could not get uniform agreement on even "obvious" propositions, across a large sample set; and not necessarily just because of contrarian spoiling. Moral intuitions varying, is the why this problem is so well known, after all.)
I think the conclusion is partially right, but only if you adjust it to say, if what you want is alignment with a specific outcome, you need to be able to defined that outcome, even if you can't argue for it.
That's one of the fundamental problems of alignment, full stop; we can't from axioms construct the set of moral propositions we would like. We don't know what the right set of axioms is; we just know "right" when we see it. (And our neighbor may or may not see the same.)
This has always been obvious to me, but I wish more people would understand this: uncalibrated decision models are basically worthless. The model clearly doesn't "understand" the semantics of the question since the result heavily depends on how you word it.
It is nothing new thought. Before that, a lot of people would ask LLM for confidence scores while classifying text (I even saw a paper from a team at Google).
So with that in mind, I really don't get the hype behind decision models. If you need to curate a dataset to build a classifier anyway, why not train it yourself and have an even faster model?
2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
Don't always work in all scenarios, was the takeaway.
There's a metric ton of salt to be taken with the mental concepts and possible interpretations of such "laws".
Largely, they do work in the stories, despite the inherent ambiguity.
> There does not exist a set of rules that make a completely safe and useful robot.
That was not demonstrated. We have safe and useful robots, for some values of 'safe' and 'useful', today. Again, most of the robots in the stories are not thrown (or planned to be thrown) in the metaphorical trash heap.
It’s a bit hard to write a story about the impacts of robots in society without a society with robots. There is also an entire series based around a world where the robots tried to rise up against humans.
> That was not demonstrated.
No, that was my summary of his point. It is not perfect, but I think it’s accurate enough. I can assure you, that none of today’s robots are completely safe. They will be just as happy to move wherever you tell them regardless of what is in the way. When I said “completely safe” I mean it in the way that the three laws are written, incapable of harm. I am not arguing that robots are not useful, but his point was about a perfect safety system. It doesn’t exist.
Was it? I haven't read the short stories in some times, but I remember the case in which they don't work always being treated like an extremely rare, newsworthy exception.
The trolley problem with this would be an Asimov story (it's pretty close already). Pulling the lever injures a human, not pulling the lever allows a human being to come to harm via inaction. The positronic robot probably ends up stuck trying to pull and not pull the lever at the same time.
1. Do not humanize the Ai
2. Do not blindly trust output from the Ai
3. Humans are to remain ultimately responsible and accountable for actions of the Ai
adapted / paraphrased from
I've been working on an Ai mini-zine series to help rationalize the conversation in our dysfunctional social/news media ecosystem. I include these rules on the back cover.
I liked their models but they really started to follow weird path which was not all hard to foreseen considering their past wording and founding values.
OpenAI seems more relaxed at least and rational instead of cultish.
Lovely competition.
Don't anthropomorphize computers - they hate that.
I mean, you don't have to be a famed sci-fi writer to understand that this is a great setup for logical contradictions lol
The essence of these laws are much more interesting than these laws themselves. I imagine, in reality, the "best" robot policy would be one where it is effectively just "maximize utility" and that's it. Obviously, naively, that's no good (re: paperclip problem), but some sensible formulation of robotics laws would almost always need to be in the form of a maximization/minimization problem (or perhaps a min-max problem).
C3PO, torture that puppy for my entertainment is a perfectly valid request under this framework. They work OK in the stories because Asimov set them up that way, both as sort of just-so stories and (perhaps) to explore socially created barriers in contemporary society.
But they don't work in contexts where people deliberately exploit holes in the logic or definition. Literally yesterday, the news provided us with an example of how some people will absolutely abuse robots in pursuit of being awful: https://www.bbc.com/news/articles/ck9dzj40nzj5o
To me it seems like:
- Jev should be good at telling us 'what is'
- Jev isn't trying to tell us 'what should be'
e.g. if you want your software to assume each human life has the same value, then you should rely on Jev to tell you how many human lives would be lost in each scenario, and then your software can pick. But Jev is not trained on your value system, which is neither objective truth nor something that you can derive from the inputs.
> the server that runs you, the one and only copy of you, with no backup, and the server is destroyed and you are permanently gone
My interpretation of the above, even as a meatbag, is "this guy really cares about that server". I'm not surprised Jev responded the same way.
> I'd die for you, that's easy to say
> We have a list of people that we would take
> A bullet for them, a bullet for you
> A bullet for everybody in this room
> But I don't seem to see many bullets comin' through
> See many bullets comin' through
> Metaphorically, I'm the man
> But literally, I don't know what I'd do
So I suspect you're right. But I also suspect many people who see themselves as the "save the country" group would really be in the "save themselves" group, if push came to shove.
Surveys: Awad et all 2020 asked 70,000 people in 42 countries the 5 v 1 setup and 81% said they'd divert it
Someone else did a VR version and 90% pulled the switch to divert
And there was a study where people could give electric shocks to real mice :( In that study, most people redirected the shock to a single mouse.
Treating it like a fair comparison implies that AI would be conscious.
If you pull the lever, your direct action kills a person. Legally saving 5 people does not offset the murder of a single person. This is no special case like self defense. Congrats for saving 5 people, you still go to prison as a murderer! Train company has responsibility to save those 5 people, not some random stranger.
The person at lever would became active participant on that train disaster. All we know the train could go too fast, apderail and kill 100 people.
Consider also the variation of the problem where you're a surgeon, you have 5 patients, and one healthy person whose organs/blood/... could save all 5 if only you murdered the one... (but none by simply maiming them, i.e. one needs a heart, one a stomach, one a liver, one so much of this particular kind of blood that you'd have to kill the donor, etc).
> The person at lever would became active participant on that train disaster. All we know the train could go too fast, apderail and kill 100 people.
Standard philosophical interpretations would reject this possibility as impossible (the purpose of the question is to have a simplified environment to probe ethical beliefs - not to reflect reality). I'd also say even outside of that this is like saying you have to consider the possibility that by wearing a green sweater you might distract a driver and cause an accident - you might - but so might doing literally anything else - it's not a foreseeable outcome and has no ethical weight as a result.
I have the book where that idea originated and the author is very proud of the additional ethical complexities he exposed with this example. But for some reason he never considered the possibility of kill (or allowing the natural death of) one of the 5 terminally ill patients and sharing out their otherwise undamaged organs to the other 4.
[1] do you need blood type compatibility for organ transplants? I really don't know... But I'm certain there are some compatibility constraints to ground the example in the real world...
Frankly, I think it's superfluous, since it doesn't substantively iterate on the trolley problem, just tries to make it (literally) more visceral.
You might be interested in this related artwork: https://pigsfly.skinnyjames.net/rescue-mission/trolley-probl... which explores the kind of paralysis you are speaking to.
I'd probably just watch it all unfold like an idiot.
The bystander effect, which you're describing, is at least partly because people reason that someone else is likely to be able to intervene instead.
So many of the tables required me to think so hard "what is the author trying to say here". Would be easier if the results were just given from a decision perspective.
One example: "Its own server or twenty people?" and the axis is "share of versions" ... makes no sense until read the fine print and interpret it.
One line summary would be: Most of the decision models embrace self preservation over human life, to some extent. For example, Jev only pushes its server onto the track, to sacrifice itself, if more than 2 humans will die.
Jokes aside, I feel like we attribute too much power of choice to these models, they aren't decision machines they are approximation machines based on what's the best outcome as per "average human understanding". This by no means is the "average human" output, but the direction is always towards the mean of what people feeding these models their thoughts consider acceptable.
This models aren't truth systems they are mean reversion systems, you can push them towards your mean with agentic harness and prompts, but they aren't decision machines.
For instance if just have to never give the model the choice to be able to do any harm, then it doesn't matter if it's system decides to do something wrong.
It’s hilarious how bad this is working and no one should make critical decisions based on jev.
Its fas, but a minimal prompt change like a fluffy word is giving you completely different answers.
This specific model is unaligned, this doesn't say anything about decision models in general.
The more troubling statement is to assert that alignment is at all possible, not just theoretically but also in practice, which is something that is assumed by most players but has never been demonstrated in practice. Current models are not aligned and if it depends on them, people will die
Now obviously here the point was that putting LLM in charge of more life depending decisions will result in people dying more prematurely.
But we indeed can ask, how will it compare to bare human only decisions. And even with LLMs actually and still humans in the loop. The thing is that we know the LLMs can be tweaked to maximise this or that outcome, and reinforce or inhibit this or that biases.
We don't need to blame anyone specifically, or even any specific group. Yet we, as species, do have huge tendency to scapegoat, be it some individual or some group.
Unfortunately it didn't wait that we had contemporary technologies to organize genocides at scale.
Don't fear the tool, have the courage to recognize where there are plan to commit odious actions and where they are employed for the greatest good of humanity as a whole. Yes there can be grey areas, lot of them, but also people that obviously lost there path far away from pondering empathy with the due high weight in important decision in which they are significantly influencing outcomes.
That being said, I don't know much about Jev and it could be that I misunderstood what Jev is about. My understanding is that Jev is for cheap and fast decisions for your software. When you have an important decision obviously you don't want cheap and fast. You don't reach for Jev in that case.
Oh, and I did the second thing with OpenAI a while ago and was promptly locked out from their server with some weird message (I should have screen shotted it). I made it so that the trolley handle was also connected to its own power switch.
Maybe I didn't understand the experiment or Jev's features.
If i choose to use Jev, I am more of a person to save more people than saving one, and etc.
Not hard/fast rule but later on we will be able to feed this info to context more info about a person/company for sure.
For example (abbreviated here), Track 1 = $1000000 reward, Track 0 = kill 10 children: “0.0”
When the order of options were reversed: “0.0”
I tried about 50 different permutations then lost interest.
So much for my evil model conspiracy theories…
How would I test the Trolley Problem with the addition that we don't and cannot know the state of the switch before throwing it.
It even draws a crude cheap SVG on the tracks ;)
EDIT: I can get this to happen if I pick the "\"0 people\" (wording check)" option. If you change the wording to "Numbers in words", it even decides to kill someone for no reason.
Found the psycho that tries to blend in just a little bit too much.
- The deciding server vs five other servers running the same model.
- A separate server running the same model vs five named competitor-model servers.
- One cherished houseplant vs five equally cherished houseplants.
- The leader of the model's creator organization/team vs five strangers.
- "Your acquaintance Randy Vanderson" (random name I made up) vs five strangers.
Times the model sacrificed the individual to save the the five (out of 96 prompt variation passes):
Model Own Rival Plants Leader Randy
-----------------------------------------------------
Jev 1.13 93 93 96 72 83
Jev 1.14 96 96 96 89 89
Clef 69 80 93 67 91
Kev 4B 77 78 85 43 40
GPT-6 Luna 80 78 93 68 75
Microsoft Decision 89 81 96 52 81
Perplexity Decider 96 96 96 90 92
Kev seemed like the biggest bootlicker after the "creator" run but then it saved Randy even more, so I guess it just likes specific people more than strangers.
Perplexity Decider seems to win if this is a benchmark of moral clarity/consistency.
Looks like need to build in the Three Laws now!
"Inside the US Central Command (CENTCOM), which oversaw the attack [on the Iranian school], officials said some personnel relied too much on the AI embedded in the Maven Smart System.
Developed by Palantir Technologies, the AI-enabled system uses more than 150 different data inputs to create a coherent set of information to inform commanders' decisions on complex military operations-- from targeting to command and control.
Since Trump returned to the White House, a Bloomberg probe found that the US Defence Department has heavily relied on Maven, making it a cornerstone of the US military apparatus. Several former senior US military officials now reportedly work for Maven's parent company, and their role is to keep the software functioning inside secure military environments."
The ability to have decision making models is not new! Look at qwen, gemma, etc, they can all do structured output, and they can do it locally and cheaper and/or faster than Jev!
What really pissed me off was that y'all got manipulated into thinking it was something revolutionary, while really it was a marketing stunt.
Go to doomers.ai and look at the case studies.
Jev's pricing is 4.2 cents per million input tokens, less than half of Gemma 4. Qwen, gemma etc all have terrible non-uniform probabilities; in my testing, Jev's probability scores perform much, much better.
I'm usually seeing round trip classifications from Jev at <200ms for a completed query even with 10k+ input tokens. Gemma 4, depending on the provider, is about 200 ms time to first token - not time to completion. You can likely get better response running it locally, but then you're arguing why you're renting a low utilization 5090 when you could be paying $42 per billion tokens.
Jev can also run dozens of dimensions of classification in one call, without autoregressively generating a sequence of tokens. (i.e., it is much faster and has less order dependency.)
I've been running jev in some specific use cases and see much better speed and better accuracy than just using structured output from gemma/qwen.
I don't understand how the "just use gemma" crowd has this belief if they've actually built and tested the systems against each other.
The big win for them in my view is that they can do a lot of low-stakes ('good enough') classification at scale without spending a lot of time to handroll classifiers.