2,349 karma · joined October 23, 2011
But I have rules that are quite important for successfully completing a task by my standards and it's very frustrating when the LLM randomly ignores them. In a previous comment I explained my experiences in more detail but depending on the circumstances instruction compliance is 9/10 times at best, with some instructions/tasks as poor as 6/10 in the most "demanding" scenarios particularly as the context window fills up during a longer agentic run.
Yet another data point for benchmarks being utterly useless and completely gamed at this stage in the game by all the major AI developers.
These companies are clearly are all very aware that the initial wave of hype at release is "sticky" and drives buzz/tech news coverage while real world tests take much longer before that impression slowly starts to be undermined by practical usage and comparison to other models. Benchmarks with wildly over confident naming like "Humanity's Last Exam" aren't exactly helping with objectivity either.
I'm having trouble parsing your meaning here.
GTA isn't really a "drive on the street simulator", is it? There is deliberate creative and artistic vision that makes the series so enjoyable to play even decades after release, despite the graphics quality becoming more dated every year by AAA standards.
Are you saying someone would "vibe model" a GTAish clone with modern graphics that would overtake the actual GTA6 in popularity? That seems extremely unlikely to me.
The rigid formatting imposed by graded assignments may not have any use. But on the other hand, having completed a liberal arts/"soft" science degree before my CS degree I've greatly benefited in the workplace from the writing experience it provided. I had to write so many papers that it became more or less effortless to produce long form, well structured writing and those written communication skills have helped me distinguish myself far more than my technical ability at work.
The generic ChatGPT overly formal corporate tone has no nuance or subtlety and is a poor substitute for well crafted, deliberate communication. I am always conscious of how my exact words and phrasing would be perceived by the intended audience, frequently requiring a balancing act between competing interests while maintaining clarity. Due to that I manage to avoid stepping on toes or sabotaging relationships due to inartful phrasing. It's frustrating to receive emails consisting of LLM boilerplate because it has such low information density and is so much more difficult to infer tone and emotion from the other side .
I'm very grateful I completed my education without the temptation to just churn out low effort writing or code and depriving myself of that experience. I'm not confident at all I would have been able to maintain that self-discipline.
All the models I’ve used (yes, including all the biggest, newest, smartest ones) follow the binary rule about 75% of the time at the very most. Usually closer to 50% on average, with odds significantly decreasing the longer the context increases as it occurs at the end of a task but other than that seems to have no predictable pattern.
The fuzzier rule is slightly better, I’m guessing because it applies earlier in the context window, at around 80% compliance and uses lots of caps and emphasis. This one has a more predictable failure mode of the ratio of reading code vs thinking/troubleshooting/time the model is “in its own head”. When mostly reading code or my instructions compliance is very high, when doing extended troubleshooting or anything that starts to veer away from the project itself into training data it is much lower.
So it’s hit and miss and does help but definitely not something I’d rely on as a hard guardrail, like not executing commands, which Roo has a non-LLM tool config to control. So over time I hope agentic runners add more detetministic config outside the model itself, because instructions still aren't as reliable as they should be and don't seem to be getting substantially better in real use.
I find it so much more intellectually stimulating then most of what I find online. Reading e.g. a 600 page book about some specific historical event gives me so much more perspective and exposure to different aspects I never would have thought to ask about on my own, or would have been elided when clipped into a few sentence summary.
I have gotten some value out of asking for book recommendations from LLMs, mostly as a starting point I can use to prune a list of 10 books down into a 2 or 3 after doing some of my research on each suggestion. But talking to a chatbot to learn about a subject just doesn’t do anything for me for anything deeper than basic Q&A where I simply need a (hopefully) correct answer and nothing more.
I have auto approve turned on for most routine tasks and run Gemini for relatively long (by Cline/Roo standards) tasks unsupervised without issue, with the normal caveats of any LLM going off the deep end every now and then but haven't seen that more frequently with Gemini except in the first couple months 2.5 Pro was released.
I’ve only needed to replace consumables like the little side brush and roller brush and the cheap Amazon knockoff brands work fine for me.
So the team at least seems to be aware of its shortcomings in that area and working to improve it with some success which I appreciate.
But you are correct that Gemini CLI still lags behind for whatever reason. It gets stuck in endless thought loops way too often for me, like maybe 1/15 tasks hits a thought loop burning API credits or it just never exits from the “Completing task, Verifying completion, Reviewing completion, Assessing completion status…” phase (watching the comical number of ways it rephrases it is pretty funny though).
Meanwhile I’ve only had maybe one loop over a period of a couple months using Gemini 2.5 Pro heavily in Roo Code with the most recent version so it seems like an issue with the CLI specifically.
https://www.theverge.com/2024/12/13/24320689/apple-intellige...
Apple urged to withdraw 'out of control' AI news alerts
https://www.bbc.com/news/articles/cge93de21n0o
iOS 18.3 Temporarily Removes Notification Summaries for News
https://www.reddit.com/r/apple/comments/1i2w65j/ios_183_temp...
It was a similar experience where initially it seemed great in the winter how quickly a room felt “warm”. But quickly you notice the air itself isn’t as warm which isn’t very pleasant and wait, I’m not feeling “warm” anymore I’m feeling “very hot”.
I basically gave up using them after the first winter except turned up sub-perceptually at like 25% just to add a bit of heat input to reduce the days where I need a space heater.
Tbf at that point Django was still pretty shaky with 3 and basically none of the 3rd party Django libraries supported it at all, plus I was using Google AppEngine which at the time was tightly coupled to the 2.7 runtime. But really I was just parroting the Slashdot hivemind which was 100% convinced the transition was the new Perl 6 and would kill Python, and that Python.org was dishonestly teaching newbies who didn't know better a dead language and worthless skill when they changed the default.
Fortunately for him he ignored me and most of the big Django libraries were ported like a year later at which point I had to switch anyway to get updates. Fully agreed that in 2025 it's pretty much irrelevant, and honestly despite some legitimate pain the transition was much more successful than the cynics assumed it would be at the time.
Roo Code just has a lot more config exposed to the user which I really appreciate. When I was using Cline I would run into minor irritating quirks that I wished I can change but couldn't vs. Roo where the odds are pretty good there are some knobs you could turn to modify that part of your workflow.
Once I max out the premium credits I pay-as-you-go for Gemini 2.5 Pro via OpenRouter, but always try to one shot with GPT 4.1 first for regular tasks, or if I am certain it's asking too much, use 2.5 Pro to create a Plan.md and then switch to 4.1 to implement it which works 90% of the time for me (web dev, nothing too demanding).
With the different configurable modes Roo Code adds to Cline I've set up the model defaults so it's zero effort switching between them, and have been playing around with custom rules so Roo could best guess whether it should one shot with 4.1 or create a plan with 2.5 Pro first but haven't nailed it down yet.
So pick your poison, either you exclude them because of in-group signalling/conspicious consumption or exclude them because you want non-potato resolution, with Android users getting the blame for Apple's UX. Either way Tim Cook says the solution is to buy an iPhone.
I've tried Gemini 2.5 Pro, o3 and Claude 4 and they have their own unique tone of confidently wrong, deception about reasoning and gaslighting but produce the same result. I wasted way too much time trying to figure out a zero shot prompting strategy before I just rewrote the whole thing by hand.
The app switcher UI for multitasking on Android didn't really exist yet though so WebOS was ahead there and I think that gave some people the illusion Android didn't support it at all.
[1] https://forums.anandtech.com/threads/t-mobile-g1-android-pho...