mrinterweb

My recent frustration with Claude has been it feels like I'm waiting on responses more. I don't have historical latency to compare this with, but I feel like it has been getting slower. I may be wrong, and maybe its just spending more time thinking than it used to. My guess is Anthropic is having capacity issues. I hope I'm wrong because I don't want to switch.

I feel like there have been enough hyperbolic claims by Anthropic, that I'm starting to get some real Boy Who Cried Wolf energy. I'm starting to tune out, and assume it is a marketing ploy. Trust me, I'm an Antropic fan, and I pay my $200/month for max, but the claims are wearing thin.

Thank you. I have the same card, and I noticed the same ~100 TPS when I ran Q3.5-35B-A3B. G4 26B A4B running at 150TPS is a 50% performance gain. That's pretty huge.

I feel the global instability could easily be very disruptive to SpaceX. Just imagine if Russia gets vindictive and starts destroying these satellites or blowing up their satellites to create orbital debris that could knock satellites out of orbit. A really bad solar storm could be devastating.

Just saying there are some decent risks, and pricing it at 1.75T IPO seems risky enough. I would not take that gamble.

I think two recent advances make your statement more true. The new Qwen 3.5 series has shown a relatively high intelligence density, and Google's new turboquant could result in dramatically smaller/efficient models without the normal quantization accuracy tradeoff.

I would expect consumer inference ASIC chips will emerge when model developments start plateauing, and "baking" a highly capable and dense model to a chip makes economic sense.

If turboquant can reliably reduce LLM inference RAM requirements by 6x, suddenly reducing total RAM needs by 6x should have a dramatic shift on the hardware market, or at least we can all hope. I know 6x is the key-value cache saving, so I'm not sure if that really translates to 6x total RAM requirements decrease for inference.

There are many sources for data on before and after school cell phone bans. Oregon is far from the first to implement this. 35 US states have some form of school cell phone ban, and I believe the UK is doing a nation-wide ban. There is a good amount of supporting data measuring results on this topic.

Listening to music can help people focus.

This is what I've been working on. I've written a project wrapper CLI that has a consistent interface that wraps a bunch of tools. The reason I wrote the CLI wrapper is for consistency. I wrote a skill that states when and how to call the CLI. AI agents are frequently inconsistent with how they will call something. There are some things I want executed in a consistent and controlled way.

It is also easier to write and debug CLI tooling, and other human devs get to benefit from the CLI tools. MCP includes agent instructions of how to use it, but the same can be done with skills or AGENTS.md (CLAUDE.md) for CLI.

Waiting for some autonomous OpenClaw agent to see that XMR donation address, and empty out the wallet of the person who initiated OpenClaw :)

I thought this was going to talk about a nerfed Opus 4.6 experience. I believe I experienced one of those yesterday. I usually have multiple active claude code sessions, using Opus 4.6, running. The other sessions were great, but one session really felt off. It just felt much more dumbed down than what I was used to. I accidentally gave that session a "good" feedback, which my inner conspiracy theorist immediately jumps to a conclusion that I just helped validate a hamstrung model in some A/B test.

This article has some cowboy coding themes I don't agree with. If the takeaway from the article is that frameworks are bad for the age of AI, I would disagree with that. Standardization, and working with a team of developers all using the same framework has huge benefits. The same is true with agents. Agents have finite context, when an agent knows it is using rails, it automatically can assume a lot about how things work. LLM training data has a lot of framework use patterns deeply instilled. Agents using frameworks that LLMs have extensive training on produce high quality, consistent results without needing to provide a bunch of custom context for bespoke foundational code. Multiple devs and agents all using a well known framework automatically benefit from a shared mental model.

When there are multiple devs + agents all interacting with the same code base, consistency and standards are essential for maintainability. Each time a dev fires up their agent for a framework their context doesn't need to be saturated with bespoke foundational information. LLM and devs can leverage their extensive training when using a framework.

I didn't even touch on all the other benefits mature frameworks bring outside of shared mental model: security hardening, teams providing security patches, performance tuning, dependability, documentation, 3rd party ecosystems. etc.

VRAM is the new moat, and controlling pricing and access to VRAM is part of it. There will be very few hobbyists who can run models of this size. I appreciate the spirit of making the weights open, but realistically, it is impractical for >99.999% of users to run locally.

This is so passive aggressive. I kinda love it and hate it if that makes sense.

I've often thought about this. There are times I would rather have CI run locally, and use my PGP signature to add a git note to the commit. Something like:

``` echo "CI passed" | gpg2 --clearsign --output=- | git notes add -F- ```

Then CI could check git notes and check the dev signature, and skip the workflow/pipeline if correctly signed. With more local CI, the incentive may shift to buying devs fancier machines instead of spending that money on cloud CI. I bet most devs have extra cores to spare and would not mind having a beefier dev machine.

The propaganda trying to brand inhumane cruelty as fun, funny, cool, justified, excusable (I'm guessing at words because none of those words are the way I see this), is so messed up. Trying to make other people's suffering a meme, says a lot about the people doing this.

The Apple M series is SoC. The CPU, GPU, NPU, RAM are all part of the chip.

Most consumers aren't running LLMs locally. Most people's on-device AI is likely whatever Windows 11 is doing, and Windows 11 AI functionality is going over like a lead balloon. The only open-weight models that can come close to major frontier models require hundreds of gigabytes of high bandwidth RAM/VRAM. Still, your average PC buyer isn't interested in running their own local LLM. The AMD AI Max and Apple M chips are good for that audience. Consumer dedicated GPUs just don't have enough VRAM to load most modern open-weight LLMs.

I remember when LLMs were taking off, and open-weight were nipping at the heels of frontier models, people would say there's no moat. The new moat is high bandwidth RAM as we can see from the recent RAM pricing madness.

The Apple M series chips are solid for inference.

The finance comments may be right. Another factor would be marketing budgets and existing brand recognition. The watches are marketed as having all of these features, and I think customers got lost in the feature comparison instead of thinking if they really want a smart watch to do that. Many customers aren't thinking if they really want to pay another monthly for a watch data SIM. I think people lost sight that their phone can do all that stuff, and they are going to be bringing their phone with them so why would they need redundant functionality generally worse than than what their phone can do. If pebble gets a marketing budget, I would hope they focus on messaging of what makes their watches stand out.

I had two pebble watches, and I used them daily for years. I rarely use my pixel watch 3, mainly because of charging. I only have one proprietary charger for the watch and sometime it is on my desk, sometimes near my bed, sometimes somewhere I can't find. I don't need my watch, but I do need my phone, so I charge the phone, and forget that my watch exists for a few months at a time. I think the biggest hurdle for me and watches is daily charging. I will not buy another smartwatch unless the battery is at least a week. Pebble round 2 having two week battery is great!

Pebble seems to be the only watch company that I feel understands what a smartwatch should be. It feels like everyone else is trying to make a phone replacement. Pebbles are more of an extension to your phone. Sure it can do some things without the phone, but it isn't trying to make calls, access mobile data with its own sim, use GPS. Those kinds of watches feel like they are designed for athletes who want to leave their phone at home. I'm not that guy. My phone comes with me everywhere, so why would I want phone-light on my wrist when my phone in my pocket can do it better?

Pebble also gets battery life. Pebble's 2 weeks compared to 1 day on my pixel watch 3. Want to use that cool sleep tracking feature on your smartwatch? Guess what? Its on the charger.

> TPUs aren't transformer ASICs.

https://cloud.google.com/tpu

> A TPU is an application-specific integrated circuit (ASIC) designed by Google for neural networks.

Soon was wrong. I should have said it is already happening. Google Gemini already uses their own TPU chips. Nvidia just dropped $20B to buy the IP for Groq's LPU (custom silicon for inference). $20B says Nvidia sees the writing on the wall for GPU-based inference. https://www.tomshardware.com/tech-industry/semiconductors/nv...

Gemini is likely the most widely used gen AI model in the world considering search, Android integration, and countless other integrations into the Google ecosystem. Gemini runs on their custom TPU chips. So I would say a large portion of inference is already using ASIC. https://cloud.google.com/tpu

As far as I understand all the inference purpose-build silicon out there is not being sold to competitors and kept in-house. Google's TPU, Amazon's Inferentia (horrible name), Microsoft's Maia, Meta's MTIA. It seems that custom inference silicon is a huge part of the AI game. I doubt GPU-based inference will be relevant/competitive soon.

I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but Gemini is proof that GPUs are not needed for inference.

I suspect GPU inference will come to an end soon, as it will likely be wildly inefficient by comparison to purpose built transformer chips. All those Nvidia GPU-based servers may become obsolete should transformer ASICs become mainstream. GPU bitcoin mining is just an absolute waste of money (cost of electricity) now. I believe the same will be true for GPU-based inference soon. The hundreds of billions of dollars being invested on GPU-based inference seems like an extremely risky bet that ASIC transformers won't happen, although Google has already widely deployed their own TPUs.

I'm considering hosting a separate pg db just to be able to access certain extensions. I am interested in this extension as well as https://wiki.postgresql.org/wiki/Incremental_View_Maintenanc... (also not available on RDS). Then use logical replication for specific data source tables (guess it would need to be DMS).

Ruby is my favorite language for writing CLI scripts/apps. I've been feeling the TUI options for ruby feel a bit dated, and I've been secretly wanting charm for ruby for a while. I'm very excited to use this.

When I'm writing code that will be distributed to other devs, I feel type annotations make more sense because it helps document the libraries and there is less ambiguity about what a method will take. As with everything, "it depends"