michaelbuckbee

It's a free site, so I was trying to limit both the privacy and risk exposure.

Making the content auto expire after a short period of time greatly decreases the attractiveness of the site to lots of SEO spammers and other types of abuse, and if someone were to get something malicious or vile posted it will clean up after itself without me having to wade into things.

Like Simon concludes the article, the main use of this isn't to say which model is "better", but to try and poke at the model to sort out things like quality vs cost vs speed.

So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.

Kimi is cheapest by 5x but also slowest by 2x

https://9gpyw4uxr2.evvl.io/

I built a simple (free) eval tool for my own uses (Github Gists + Model Outputs) after not being able to find a suitable one in the market.

The market's being split into

1. Longitudinal LLM observability tooling

Most eval startups have gone down the route of something more like being an observability platform for LLM inference. They want to be in your stack and running the inference to collect data on performance of it.

They collect things like how often a model returns JSON that's out of spec or returns values that aren't expected as well as general timing and cost info.

2. Safety Limiting / Pentesting

Say you're doing something in the medical field or that's sensitive in some way and you want to figure out what model has the best outputs for your task that won't fly off the guardrails.

3. Simple cost + performance + quality swapping

This is what my tool does, basically lets you test if you _really_ need to be running that frontier model in a loop across a million records or if you'd be better with an older model or something else.

https://evvl.ai/

Example eval: https://giyd8stidy.evvl.io

Vesting schedule?

Something that's improved my life has been buying a sticker sheet of those LED darkening dots. They're only a couple bucks and look much cleaner than other solutions I've tried while still allowing for _some_ light to come through.

This, more than anything else I'd ever read about Inform, really makes me want to give it a try.

I ran a quick eval to see what this looks like qualitatively vs just calling Opus 4.7 or GPT 5.5 directly.

As expected, Fusion was 7x slower and 4x the cost.

This isn't a knock against it, just that it I think this places Fusion into a "use it only when you need it" category.

https://3fpi5avcqq.evvl.io/

A counterpoint to this is that we have some real different definitions of AI.

If you consider things like the machine learning filters in your smartphone camera and Google's AI Overviews for searches it's entirely plausible that the US is currently at 75%+ of AI usage.

I think it's your last point that's actually the strongest.

There's always gaps between theoretical and practical, but to see China investing so hard in the future while the US digs in it's heels is infuriating.

I thought it was more implied, but let me be more explicit:

- This is something I made for myself without a lot of commercial thought, so I still haven't thought through pricing + usage + limits + operational limits. In it's current wildly unoptimized state it's still very cheap to run.

- For the specific concern about API Key leakage there's not a lot I can do about that (that I'm aware of) as the logic of what gets sent is handled by the client AI. It is possible to pull down and audit both the tools + instructions that are published by the MCP server if there are concerns on that side.

This is all very fair criticisms. This thread asked: "What are tools you have made for yourself?" and that's genuinely what this is. I wanted it so I made it and then AI makes it so easy to just throw up a marketing page.

I've a a handful of dev friends that have started to use it as well and give their feedback and it's been slowly growing as I've added sharing/invites.

I would absolutely not recommend putting big production data into it currently.

My vision for it was something more like how the #1 use of spreadsheets is actually people making lists and not actually people doing lots of calculations.

Given the uptake today (thanks everybody!) and your feedback (thanks Mystery-Machine) I'm going to work at addressing your concerns.

The funniest thing I've made is a free utility called "Moniker" that contextually renames files based on their contents.

Uses local AI models and I was able to snag this great domain name.

https://finalfinalreallyfinaluntitleddocumentv3.com/

But hands down the most useful thing I've made is HutchDB, which is a MCP service that you can call from any AI chat or Agent setup to store data for you.

Literally from your AI you just say "save that to Hutch" and then it figures out:

- The schema + fields - Builds nice webviews (Kanban, Timeline, Grid, Calendar) - Lets you share the output with people

So people use it for all kinds of things like time tracking ("every hour save a summary of my activities to Hutch"), for Agent to Human handoff ("Every day check social media for mentions of my company and save them to Hutch").

I use it for things like recording all of our marketing activities and then having my AI compare those to signups for rough attribution, etc.

Dead useful and at https://hutchdb.com

It's not just comparing all the models, it's also comparing all the providers and configurations of those models.

If you're doing any kind of production AI work you'll end up with outages caused by calling a single provider, OpenRouter seamlessly switching between providers is a godsend for uptime.

But even more than that there's meaningful cost+speed differences.

Here's Sonnet 4.6 being served direct, via Amazon and via Google

https://la9q13gg8w.evvl.io/

(spoiler: Google was both fastest and cheapest)

WRT the native grammar, consider adjective order. Few native english speakers (me included) can off the cuff name the proper order, but everyone knows the "right" order.

https://dictionary.cambridge.org/us/grammar/british-grammar/...

Assassin's Creed Brotherhood is kind of like that for the architecture and period it covers.

There's an interesting small YT channel that did a series on ACB + History

https://www.youtube.com/watch?v=hebq-fObdhY

That's a a lot less than I expected. Is it difficult to get coverage?

We're already seeing other states struggling with soaring house insurance rates (California and Florida) and a pattern of spikes in rates leading government mandated caps on rates leading to insurers pulling out of the states.

I've no idea what the going rate for insurance is currently in New Orleans but it has to be crazy right?

I was curious just how much of a difference there was, so ran a quick eval comparing them and fwiw DeepSeek is considerably slower but much much ~5x cheaper than Haiku and fwiw ~35x cheaper than Claude Opus 4.7.

https://07ytscmybx.evvl.io/

It it reminds me of a lot of friends who wanted to "start blogging" and their first step was writing a new static site system from scratch.

I was trying to get a better sense of the time cost quality matrix of these, so I threw together a quick eval of Sonnet 4.6, Mistral's dev model, and Opus 4.7 (figuring it's what you'd use if you were on Max).

The results for a function implementation and test of levenshtein distance in js are pretty similar but Mistral is 30x cheaper than Opus 4.7 and 4x faster than Sonnet 4.6.

https://5m6qnuhyde.evvl.io/

My plan was to just see if anyone wanted to actually use it first. That if I couldn't give it away I'd not invest the time in selling or open sourcing it.

I'd sort of designed it for my own needs first and hadn't thought too far beyond that.

I'd be curious to know if you consider a Lego store a "toy store". There's one that opened in my city fairly recently and is in an area of smaller boutique shops (kind of like what you described).

I made a B2C AI app that's fully local (and free) to do AI based contextual file renaming.

So if you give it a bunch of screenshots it will try and intelligently name them based upon what is in the screenshot. Same for videos, PDFs, etc.

But to your point I haven't even tried charging money as it feels like something Apple is just going to bake in as a feature.

https://finalfinalreallyfinaluntitleddocumentv3.com/

That's fair and fwiw something I'm in firm agreement with you, but also just not what I took from your comment.

Search ads do seem like the one ad type that kind of flips that though. Where it's not based on some general set of interests, but literally the thing you're searching for at that moment.

All of the cradle-to-grave studies I've seen about greenhouse gas emissions for renewables versus coal/oil still indicate massive improvements.

This government meta study of 3,000 such studies puts PV solar at roughly 20x less emissions than coal.

https://docs.nrel.gov/docs/fy21osti/80580.pdf

I took this a different way which was that to google railway is their customer and out of a variety of professional and security considerations want the communications to come from their customer and not them.

Something that the Overcast podcast player does (and probably others) is silence removal, which in some ways is even better than the raw speedup.

Exactly. The larger the organization the less percentage time devs actually are doing dev work and the less direct benefit there is from AI assisted coding tools.

Slightly different slice into this a very similar situation (local vs OpenRouter AI inference).

But in _every_ metric other than privacy it was better to run via OpenRouter than a local model, and not by a small amount.

Direct link to the comparison charts:

https://sendcheckit.com/blog/ai-powered-subject-line-alterna...