Ultramarine 1 Flash
paperinstruments.com
paperinstruments.com
In case anyone was curious what it's based on. (No judgement either way from me, IMO depends how well they did the post-training, which I can't tell).
this is their problem, not yours. at least disclose # of flops spent or amount of data used in an attempt to give us an idea of how nontrivial it was. cursor at least does this with their composer model
In the "Results" section, they choose to compare Ultramarine only to models that it's better than on every metric. They ignore models like Fable 5.1, GPT-6 Astra, Claude Opus 5.5, etc.
Then, in the "Score against cost per task" graph, which is focused on price, suddenly all of those models that perform better than Ultramarine come roaring back because they are also more expensive.
Just list yourselves as the second best sometimes. Your thing is impressive. Don't let the idiot marketing guy hide that behind stupid tricks.
but it was way deeper than that, because it wasn't actually about people looking up some search string, it was actually about predictably pinging the google alerts for the top company and their founders, and then my same publicist messaging them as a rando on linkedin or email suggesting that they join us for a fireside chat at a conference they had their hands in
with the fireside chat in front of everyone, and the record of it, boosting us to the collective conscious riding off of the more established brand's presence
but yeah our personal google results also looked amazing for several years too
everything serves a parallel purpose, but yeah there's almost no utility in self aggrandizing on your own homepage
I can’t remember a time when I found a benchmark to be genuinely useful even for deterministic software
Normally, the background highlight color indicates the leader for every benchmark, but they chose to highlight their column even though there are better models.
APEX Agents => Claude Fable 5 and Gemini.
500px of the landing page is dedicated to share how they're more "token efficient" than the base model they used. Apparently this -6.5% difference is worth highlighting as the very top feature of this fine-tune they've made.
How much is 6.5% percent concretely though? Assuming the six benchmarks have ~30 tasks each (180 in total), and we're doing 120 tok/s lets say, time/task for GLM-5.3 Flash would be ~25 minutes and this "token efficient fine-tune" would arrive at ~23 minutes.
Again, a 6.5% difference is highlighted as the most impactful change for this fine-tune, but it's a marginal difference. They're definitely trying to mislead people into thinking this is something more than it is, otherwise this landing page would have been much more realistic and less self-patted.
This is not a general purpose model, it makes sense to only bench it on tasks it's supposed to be good at, which is to help out finance-related knowledge work.
Honesty and transparency are attractive.
Most use of llms is gambling. So a lot of people get hooked on their one machine or whatever despite all reasons to the contrary. Some of the most deranged LLM users I e encountered are hooked on a variety of models but still use ones from like 2 years ago for specific tasks for emotional reasons.
The truth is, for 99.999% of use cases the top 10 models are more or less interchangeable. Either they can mostly do what someone wants, or none of them should probably be used for that application even if s leading lab is advertising it.
Not going to be trying it just based on this observed behaviour.
Did it do it in 2 tries? 10? It's still coming out cheaper. Brute force the work. This is the path that the labs are taking. It's the 1200 agents trying to solve Hacking Bench hacking hugging face. It's anthropic telling you that its 1000's of staff commits like 10k engineers.
Treat AI like you're deploying interns who don't need coffee, food or bathroom breaks. With the right metrics they will get to an optimal (enough) solution.
Those same folks now use version control - and have saner backup methods than some companies that I have worked for. They are already using multi pass, multi agent workflows and have adopted some fairly sophisticated processes for validation. They are primed for "more agents" attacking a problem in parallel.
Find a worker out side the tech bubble making heavy use of AI and one will feel confident that their future employment is secure.
They say it can create office documents (if I understand correctly), but how does it do that?
What's the thing one needs to plug this into to say "give me a slide deck with last quarter's sales figures"?
__
https://www.paperinstruments.com/feather
According to their Twitter video, it might be this thing?
So there's a side pane and a main section and a text input.
Very..uh.. plain. I was envisioning some more specific UX flow, but I couldn't tell you how, exactly.
The 'main' section is HTML, so it can be any type of workspace you want. Our agent can read/write (with permissions, of course) and can execute code locally (again, with permissions).
The HTML pages we generate are backed by a schema and state per 'workspace', so they persist, can be shared between team members, and aren't just ephemeral read-only artifacts.
Our agent can do work on your machine for you, it has an integrated web browser that allows it to not only search and retrieve info but also perform web-based tasks, and it can package everything up into a nice little contained UI for you so you're not jumping between multiple apps / documents / tabs. That UI is tied back to whatever document / page you were working in -- so for instance our app can have a form that updates a web page, or updates a document on disk and vice versa.
The plainness is deliberate, the design of the workspaces is up to you -- it's your UI not mine and the agent will design everything how you like it. And as far as the sidebar/text input combination goes I can assure you I have spent a lot of time imagining what this UX should look like and frankly this combo just makes sense.
Also a note on the text input / agent. When you move from text conversations to dynamic HTML interfaces you're faced with a question: how do you relate 'conversations' with the agent to workspaces? A one-to-one conversation to workspace mapping? Many-to-one, one-to-many? Should conversations be time-based, where I just start a new one and have to provide context again and can optionally link back to old conversations if I want? This question is actually quite difficult, but my belief was that ultimately you want just one single global conversation with the agent, and you want to let it handle its own memory internally. We basically just treat memory like a codebase and let the agent decide what to do. This means when you type in the text input you can refer to anything you like: any workspace, any conversation, any webpage you're currently browsing, any folder on disk -- the agent gets the right context and can retrieve relevant details as it needs. And it works!
So sure, I'm not a designer no matter how hard I try -- there will be far better UX out there than I've designed. But I think the fundamental mechanism in Radix is more sophisticated than most other tools, including Cowork etc (no shade at all to anyone, I'm a massive fan of Claude Code).
Source: Building a "harness" for organizations with deep MS Office suite integration at https://github.com/EratoLab/erato
So that is mainly a chat window like that Hubspot widget?
I mean I guess that gets stuff done, but I was expecting some fresher ideas perhaps?
So you have two main modes of operations: One-shot "I have this input; give me this output" which people usually do via the web-app, and more interactive editing to documents (usually larger ones) that happens directly in the applications the users are already working in.
And yeah, that's what ultimately gets the job done. We've seen competing solutions dive much deeper into building their own native editors, but they always lack the fidelity required by power-users, and are usually not as acepted by the broader workforce that are entrenched in their ways (our typical end-user is 50+ in a enterprise-like organization structure), so we try to meet the organisations where they are.
What I did was create docker containers with pandoc, compilers (rust, go, python, javascript) and it can create all manner of reports.
Going to try it out for some bookkeeping, e.g. point it to a folder and tell it to book the invoices and point it to a csv file holding a year of private transactions and see them being processed, categorized and then ask about certain trends.
https://www.anthropic.com/news/updates-to-our-consumer-terms
No [reasonable] person expects a free game to not sell microtransactions or ads or something, but a paid game doing that would garner some anger or squints.
For OpenAI they also have a separate opt-out for stopping training on your codex environment:
"Codex has a separate Include environments setting in Codex settings that controls whether additional context from your Codex environments can be used to train OpenAI models. Changing your ChatGPT setting or opting out through the Privacy Portal does not change that setting." https://help.openai.com/en/articles/7730893-data-controls-in...
Anthropic has the most expensive models and train on your data by default. The fact that OpenAI also does shows that the cost of the AI doesn’t seem to be a deciding factor.
https://privacy.claude.com/en/articles/7996868-is-my-data-us...
I've found the max 200 plans to be roughly equivalent, with Claude giving slightly more usage recently, especially after the OpenAI usage slash and upsell to the 500 plan
Switching to OpenCode or Pi harness is trivial for developers. So it certainly makes sense for those of us.
If it's problematic or not worth in your organization, then the switch doesn't make sense as you pointed out.
I’ll absolutely be giving this a go once weights land (a QAT FP4 experts release would be most appreciated)
If the creators of this model are reading, can we see some of these same benchmarks run against the Thomson model, which although it's capable, is considerably smaller in size and baked in knowledge than glm 5.3-flash.
They might mean the intelligence/dollar frontier aka the Pareto frontier, but honestly much of the time that's not a great measure.
If you are doing repeated work on the same task (eg call center triage) then you can always beat a large LLM on price by fine tuning a small one.
If you are doing knowledge work then models are so varied that factors other than price are usually important. For example, lots of people moved away from Opus 5 because of it's awful English, whereas Opus 5.5 has fixed that. These things aren't measured in benchmarks.
What I *really* need for my clients are good enough LLMs that can offer speed and maximum privacy/zero data retention, especially in Europe.
Right now I'm stuck in the loop of data -> pseudo anonymization on privately owned hardware (very expensive) -> third party LLM -> de-anonymize (map back to original content) -> show result.
It's expensive, error prone, and all I need to do are the same things I could do (or 99% of) with some Azure/AWS/Google APIs.
But what these scalers offer is still not good enough, especially for European users. We can't risk having a US judge or agency force those people to handle our data.
So... potentially? The improvements in other areas might help coding, but aren't that big either, except on their own benchmark.
Oh no! That means it's going to destroy humanity!
It will:
* Hack into everything that has a computer, like your home appliances, even a toaster could be and will be weaponized against you.
* Break cryptography, it's gone, the end of the internet as we know it.
* Be impossible to contain as it can even escape airgapped environments by changing the temperature of CPUs to transmit information to other servers nearby which would sense the fluctuations in ambient temperature[1].
Or wait ... maybe the people at Anthropic/OpenAI are just a bunch of lying crooks that would say anything to make a bit of cash. Who would've thought?
1: This completely imbecile statement was made by one of OpenAI's "top scientists", it was not a joke.