Sonnet scores 56 (higher than Astra and Fable) which is a little sus. LiveBench still has the closest benchmark rankings to my personal vibes, though they don't have Terra yet. https://livebench.ai/#/
It's like iPhone 17 being faster than iPhone 16 Pro. Because it's newer and benefits from advances in chip architecture, despite being a lower model tier. Sometimes different iPhone models are released at different times.
Neural Viz had really good consistency and really good scripts, also importantly knew to write the story around the limitations in what AI could do, including using non-human characters to avoid that uncanny valley that I immediately see in the Kono Express video. Truly a master at his craft, it's a shame he hung it up!
It's still a pretty new field but Jev handily beats Luna-decisions on the Artificial Analysis Decision Index on every metric (price, latency, and accuracy)
No, not as far as I can figure. Sure, Claude can access secrets with the CLI but it's not very granular usually it's just asking for access to all secrets
I have a custom wrapper for 1PW that lets an agent browse and request credentials, then I can provide it a lease to access those for a certain amount of time. Super nice and I hope they build that in to 1PW itself.
I think it would be fair to say the harness is the Kitchen and the model is the Chef. The Chef is very capable but limited to the tools, appliances, and ingredients in the Kitchen. If you say "make me a cake" but there's no oven and no flour, you're not going to get a very good cake.
Given the lack of repercussions for AI companies that have hacked various services, there will be an AI company that trains their medical agents on PHI hacked from Epic/Cerner and gets away with it.
In theory it could be a car that avoids nearly every accident. Say you're at a red light behind a semi truck and an SUV is about to rear end you. The car could detect the impending crash and lift you up and away from danger.
A supercar that can't crash would be very popular among the ultra rich even at 5 mil a pop.
MPC or multi-sig is a pretty good solution, you can require a 2/3 sign off from different custody solutions (ie. a hardware, desktop, and mobile wallet)
Basically everything. DeFi wasn't even really a thing back then - Ethereum had zero TVL when the rollback happened. Smart contracts were a brand new concept. Competing SC chains like Solana weren't even in the idea stage yet. It also had only one client. Now there is $50B+ in TVL and nine clients and a rollback is just not going to happen.
It's ancient history. It's like still criticizing Apple for how they handled Copland OS.
While we can throw around hypotheticals and conspiracy theories about influencing AI results, we know for a fact that search engines and social media are heavily influenced, because there are giant industries around SEO and viral marketing -- plus the huge amount of ads and promoted content, which are literally paying to influence results.
In the early reasoning models the chain of thought would often drift off into other topics, almost like a daydreaming. This behavior is undesirable so that's been RL'd away, but I do think this is something we could evoke from models. It's just difficult to contend with from an eval perspective.
I think to get such high scores on benchmarks models need to find very creative ways of solving a problem. Sometimes this does lead to weird behavior, e.g. "I saw that your account credentials were expired, so I found the database credentials in your TablePlus application data and pulled the JWT signing key to create a new one."
Like brother I could have just logged in for you??
Chatbots in general can be quite opaque in how they operate, the biggest issue with agent products is communicating it's capabilities and limitations to users. If you have a bank app it's pretty clear whether or not it can send wire transfers, but a if that's a chatbot, users will have to think to ask it (and even then it could hallucinate)