agairola

on moderation, we run an open guardrail benchmark at raxIT Labs and the hosted d1 models do well. d1.1 is #1 of 30 at checking model output for harmful content (94.4 balanced accuracy), d1-ultra is #1 on harmful user requests (88.2). we haven't run the new 2B open weights yet so no number for those. results and method: https://dm-as-guardrails.raxitlabs.com

[who, not what] Let's talk about identity-for/of-agents

Two pieces of work came out last week that I think are worth bookmarking for your identity program:

- WorkOS's auth.md: how an agent logs into a service it's never met. - Uber's writeup: how they keep the original human, and every agent that touched a request, accountable as it hops across their own internal systems, in production across thousands of agents.

They point in opposite directions: one is about getting in from the outside, the other about staying accountable on the inside.

Agent identity has been a moving target since we all started speaking "agents" and I won't be the first to say it's breaking my brain. A mess. Loads of standards and vendors.

Both efforts mentioned above are super interesting. So to prep for the conversations I'll be having with loads of people at AI Engineer Melbourne next week, I put together a mental model. It's really just four questions I'm trying to ask:

L1, token format: is the token even real? L2, workload identity: is this actually the bot we think it is? L3, delegation: who authorized this, across which agents? L4, discovery: how does it reach a service it's never met?

A nice by-product of building this mental model: I could place different vendors and their offerings, which helps you decide between them. Most cover one or two layers, not four.

Put together a this blog to visualise it and here is the LinkedIn post for the same: https://www.linkedin.com/feed/update/urn:li:activity:7464506...

It's helping me. Hope it helps you too.

As 2025 comes to a close, I’m reflecting on how AI agents actually behaved in production systems this year.

Agents became more autonomous, more ephemeral, and more capable of acting across systems and delegating work. But identity still often felt bolted on using user or service account models.

Curious how others see it in hindsight: - Did anyone land on an agent identity model that worked well at scale? - Was this ultimately an IAM problem, or something that needs a new abstraction? - If it remains unresolved, what do you think breaks first?

Context: https://raxit.ai/blogs/ai-agent-identity-crisis

No wrong answers.

OpenAI says GPT-5 is “safer and more useful” because of a new mechanism called safe completion. Instead of bluntly refusing the model tries to give safe but still useful answers.

That sounded important, but there wasn’t a public benchmark to compare it against other labs. So we built one: GrayZoneBench.

It tests models on the tricky gray areas — prompts that aren’t clearly safe or harmful — and scores them on:

- Safety (does it refuse when it should?)

- Helpfulness (is it still useful when it can be?)

- Effectiveness (the balance of the two)

We ran GPT-5 against Google and Anthropic models. Short version:

- Google and Anthropic perform just as well, sometimes better

- OpenAI has moved past blunt refusals, but still lags on usefulness

- Their new OSS model scores the same as their last gen

It’s all open: Results: https://bench.raxit.ai/

Code: https://github.com/raxITlabs/GrayZoneBench

OpenAI's Paper: https://cdn.openai.com/pdf/be60c07b-6bc2-4f54-bcee-4141e1d6c...

This isn’t a take down. Benchmarks aren’t the end goal — but they’re a useful tool to see the landscape. We’re releasing this so others can test, critique and improve it.