Hacker News Reader

Comment by luis_valerio | Hacker News Reader

Hi, thanks for the detailed answer! Yes it's pretty similar to the approach we are following the only difference is that we cannot provide read permissions into the endpoint so we have some kind of compliance or auditor subagent that checks against forbidden actions and polices and as you mention also classify all the request in level of impact enforcing hitl for admins.

We are planning to do the policy check as much determistic as possible with OPA rules and so on, but we also have the policies in place at cloud level

View on HN

Makes sense. On not being able to get read-only creds from the endpoint: that's actually where a proxy helps most. A big part of what we've built is giving you fine-grained permissions even when the underlying provider doesn't have them. The key can be full-power, but the agent only ever reaches the API through the proxy, so "GET only under these paths" is enforced there and a write never reaches the provider even though the key could do it.

On the compliance subagent, the one thing I'd watch is that it reads the same untrusted context as the agent, so it can be injected too. We treat the model as advisory only. We have an AI review rule that rates a request's risk, but it can only escalate to a human approval. It can't deny, and it can't allow anything the deterministic rules didn't already allow, so a fooled reviewer costs you an extra approval, never a bad action. Your impact-level classification maps pretty directly onto that.

To be precise the thing provide permission to do any operation, including destructive actions to the agent but we do set policies, auth controls and hitl force approval to validate the destructive or high impact actions before apply them

Really good point about using the same context for the compliance, currently it's a separate subagent but read rules/policies and then check against the other agent proposal, no possible to inject since we control totally via a rail the prompt for the compliance subagent but it's true that it checks policies so it's not deterministic (works pretty well but not deterministic) so that's we are targeting checking that policies deterministically with OPA and then the compliance will only add to the final result some other checks that need to be LLM inferences.

So it's pretty similar to yours, you have any other additional harness to be as much determistic as possible?

Yeah, sounds very close. The main thing we did to keep it deterministic is that nothing at request time is a model call. Rules are evaluated in the proxy against the actual request: method and path, the provider operation it maps to (we classify each provider's API into operations and families like delete or payment, from their published spec), parameter values, IP, time window, quotas. Default deny, and every rule can only narrow.

The other half is treating policy like code. Rules live in a file with test fixtures ("this request should be denied, this one should need approval"), `alter policy test` runs them, and `plan` / `apply` works like Terraform: you review a diff, and apply refuses if the plan was edited or the live policy changed since you reviewed it. `alter policy simulate` shows which rule fired for a given request and why.

Understood yeap. Yes exactly is terraform/opentofu. The point is that we cannot do it at proxy level since we do allow any kind of operation attempt (we just filter that it's an infra req), then we generate the code based on the policy and, as you mention, once we have a validated plan we crosscheck against the compliance checker and provide an assessment about criticity and policy compliance...and trigger an automatic hitl for admin if needed otherwise it allows approve the pr an apply afterwards

Makes sense, sounds like a solid setup. If your agents ever start calling cloud/internal APIs directly instead of going through the PR flow, that's exactly what we built Alter for, and we'd love to help. Either way, good luck with it!

what does the agent do while it’s waiting on the approval to happen?

It doesn't sit there blocked. The call comes back right away with a pending status and an approval id, so the agent can tell the user "this needs sign off from X" and get on with other stuff. Then you either await it inline with a timeout or stash the id and have a worker poll it later. Once it's approved we run the exact request that was approved and the agent gets the provider's response back. If nobody answers in time it expires (10 min by default, you can set it up to 24h) and the agent has to make the call again.

Reply on news.ycombinator.com