76 karma · joined August 23, 2021
In particular, the vision-based approach used in the evaluation has clear limitations with regard to efficiency due to its nature (small observation window, heterogeneous modality)
At Smooth we use an hybrid DOM/vision approach and we index very strongly on small models. An interesting fact is that UIs are generally designed to minimize ambiguity and supply all and only the necessary context as token-efficient as possible, and the UX is cabled up to abstract the APIs in well-understood interface patterns e.g. dropdowns or autocompletes. This makes navigation easier and that's why small models can do it, which is another dimension that must be considered
We typically recommend using APIs/MCP where available and well designed, but it's genuinely surprising how token-efficient agentic browser navigation can actually be
I agree that there is communication overhead between the agents, although so far it looks like they can communicate very effectively and efficiently. We’re also working on efficient ways to transfer more contextual information
we run headful browsers with a fingerprinting that is as stealth as possible and on top of that we can use your ip
anecdotally, making all requests originate from your own residential address has been a major success compared to other cloud-only solutions
it will be interesting to see how this will play out, having to “hide the agent” feels like a temporary work around until society accepts that agents actually do exist
the main issue with this interface is that the commands are too low-level and that there is no way of controlling the context over time
once a snapshot is added to the context those tokens will take up very precious context window space, leading to context rot, higher cost, and higher latency
that's why agents need to use very large models for these kind of systems to work and, unfortunately, even then they're very slow, expensive, and less reliable than using a purpose-made system
I wonder if a standardized interface will organically emerge over time. At the moment SKILL.md + CLI seem to be the most broadly adopted interface - even more than MCP maybe
I'd be curious to try and see if practically the difference is so little that it doesn't matter
1. it exposes low-level tools which make your agent interact directly with the browser which is extremely slow, VERY expensive, and less effective as the agent ends up dealing with UI mechanics instead of thinking about the higher-level goal/intents
2. it makes Claude operate the browser via screenshots and coordinates-based interaction, which does not work for tasks like data extraction where it needs to be able to attend to the whole page - the agent needs to repeatedly scroll and read one little screenshot at the time and it often misses critical context outside of the viewport. It also makes the task more difficult as the model has to figure out both what to do and how to do it, which means that you need to use larger models to make this paradigm actually work
3. because it uses your local browser, it also means that it has full access to your authenticated accounts by default which might not be ideal in a world where prompt-injections are only getting started
if you actively use the --chrome command we'd love to hear your experience!
you should be able to tell it to go to your localhost address and it should be able to navigate to your local app from the remote browser
let us know if you have any questions!
Agents that start using subagents rather than humans using the subagents directly
our current model is a subscription plan that determines the number of browsers available + credits top-ups for increased usage
there will be demand for AI-first online services as people continue to delegate more and more tasks to agents and this will drive implementation
For example, with remote browsers you get to give your swarm of agents unlimited and always-on browsers that they can use concurrently without being bottlenecked by your device resources
I think we tend to default to thinking in terms of one agent and one browser scenarios because we anthropomorphize them a lot, but really there is no ceiling to how parallel these workflows can become once you unlock autonomous behavior
We'll continue to make Smooth more affordable and accessible as this is a core principle of our work (https://www.smooth.sh/images/comparison.gif)
For example, the playwright MCP is very unreliable and inefficient to use. To mention a few issues, it does not correctly pierce through the different frames and does not handle the variety of edge cases that exist on the web. This means that it can't click on the button it needs to click on. Also, because it lacks control over the context design, it cannot optimize for contextual operations and your LLM trace gets polluted with incredible amount of useless tokens. This increases cost, task complexity for the LLM, and latency
On top of that, these tools rely on the accessibility tree, which is just not a viable approach for a huge number of websites
I think this paradigm was very visible in yesterday's blog post from Anthropic (https://www.anthropic.com/engineering/building-c-compiler) when they mentioned that giving the agents the ability to verify against GCC was the key to unlock further progress
Giving a browser to these agents is a no brainer, especially if one works in QA or develops web-based services
We take security very seriously and one of the main advantages of using Smooth over running things on your personal device is that your agent gets a browser in a sandboxed machine with no credentials or permissions by default. This means that the agent will be able to see only what you allow it to see. We also have some degree of guard-railing which we will continue to mature over time. For example, you can control which URLs the agent is allowed to view and which are off limits.
Until we'll be able to run everything locally on device, there must be a level of trust in the organizations that control the technology stack, passing from the LLM all the way to the infrastructure providers. And this applies to every personal information you disclose at any touch point to any AI company.
I believe that this trust is something that we and every other company in the space will need to fundamentally continue to grow and mature with our community and our users.
Our take is that, while that happens, agents today need to be able to access all the web resources that we can access as humans
Also, browsers are a really special piece of software because they provide access to almost every other kind of software. This makes them arguably the single most important tool for AI agents, and that’s why we believe that a browser might be all agents need to suddenly become ten times more useful than they already are
Also, our service comes with batteries included: the agent can use browsers in our cloud with auto-captcha solvers, stealth mode, we can proxy your own ip, etc