45 karma · joined November 8, 2016
My project is https://github.com/colinator/Ariel - basically, no VLAs - instead, "just write code". Or have the agents do it.
I don't have a writeup yet about applying Ariel to _this_ robot, but this is for a previous one: https://colinator.github.io/Ariel/post1.html.
Excited to follow your progress!
As for memory: my approach is to give the robot a python repl and, basically, a file system - the LLM can write modules, poke at the robot via interactive python, etc.
Basically, the LLM becomes a robot programmer, writing code in real-time.
But seriously, good luck!
So they can go 'slow', by taking a camera image, controlling the robot, repeating. Or they can write code that runs closer to the robot in a loop, either way. I thought the latter was somehow more impressive, and that's what you see in the hand-tracking example.
https://colinator.github.io/Ariel/post1.html
I just got a bigger robot, further results forthcoming!
Just a thought, what do you think?
The MCP spec allows MCP servers to send back images to clients (base64-encoded, some json schema). However:
1) codex truncates MCP responses, so it will never receive images at all. This bug has been in existence forever.
2) Claude Code CLI will not pass those resulting images through its multi-modal visual understanding. Indeed, it will create an entirely false hallucination if asked to describe said images.
3) No LLM harness can deal with you bouncing your local MCP server. All require you to restart the harness. None allow reconnection to the MCP server.
I assure you there are many other similar bugs, whose presence makes me think that the LLM companies really don't like MCP, and are bugly-deprecating it.
Although tbh, these days I'm questioning the utility. If I'm the one writing the robot code, then I care a lot about the ergonomics of the libraries or frameworks. But if LLMs are writing it, do I really care? That's a genuine, not rhetorical question. I suppose ergonomics still matter (and maybe matter even more) if I'm the one that has to check all the LLM code....
Fascinating tech, but seemed to go nowhere.
There are now several 'manufacturing in space platform' companies, like Varda. It's not enough to just be a platform. There needs to be an actual killer app.
One advantage biotacs have over these is that I can send a guy a (very large) check and buy them. Most academically-sourced things like this cannot be gotten for any price. These look cool, I'd love to have a few.
ROS is irredeemable. I've seen it used in large projects, and the amount of time wasted "wrestling with ros" was ridiculous. I believe this will never change. Just because tensorflow existed meant folks shouldn't have looked for an easier way, and created PyTorch? No! Forge on, intrepid ros-replacers!
I feel like SNNs are like Brazil - they are the future, and shall remain so. I think more basic research is needed for them to mature. AFAIK the current SOTA is to train them with 'surrogate gradients', which shoe-horn them into the current NN training paradigm, and that sort of discards some of their worth. Have biologically-inspired learning rules, like STDP, _really_ been exhausted?
Basically a simpler version of ros. Easily, and performantly (is that a word?) connect cameras and other sensors, via xtensor/eigen/numpy to whatever algorithms you have, and control actuators/robots.
For the moment, biological neural networks (brains) are, I believe, many orders of magnitude more powerful, and especially more efficient than, current computers. But it's hard to say, because it's so hard to tease out the important parts of what the brain does, because it does so much. There are so many types of neurons, so many types of connectivity, so many interacting neurotransmitters, so many support cells that affect computation... almost like the brain is the result of a billion years of hack-upon-hack evolution...
Sounds familiar.
What people really want is for their group to beat other groups. It's always been that way, and always will.
This factionalization can occasionally be overcome by great leaders or external enemies. But now, with this 'party over country' attitude that we have, it'll can't be. Shame on us.