trzy

Why would I want to work from home in a terrible location?

I don’t want your pink eye infected skin flakes in my eyes. Have you seen how airlines “clean” plane seats?

At $20k or even $5k, the alternative is still having a real human do the job better, faster, and cheaper.

How do you scale up and down with $20k robots that have to be operated by individual humans?

Teleop is used already in these applications. But a human form factor is not required.

Would you pay $20k + $3/hr for a teleoperated housekeeer that works slowly instead of a $20/hr human? It just doesn’t make any financial sense. Automation is the only way but even then, at say $200-$300/mo, the break even is ~2 years on a $5000 robot, and that’s ignoring maintenance costs.

There are also potential ethical issues around teleoperation as a form of labor arbitrage. Should employers be able to outsource low-skill physical labor to actual humans abroad to circumvent wage and immigration laws? Is the benefit worth the economic dislocation to society? If you say yes, ask yourself why even bother reshoring manufacturing at higher cost and with trade barriers.

Teleoperation is argued to be an interim solution that trains autonomous systems the way driving trains FSD but as of right now, given what we know about just how much data is needed for current imitation learning approaches to work for fine manipulation tasks, it doesn’t seem scaleable. Maybe there will be a modeling breakthrough.

My bet is on cheaper non-humanoid form factors and data collection that doesn’t require a robot being operated for every data collector in the field.

Would love to see you do this!

ACT worked well and my repo is here: http://github.com/trzy/robot-arm

Re: LLM for supervising an imitation learning policy, check out the recent paper “Robot Utility Models” :) They even bought a TLD for it: https://robotutilitymodels.com/

I’m actually very interested in imitation learning and am building a system to do exactly this. It can be done much cheaper now, probably for about $7k total. I’ve replicated ALOHA’s ACT results using a cheap $250 table top arm but am very interested in doing something bigger, more complex, and useful. RoBart originally started as a way to build a cheap mobile base for a robot that could be operated and trained at home. Turns out that the hoverboard based set up isn’t quite optimal so for fun, I wanted to see how far I could push GPT-4 and Claude to autonomously operate it.

Sure, I uploaded it to my YT: https://youtu.be/-iW3Vzzr3oU?si=QDhB3wxnP0KYvocl

I embodied Claude, allowing it to see the world with the help of ARKit and move around freely with a hoverboard robot base.

That’s ok. Having to learn web development is an impediment to the flourishing of the human spirit.

If your project TODO lists are stretching into years, it's time to get a better project management system. I recommend paper.

Modern robotics with deep learning/imitation learning is surprisingly accessible. The low-cost robot arm I used in this project is very easy to 3D print and assemble: https://github.com/trzy/robot-arm

An iPhone app is used to teleoperate the arm and gather examples of an action. You then train the model and deploy it and the arm performs the actions based on current camera input and joint angle state.

Using an iPhone to teleoperate a robot arm and collect training data for imitation learning. Training and inference code included.

Not possible. The US is on the fast track to be humbled and humiliated by its geopolitical rival, China.

What's SOTA for ViTs that can generate good bounding boxes and easily be fine-tuned or retrained (e.g. I might want to identify household objects or UI elements)?

Aren’t agents bottlenecked by the underlying models? I’ve read that the number of “chain of thought” steps needed is proportional to task complexity. And if each step has the same probability p of success, probability of success is p^n, where n is the number of steps needed (potentially high). At a 99% success rate per step and 5 steps that’s a 95% overall success rate. 90% drops down to 60%. Not sure what the real numbers are but this seems like it could be a problem without significantly more intelligent ML models?

Hard time believing this is legit given how much time the CEO spends goofing around on social media. If it were possible to short startups, this would be a top candidate.

I personally don't have this device but we would absolutely be happy to accept a contribution adding it as a capture device to our repo :)

Thank you! So you were using it primarily as direct voice input? We'd love to explore potential use cases for always-on STT and AI.

We built an always-on open source wearable AI. Listens to (and will soon see) everything you experience, all-day long. Supports a number of capture devices, including Apple Watch and a tiny wearable with days of battery life. Interactive assistance coming soon.

The convergence of powerful LLM and VLM models with low-power wearable devices opens up entirely new frontiers for human-computer interaction, including memory augmentation, pro-active life assistance, and distributed knowledge gathering. We believe in exploring this transparently and openly, and look forward to contributions :)

Today, we launch with Apple Watch, Xiao ESP32S3 Sense, and Sony Spresense support. Code and design files for the ulta-low-power reference wearable coming soon. We are looking for select people to give out units to and set up private servers for, to help us test and spread the word!

The server is written in Python and the display client is a native iOS Swift app (but a web client is included in the project as well for those outside the Apple ecosystem).

Why wasn’t this chip more common in arcade and game consoles vs. 6502 and Z80? Cost?

I was working in HFT in New York and messing around with HoloLens in my spare time. I went to GDC and gave a talk (https://www.gdcvault.com/play/1024862/HoloLens-and-Beyond-AR...) and Apple hired me to work on AR prototyping and ideation because the creator of the HoloLens, then working at Apple, attended the talk :)

Awesome! Thanks for the detailed write up and I hope to have an excuse in the next few months to check you guys out :)

Question: Luxonis seems comparable (I’ve used their products). Are you pursuing the same use cases? How do you plan to differentiate?

llama.cpp does not have interactive mode support for LLaVA nor the web server yet. So if they're just invoking the app, there is a lot of overhead for each call until they add that in. This server doesn't have that issue.

Neither of which run LLaVA as far as I can tell. Use this until that happens.

Also, this is way easier to set up. No dependencies. Easy.

Simple API server for LLaVA based on llama.cpp. The llama.cpp project will add server and interactive support for LLaVA but I just couldn't wait. Runs great on my M1 Pro Macbook w/ 32gb RAM. Enjoy! :)

Why would you not want it in a simple, portable language that's easy to build and deploy anywhere?