Jarvis-1: Open-World Multi-Task Agents with Memory-Augmented Multimodal LLMs

Given how the desire to have great looking games provided the genesis of ML hardware, it would be a delight if AGI sprouted in the same context.

Over 200 actions? This is impressive. Would be interesting to see it applied to other games.

Id like to see anything besides Minecraft.

There is a TON of Minecraft content on the internet and it’s been used for a few ML showcases over the past few years (mine dojo for example.)

I wonder how much of Minecraft GPT already has “memorized” or in its training data.

I’m not sure how well this approach could generalize.

I’d be impressed to see it play older games with less content on the internet like morrowind or oblivion.

Very cool!