loclol101

Amazing! this is beautiful.

Will it support multi-agent setup across heterogeneous devices (macbook pro, RTX5090, mac mini, etc)?

We are actively exploring how frontier agents behave over different task distributions. Whether the tasks are longer horizon, multi-objective or they require more robust control signals and more. Hope this answers your question

Author here. We ran eight frontier models on web tasks across offline physical domain tasks from Bench2Drive, VLABench, IndEgo and Assembly101.

We expected these models to be jagged, but the shape of it surprised us. In all 15 model pairs, the lower-scoring model solves at least 3 tasks the higher-scoring one fails. We find that an update inside one model family moved the mean by -1.1pp, while flipping 36 of 177 episodes in both directions. The same was found on the four physical domains as well. Whether models would succeed or fail on a task is hard to predict before hand, since human labeled benchmark difficulty levels don’t necessarily mean the same to frontier models.

We also release the per-item results and model traces for exploration: https://huggingface.co/datasets/figai/RIDGE.

How general really are these jev type models? Has anyone done any broad very cross-domain eval on them?

Using jev for data labeling would be interesting. I wonder how kev compares

Definitely need to try this out locally.

Very informative! Interesting demos. Looking forward to seeing more RL environments in real world and not just simulators, especially after nvidia's physical autoresearch

Definitely ready to try this. Some scifi stuff